The easiest solution does not have to be bad.
You need neither ssh-agent nor ssh-add when you use a passwordless private key, see https://stackoverflow.com/a/48290333/11154841.
That does not need to be insecure: you simply need to delete both keys of the key pair right after their usage. That means, you must delete the public key on the server and delete the private key on the client. It goes without saying: never use them again, do not even keep a backup somewhere.
Mind that you can create the public key from a private key, but not the other way round. Normally, it should suffice to simply delete the public key from the registered keys on your Git portal, but they should better be both deleted, so that the same public key can never ever be used again. Even if someone had stolen your private key, it would be of no use if you simply never use its public key again.
With a passwordless private key, you can even use it in Docker to get around any password entries. You can clone a git repo without any password, the passwordless private key is all you need. See Dockerfile: clone repo with passwordless private key. Errors: “authentication agent” or “read_passphrase: can't open /dev/tty” as an example.