If you have control of the machine to the point that you are automating tasks on it, then why is adding your key to authorized_keys not an option?
ssh-copy-id -i ~/.ssh/foo [email protected]
Then you don't have to enter a password every time you connect.
If the biggest problem is that connections take a long time to connect, you could reuse a single connection by adding control master to your ssh config. Leave that one connection running, an any subsequent connections will be nearly instantaneous.
Host *
ControlMaster auto
ControlPath ~/.ssh/sockets/%r@%h-%p
ControlPersist 600
https://puppetlabs.com/blog/speed-up-ssh-by-reusing-connections
In the long run, if you are automating tasks, you are proabably better off using an automation framework that handles establishing the connection for you.