There are two main things which usually make SSH slow: network latency and the time to establish a connection. Those two things are correlated, but initial handshake can take considerable time even if you try to connect to machine located in the same room.

The more you use SSH, the more time you waste waiting for SSH handshake. The extreme example of this overhead is Ansible playbooks. An average playbook has lots of tasks and every task needs a new SSH connection. You can easily spend 80% of your time just making handshakes. This handshake overhead is the only reason why you need to wait before logging in to remote machine.

Fortunately, SSH offers a way to avoid handshakes by keeping your connections open after you log out. This way, every subsequent login will be instantaneous. Here is how you can enable connection caching:

mkdir -p ~/.ssh/sockets

Then, open ~/.ssh/config and add the following lines:

Host *
ControlMaster auto
ControlPath ~/.ssh/sockets/%r@%h-%p
ControlPersist 600

You can adjust ControlPersist to change caching time or to ask SSH to keep those connections forever (well, till you reboot your machine).

Before:

time ssh nextcloud exit

real	0m2.822s
user	0m0.003s
sys	    0m0.003s

After:

time ssh nextcloud exit

real	0m0.019s
user	0m0.000s
sys	    0m0.003s