6

Updates

  • I've narrowed it down to only occurring on the wireless network at work. From home, I experience no lag/delay
  • It also seems that after "heavy" SSH use and tunneling through one or two machines, the network drops me completely and I am forced to manually reconnect
  • Looking at WireShark captures of the SSH session from OpenSSH versus JSch, it seems that OpenSSH has a handful of TCP Dup Ack, TCP Retransmission, and TCP Spurious Retransmission packets. But these should sort themselves out, being TCP..

Problem

Recently when connecting to remote machines using SSH, I've noticed that there is a distinct delay of around 1 second between me typing something and the characters actually appearing. If I hold down keys, they'll appear in "batches" of around 20 characters at ~1s intervals.

System Details

$ ssh -V
    OpenSSH_7.2p2 Ubuntu-4ubuntu1, OpenSSL 1.0.2g-fips  1 Mar 2016        

$ echo $SHELL $TERM $DISPLAY
    /bin/zsh xterm-256color :0
$ uname -r; cat /proc/version
    4.4.0-22-generic
    Linux version 4.4.0-22-generic (buildd@lgw01-41) (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2) ) #40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016

Observations

An odd observation is that the delay is there only when the allocation of a pseudo terminal occurs (i.e. the default, or the -t option). When the pseudo terminal is forcedly disabled, I can type commands and get their output without delay.

  • My ~/.ssh/config file only contains "plain entries of the following format:
Host somename
    HostName 192.168.1.0
    User someuser
    IdentityFile ~/.ssh/id_rsa
  • There is no delay when I explicitly disable the pseudo terminal (i.e. using the -T option)
  • Altering the settings mentioned in similar questions does not resolve the issue
  • IntelliJ Idea Ultimate Edition can create SSH connections which do not have this issue. The IDE is running the Java SSH client JSch
    • This has convinced me that it is a local issue and presumably not related to my network interface
  • My coworkers on the same network do not have this issue, though they're using Macbooks

What I've Tried

  • Enabling compressing, disabling X11 forwarding, various methods for speeding up the login process
  • SSHing to localhost (no lag/delay here)
  • SSHing to a variety of machines (i.e. not just one) -- delay is present
  • Downloading the latest stable version of OpenSSH and building it for my system
    • The freshly compiled binary has the same issue, leading me to think it's caused by some wonky configs
  • Ensuring my /etc/ssh/ssh_config looks "normal"
    • I've included it at the end of the post just in case
  • Purging SSH and re-installing using apt-get
  • Dropping into a virtual TTY (I think that's what they're called...) using Ctrl + Alt + F{1-6} and SSHing from there
  • Rebooting
  • Switching to a different network
    • The lag/delay only seems to be an issue on the network at work. At home, there's no noticeable delay/lag
  • Some other stuff that I'm forgetting

tl;dr

My SSH binary/config/something seems to have a delay when typing and I've tried a lot of things with no success. Maybe it's some character buffer thing? Who knows! Hopefully you do.

ssh -vvv

OpenSSH_7.2p2 Ubuntu-4ubuntu1, OpenSSL 1.0.2g-fips  1 Mar 2016
debug1: Reading configuration data /home/redacted/.ssh/config
debug1: /home/redacted/.ssh/config line 6: Applying options for ops1
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "xx.xx.xx.xx" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to xx.xx.xx.xx [xx.xx.xx.xx] port 22.
debug1: Connection established.
debug1: identity file /home/redacted/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/redacted/.ssh/id_rsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to xx.xx.xx.xx:22 as 'redacted2'
debug3: hostkeys_foreach: reading file "/home/redacted/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/redacted/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from xx.xx.xx.xx
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: [email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa,[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: MACs ctos: hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: MACs stoc: hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: [email protected] compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: [email protected] compression: none
debug3: send packet: type 34
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048 closed
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
logout
debug3: channel 0: will not send data after close
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug3: send packet: type 97
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)

debug3: send packet: type 1
debug1: fd 1 clearing O_NONBLOCK
debug3: fd 2 is not O_NONBLOCK
Connection to xx.xx.xx.xx closed.
Transferred: sent 3376, received 3072 bytes, in 0.8 seconds
Bytes per second: sent 3981.4, received 3622.9
debug1: Exit status 0

ssh -vvvT

OpenSSH_7.2p2 Ubuntu-4ubuntu1, OpenSSL 1.0.2g-fips  1 Mar 2016
debug1: Reading configuration data /home/redacted/.ssh/config
debug1: /home/redacted/.ssh/config line 6: Applying options for ops1
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "xx.xx.x.xx" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to xx.xx.x.xx [xx.xx.x.xx] port 22.
debug1: Connection established.
debug1: identity file /home/redacted/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/redacted/.ssh/id_rsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to xx.xx.x.xx:22 as 'redacted'
debug3: hostkeys_foreach: reading file "/home/redacted/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/redacted/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from xx.xx.x.xx
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: [email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa,[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: MACs ctos: hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: MACs stoc: hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: [email protected] compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: [email protected] compression: none
debug3: send packet: type 34
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048 drain
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug3: send packet: type 96
debug2: channel 0: input drain -> closed
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug3: send packet: type 97
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)

debug3: send packet: type 1
debug1: fd 1 clearing O_NONBLOCK
debug3: fd 2 is not O_NONBLOCK
Transferred: sent 3040, received 2800 bytes, in 1.9 seconds
Bytes per second: sent 1598.6, received 1472.4
debug1: Exit status 0

/etc/ssh/ssh_config


# This is the ssh client system-wide configuration file.  See
# ssh_config(5) for more information.  This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
#  1. command line options
#  2. user-specific file
#  3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for some commonly used options.  For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.

Host *
#   ForwardAgent no
#   ForwardX11 no
#   ForwardX11Trusted yes
#   RhostsRSAAuthentication no
#   RSAAuthentication yes
#   PasswordAuthentication yes
#   HostbasedAuthentication no
#   GSSAPIAuthentication no
#   GSSAPIDelegateCredentials no
#   GSSAPIKeyExchange yes
#   GSSAPITrustDNS no
#   BatchMode no
#   CheckHostIP yes
#   AddressFamily any
#   ConnectTimeout 0
#   StrictHostKeyChecking ask
#   IdentityFile ~/.ssh/identity
#   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
#   IdentityFile ~/.ssh/id_ecdsa
#   IdentityFile ~/.ssh/id_ed25519
#   Port 22
#   Protocol 2
#   Cipher 3des
#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
#   MACs hmac-md5,hmac-sha1,[email protected],hmac-ripemd160
#   EscapeChar ~
#   Tunnel no
#   TunnelDevice any:any
#   PermitLocalCommand no
#   VisualHostKey no
#   ProxyCommand ssh -q -W %h:%p gateway.example.com
#   RekeyLimit 1G 1h
#    SendEnv LANG LC_*
    HashKnownHosts yes
   # GSSAPIAuthentication yes
   # GSSAPIDelegateCredentials no
Improve this question
5
  • 1
    I don't think it's your config; I'm pretty sure it's network lag. Commented May 27, 2016 at 20:00
  • That's what I thought, too, but my coworkers on the same network do not have this issue, and IntelliJ IDEA can create delay-free SSH connections. Commented May 27, 2016 at 20:06
  • You said you tried sshing into your localhost. What were the results of that? Same lag? No lag? Commented May 27, 2016 at 20:47
  • Ah sorry. No lag Commented May 27, 2016 at 21:03
  • I was recently getting a ton of lag either from my host to a remote host or to my host from a remote host. Ethtool showed my connection at 1Gb, even the light on my switch said 1Gb. However it still was slow. I pulled all the power to my computer then pressed the power button on my computer a few times. Plugged in power and powered it back up. My connection was then solid and i was getting my full 1Gb speed. Commented May 27, 2016 at 21:19

2 Answers 2

Reset to default
1

Unfortunately I can't comment yet.

What comes to my mind on the client:

Have you had a look at top to see the CPU load? Maybe the SSH process uses up CPU for encryption.

Have you had a look at ssh -v[vv] and inspected for some strangeness? Maybe server and client agree on a very secure cipher or MAC. Have a look for

debug2: ciphers ctos: arcfour
debug2: ciphers stoc: arcfour
[...]
debug1: kex: server->client cipher: arcfour MAC: [email protected] compression: [email protected]
debug1: kex: client->server cipher: arcfour MAC: [email protected] compression: [email protected]

(where arcfour is really one of the weakest, lowest-CPU algorithms. I am connecting via an SSH proxy). Also, look for rekeying messages.

Also, compression might be an issue. However, ssh does not seem to be too explicit about the compression level here.

debug2: compression ctos: [email protected],zlib,none
debug2: compression stoc: [email protected],zlib,none

You didn't say explicitly, how 'remote' your remote servers are. In the same network, in a different network on the same campus, via the internet?

If via the internet, maybe your corporate firewall is deep-inspecting SSH traffic on port 22. Maybe a port change could help, if you can control the SSH servers /etc/ssh/sshd_config file.

If via the internet, is SSH the 'only' tunnel you use, or are you using SSH via an additional VPN?

If on the same campus, also firewalling, routing, inspecting issues come to my mind. I once was blaming my internet provider for problems with my internet; it was damn slow. Went on for days. Until I found out that I enabled debugging on my Cisco router, which ate up resources there.

If local or remote, are you connecting directly or via an SSH proxy, gateway or jump host? In that case, like with the VPN, you have double encryption.

Improve this answer
6
  • Really good thoughts, thanks. I forwarded the question to the IT department at work as well. Most of the remote machines are via internet located in Amazon's us-east-1 and I am currently on the east coast. No add'l VPN is used. The main thing that's stumping me is why it's only an issue for me and not my coworkers. Commented May 31, 2016 at 15:57
  • What else is going on on your client? Is there an ssh -v[vv] logging to a file? What about iotop, does it show much more disk i/o than on your colleagues' workstations? Do you have significantly less RAM than them? Same drivers for the network interface? Commented May 31, 2016 at 20:28
  • Cabling, interfaces and connectors OK? Duplicate MAC addresses in the net? Jumbo frames vs. normal frames? Duplicate IP addresses (unlikely)? Subnetting issues? Complex NAT or something in your iptables? TCP windowing values too low/high? Commented May 31, 2016 at 20:34
  • Wouldn't most of those be ruled out by the fact that both JSch and ssh -T me@remote are connecting and performing without the delay? Commented May 31, 2016 at 20:37
  • I've included the outputs of ssh -vvv me@remote as well as ssh -vvvT me@more (first one lags, second does not). Regarding the hardware specs questions: our specs are almost the same. I'm on a Dell XPS 15 with 16GB RAM, 512GB SSD used at ~15%, nothing else running except Google Chrome. Not entirely sure about the network drivers except that I'm on the latest stable available. Commented May 31, 2016 at 20:42
1

I had also this problem but after disabling NAT/NAT boost on my router it has improved. NAT disadvantages

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.