Return to Answer

Another option: use "parallel scp" (somehow or other), then it will use more cores...

nc: 0m18.706s (100% cpu, v1.84, no -q option? without -q option can transfer truncated files...)
iperf3: 0m10.013s (100% cpu)
socat: 0m10.293s (88% cpu, possibly maxed out)

So on one instance netcat used less cpu, on the other socat, so YMMV.

nc: 0m18.706s (100% cpu, v1.84, no -q option
iperf3: 0m10.013s (100% cpu, but can go up to at least 20Gbe with 100% cpu so not sure it matters)
socat: 0m10.293s (88% cpu, possibly maxed out)

So on one instance netcat used less cpu, on the other socat, so YMMV.

With netcat, if it doesn't have a "-N -q 0" option it can transfer truncated files, be careful...other options like "-w 10" can also result in truncated files.

worth double checking after tweaking to make sure it didn't at least hurt.

suggestions from linode, add to /etc/sysctl.conf:

net.core.rmem_max = 268435456 
net.core.wmem_max = 268435456 
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.core.netdev_max_backlog = 250000
net.ipv4.tcp_no_metrics_save = 1
net.core.default_qdisc = fq 

Additionally, they would like you to run:

 /sbin/ifconfig eth0 txqueuelen 10000 

worth double checking after tweaking to make sure changes don't cause harm too.