1

we're copying a large zfs snapshot (zfs send) from site-1 to site-2. site-1 and site-2 are remote, and are connected via IPsec (read: slow)

on site-2, we need the data on two different zfs servers. currently we're using two simultaneous ssh "zfs send" | zfs receive commands. this is of course a waste of bandwidth.

we've experimented with the following (1 remote ssh, tee, 1 local ssh):

[root@site-2-1 ~]# \
    time \
    ssh root@site-1-0 \
        "cat /tmp/100000000.rnd | pigz -1" \
    | pv -B 10m -s 100000000 \
    | tee \
        >(ssh root@site-2-2 "pigz -d > /tmp/test.rnd.1") \
    | pigz -d \
    > /tmp/test.rnd.1

95.4MiB 0:00:52 [1.83MiB/s] [===<snipped>===>] 100%

real    0m52.062s
user    0m0.979s
sys 0m1.100s

which turned out to be slower than a simulation of the original solution (2 remote ssh's simultaneously):

[root@site-2-1 ~]# \
    time \
    ssh root@site-1-0 \
        "cat /tmp/100000000.rnd | pigz -1" \
    | pv -B 10m -s 100000000 \
    | pigz -d \
    > /tmp/test.rnd.2
95.4MiB 0:00:30 [3.09MiB/s] [===<snipped>===>] 100%

real    0m30.837s
user    0m0.827s
sys 0m1.003s


[root@site-2-2 ~]# \
    time \
    ssh root@site-1-0 \
        "cat /tmp/100000000.rnd | pigz -1" \
    | pv -B 10m -s 100000000 \
    | pigz -d \
    > /tmp/test.rnd.2
95.4MiB 0:00:38 [ 2.5MiB/s] [===<snipped>===>] 100%

real    0m38.187s
user    0m0.840s
sys 0m0.846s

we've looked into tee's code, our prime suspect, and it seems it operates out of a single thread.

can anyone recomment a faster alternative to tee, or a totally different solution maybe?

thanks in advance, guven

Improve this question

1 Answer 1

Reset to default
1

suggestion:

zfs send > file
scp file server-at-site2:
ssh server-at-site2 zfs receive < file
ssh server-at-site2 cat file | ssh second-server-at-site2 zfs receive

That requires two transfers, but presumably the second will be faster (because local network, etc)

Improve this answer
3
  • our file is huge as well (> 1TB) so we'd need another storage for that Commented Jun 22, 2016 at 13:30
  • I was afraid you'd say that :-) How about receiving it on the first server, and then sending it again from that server to the second one? That might take a little longer depending on the speed of your ZFS, but at least you don't need the extra storage and waste less bandwidth. And since a ZFS receive requires that the start snapshot is present and creates (at least) the end snapshot, this doesn't require extra snapshots of something. Commented Jun 22, 2016 at 14:54
  • that is one of our last resort options, if all else fails :) Commented Jun 24, 2016 at 9:56

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.