Is there a program that concatenates non-seekable streams (size not known in advance) and can separate them again?

Question

I'm trying to concatenate multiple input files/streams into one stream (using the imaginary command stream-cat), pipe that stream into ssh and on the remote host separate it back into individual files/streams (stream-sep), like in this example, which is for demonstration purposes only:

stream-cat <( zfs send tank/vm@snapshot ) somefile.txt | ssh user@host "stream-sep >( zfs receive tank/vm@snapshot ) somefile.txt"

Explanation of the example: zfs send outputs a large stream of data whose size isn't known in advance (that's why tar can't handle it). That data stream is concatenated with the content of a regular file somefile.txt. The resulting stream is piped into ssh, where it is separated again. The first stream gets piped into zfs receive, while the second is written to a regular file.

Such a program should be straightforward to implement by reading non-seekable streams in chunks and always writing the chunk size followed by the data, until the end of the stream is reached. The overhead would be minimal.

Does such a program exist already?

Answer 1

What you're describing is multiplexing; something that requires a protocol (i.e., a formal specification on how to deal with data).

There's a lot of approaches to that. For example, you'll notice that your computer can perfectly well download multiple files at the same time via HTTP – even from the same server. That feature is foremost brought to you by TCP, which, as the transport protocol, allows different streams to be sent and "taken apart" at the receiving end.

So, TCP already offers that functionality, and you can simply start two concurrent SSH connections and make use of it!

zfs send | zstd -10 | ssh user@host 'zstd -d | zfs receive tank/vm@snapshot' &
# ^         ^    ^     ^             ^     ^   ^                             ^
# |         |    |     |             |     |   |                             |
# |         |    |     |             |     |   |     Tell your own shell to run
# |         |    |     |             |     |   |     this in the background and
# |         |    |     |             |     |   |     not block
# |         |    |     |             |     |   |                         
# |         |    |     |             |     |   |                            
# |         |    |     |             |     |   \---- Program to receive in the end
# |         |    |     |             |     |
# |         |    |     |             \-----+-------- use zstd to decompress
# |         |    |     |                             received data
# |         |    |     |            
# |         |    |     |
# |         |    |     \---------------------------- our first ssh invocaton
# |         |    |
# |         \----+---------------------------------- use zstd to compress at medium
# |                                                  high compression level (10)
# |
# \------------------------------------------------- the first program whose output
#                                                    we send

cat somefile.txt | ssh user@host 'cat > somefile.txt'
#                                                    Second SSH connection

Of course, instead of the slightly inelegant cat somefile.txt | ssh … > somefile.txt, you would probably just use scp somefile.txt user@host:somefile.txt (which uses SSH under the hood, but doesn't do a shell connection, but uses the built-in SCP layer in SSH to copy the file).

You can make the connection establishment for the second connection faster by adding the following to your ~/.ssh/config file:

ControlMaster auto
ControlPath /tmp/.ssh-socket-%h_%p_%r

This will tell SSH to try and re-use the one SSH session to send multiple encrypted streams at the same time (this works with any combination of scp and ssh as well).

Answer 2

Crude implementations of those stream-cat, stream-sep could easily be written in perl:

stream-cat() {
  perl -ne 'BEGIN{$/ = \0x7fff}
            print pack("n", $c|length()<<1), $_;
            $c = !eof' -- "$@"
}
stream-sep() {
  perl -e 'while($/ = \2, $_ = <STDIN>) {
             $n = unpack "n";
             open OUT, shift@ARGV unless $n & 1;
             if ($n>>=1) {$/ = \$n; $_ = <STDIN>; print OUT}
           }' -- "$@"
}

or same as #! /bin/sh - scripts instead of sh functions.

(error handling left as an exercise to the reader :-).

stream-cat sends records up to 32767 byte large prefixed by a network-encoded (big-endian) short whose lowest bit indicates whether it's the start of a new stream (0) or the continuation, and the remaining bits are the size.

And then, for example:

$ cat a
test
$ stream-cat 'seq 10|' a | stream-sep '|wc -l' '>b'
10
$ cat b
test

So in your case:

stream-cat 'zfs send tank/vm@snapshot|' somefile.txt |
  ssh user@host 'stream-sep "|zfs receive tank/vm@snapshot" ">somefile.txt"'

A rare case where the unsafe form of open (here also used by <> as used by -n), which allows <file, >file to open a files in readonly or write-only mode or |cmd, cmd| to pipe to/from a command is actually useful.

Using those |cmd / cmd| scales better than your <(cmd) / >(cmd) as only one is open at a time, so you could send thousands of separate streams without issues.