Intel Introducing USB4STREAM Protocol For Linux - Opening Up Some Nifty Uses For USB4

An exciting Intel innovation expected to be added for the upcoming Linux 7.2 kernel is introducing the new USB4STREAM protocol for USB4/Thunderbolt as a "super simple" way to "basically just transfer raw packets from one host to another". This can be useful for quickly backing up a system from one host to another, sharing of web cameras or other peripherals across systems, or other environments where not having networking or wanting to avoid the traditional Linux networking stack.

Intel Thunderbolt maintainer Mika Westerberg has been working on the USB4STREAM support with the thunderbolt_stream driver that looks like it's all buttoned up in time for next month's Linux 7.2 merge window.

The thunderbolt_stream driver exposes /dev/tbstreamX devices on each host of a directly-connected USB4/Thunderbolt cable. From there data can be transferred using regular file-system operations, e.g. you can dd or cat from one system to another or similar commands.

Westerberg further elaborated on the patch introducing the USB4STREAM support to the Linux kernel with some additional use-cases/examples:

"Introduce USB4STREAM protocol and Linux implementation. This allows two (or more) hosts to transfer data directly over Thunderbolt/USB4 cable through a character device without need to go through the network stack.

Any application that supports read(2) and write(2) in some form should be able to use the device without changes. The data is sent out to the other side over a tunnel inside Thunderbolt/USB4 fabric. The character device is called /dev/tbstreamX where X is the minor number starting from 0.

All stream devices need to be configured first. This is done through ConfigFS interface. There can be multiple streams at the same time (this depends on number of DMA rings and available HopIDs) and a single stream supports traffic in both directions. For example there could be an application that uses one stream as control channel and another one as bi-directional data channel.

A real use-case for this is to take a backup as a part of recovery initramfs tooling (no need to setup networking or have ssh or similar tooling as part of the initramfs). Say we want to backup the disk of host1 to host2. First Thunderbolt/USB4 cable is connected between the hosts (there can be devices in the middle too) then the receiving side configures the stream:

host2 # mkdir /sys/kernel/config/thunderbolt/stream/0-1.0
host2 # mkdir /sys/kernel/config/thunderbolt/stream/0-1.0/backup
host2 # echo -1 > /sys/kernel/config/thunderbolt/stream/0-1.0/backup/in_hopid
host2 # echo -1 > /sys/kernel/config/thunderbolt/stream/0-1.0/backup/out_hopid

We use automatic HopID allocation (writing -1 to HopIDs) for simplicity. From this point forward the /dev/tbstream0 can be used pretty much as regular file:

host2 # dd if=/dev/tbstream0 of=/tmp/host1.nvme0n1.backup-$(date +%F) bs=256k

The host that is being backed up then configures the stream accordingly:

host1 # mkdir /sys/kernel/config/thunderbolt/stream/0-503.0
host1 # mkdir /sys/kernel/config/thunderbolt/stream/0-503.0/backup

Here we take advantage of the fact that host2 also announces the active streams through XDomain properties so the name "backup" gives us the HopIDs. It is also possible to configure them manually in the same way we did for host2.

Then it is just a matter of copying the data over:

host1 # dd if=/dev/nvme0n1 of=/dev/tbstream0 bs=256k

Similarly it is possible to transfer parts of the filesystem. For example copy contents of mydir over to the host2:

host2 # gunzip < /dev/tbstream0 | tar xf -
host1 # tar cf - mydir | gzip > /dev/tbstream0

Other end of the spectrum use-case is "borrowing" laptop (host1) camera to desktop (host2):

host2 # gst-launch-1.0 filesrc location=/dev/tbstream0 ! jpegdec ! videoconvert ! autovideosink

host1 # gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080 ! jpegenc quality=90 ! filesink location=/dev/tbstream0

Once the streams are no longer needed they can be removed:

host1 # cd /sys/kernel/config/thunderbolt/stream/
host1 # rmdir -p 0-503.0/backup

host2 # cd /sys/kernel/config/thunderbolt/stream
host2 # rmdir -p 0-1.0/backup"

The patch is in the Thunderbolt.git's "next" branch. With those patches making it into the "next" branch, assuming it's then submitted in time for the USB/Thunderbolt Git tree proper ahead of the Linux 7.2 merge window in mid-June, this nifty protocol should make it for the Linux 7.2 kernel.

There is also this documentation patch that further lays out how to make use of USB4STREAM and its innovative capabilities.