Creating a tarball from named pipes using tar or python's tarfile module

Question

I didn't realize this would be so difficult. I'm just using tar. I have two pieces of data, one's a large payload of pcap data, the other is some additional metadata in the form of a python pickled file. My goal goal is to create a .tar file containing both files, and compress with with XZ.

I want to be able to index the tarball, so that I can extract the metadata file quickly from the .tar.xz file. I am using PIXZ for xz compression, which automatically indexes tar files. Because the pcap data I am compressing is so large, I don't want to touch the disk until after it has been compressed. I am making several other changes to it with tcprewrite, but those aren't important.

I have all my tools set up to use named pipes, it's beautiful and very quick. The unfortunate problem is that tar doesn't work with named pipes. Every time I try to compress a named pipe, it just writes /dev/fd/#numbers to the tarball. And I can't just write the pcap data to disk or /dev/shm to tar it, it is critical that the pcap data doesn't hit the disk until is is compressed.

I'm writing this whole utility in python, so I tried using python's tarfile module. The documentation claims that it works with FIFO, but when I use tarfile.add() I run into the same issue. I've tried investigating alternatives like zip (which has a -FI option for working with named pipes), but I need to use tar for our existing infrastructure.

I tried both gnutar and bsdtar and neither of them will work with pipes. It ultimately doesn't matter what tools I use as long as the end result is a .tar.xz file which is indexed and allows fast access to the metadata.

Answer 1

Don't get depressed over this... I'm sure you'll manage to solve this problem!

I think what's tripping you with tar is that by "supports named pipes" they mean that it can recognize the named pipes and store them as named pipes inside the archive, so that you can later restore them as named pipes again... Which is not really what you want.

Also, the format of tar files is not very suitable for what you are doing, since the entry describing a file is stored before its contents and the file entry must contain the file size, so unless you know the exact size of the contents in advance, it's hard to do that...

There is this solution (see TarFileStdin), which uses a hack to solve that problem. It inserts a TarInfo with file size zero, then stores the contents of the file and finally seeks back to the offset of the original TarInfo and overwrites it with the correct size... It's a bit too hacky, but it should work... But read on.

---

You mentioned "I want to be able to index the tarball, so that I can extract the metadata file quickly from the .tar.xz file", so that looks more like a ZIP file! The ZIP format stores the contents of all files first and then stores a table of file information and offsets at the end of the ZIP. In that sense, it is indexed, like you mentioned. Listing the contents of the ZIP can be done quickly, since it's easy for tools to find that file table by starting from the end of the file.

You could use ZIP's native compression format, or you could just use ZIP's "store" mode (uncompressed) and then add a xyz.pcap.xz file inside it. Adding a *.xz file to the ZIP would have the convenience that you can use an external compressor, such as a parallel xz.

Python 3's zipfile.ZipFile objects have an open() method which allows you to add a file by name only and receive a file object to which you can write the contents.

You could use that API and shutil.copyfileobj() to add your compressed pcap from a named pipe to the ZIP file:

import shutil
import zipfile

with zipfile.ZipFile('mydata.zip', 'w') as zf:
    with zf.open('xyz.pcap.xz', 'w') as outputf:
        with open('/path/to/namedpipe', 'r') as inputf:
            shutil.copyfileobj(inputf, outputf)
    zf.write('metadata.pickle')  # from local directory

This code snippet is assuming you're writing the already xz-compressed pcap data to the named pipe and that you have the metadata already serialized to a file named 'metadata.pickle' in the current directory. (Of course, you could use ZipFile's open() to serialize the pickle metadata directly into an entry in the ZIP file too!)

If you want to use zipfile's native compression, you can set a default compression to ZipFile:

with zipfile.ZipFile('mydata.zip', 'w', zipfile.ZIP_LZMA) as zf:

(The default is ZIP_STORED, which means no compression, which is probably what you want if you're gonna pipe xz-compressed data there.)

See the documentation of zipfile for more details. Newer Python's have even more features, for instance with Python 3.5 you can actually write the zipfile to a pipe so you could for instance upload it straight to a remote host through SSH.

---

I hope you find this useful! If you really need a tarball, try this answer, but I really think the zipfile solution using Python 3 is a better approach for the use case you describe! So if that format is a possibility, I'd really recommend it.