Skip to main content
edited tags; edited title
Link
don_crissti
  • 85.6k
  • 31
  • 234
  • 262

Bash: How to concatenate results of multiple commands and pipe into another without intermediate file?

Source Link
mngeek206
  • 3.1k
  • 4
  • 22
  • 29

Bash: How to concatenate results of multiple commands and pipe into another without intermediate file?

Suppose I have four very large text files, all compressed with xz.

file1.log.xz
file2.log.xz
file3.log.xz
file4.log.xz

What I'd like to do is concatenate the uncompressed contents of these four files into a new file file.xz. The thing is, I would ideally like to not have to go through intermediate files.

The files are very large log files that are gigabytes in size. Compressed, they're under 100MB, but if I were to expand all four files then re-concatenate, I'd need at least 30GB of storage to store the uncompressed files. I could, of course, then cat all the uncompressed files into xz to recompress them:

cat file1.log file2.log file3.log file4.log | xz -ve9 - > newfile.log.xz

I know how I could concatenate two files at the command line without an intermediate, assuming one was uncompressed and one was compressed:

xz -d -c file2.log.xz | cat file1.log - | xz -ve9 - > files1and2.log.xz

But this will only work for one file, and one of them has to already be uncompressed.

I'm not sure if I can just cat the various .xz files together - let's assume they may have been compressed with different parameters.

On a higher level, the question itself could be asked: can you take the output of multiple (more than two) commands, concatenate those outputs, and pipe them into another process without intermediate files? (Hypothetical scenario: imagine I'm doing some kind of processing on all four very huge files using a script that outputs to stdout, and wanting to put the output into another compressed file.)

Is it possible to do this using only shell commands?