How to pass multiple files in one stream to process them pipe-like (without saving) on the receiving side?

Question

I need to determine the filetype on an indeterminate number of base64-encoded filestreams coming from stdin (image files base64-encoded and cat'ed).

A single file would simply be ... | base64 -d | identify -.

The issue for multiple files is determing the EOF for each file in the stream. Even if I separate the files with \004 before sending them to stdout, the receiving end of the pipe (identify or file, et al) appear not to recognise there is more than one file in the stream (yes, I know the stream is one file, but I was hoping en EOF mid-stream would somehow, for various values of "somehow", work.

I've tried reading stdin in a while read REPLY loop, but read is line-based, not file-based, so appears to not work as I want.

[edit, later] There is between 3-10 files all less than 400KB, so size and processing isn't an issue for my use case, but I'm interested in the question generally.

[eidt, later] I'm trying to avoid tmp files (which is my current solution), mainly because I'm philosophically opposed to using the filesystem as a buffer between two adjacent processes when an inter-process stream is much more efficient. I know that sounds pompous, so for a solution that needs to work right now of course I use tmp files. However, I realised there is a gap in my knowledge and I'm trying to find the answer for the general case.

Answer 1

Even if I separate the files with \004 before sending them to stdout ...

Good thing you can modify the sending procedure. My solution is as follows:

for f in *.jpg; do echo S; base64 "$f"; echo ""; done |
# the above is just an example sending process
while read dummy; do
  sed -u '/^$/q' | base64 -d | identify -
done

Clarification:

• Single "file block" starts with an expendable line ("S" in this case) that carries no data. If read cannot find a line, the whole command ends.
• sed passes data to decoder until there's an empty line (note: additional empty line doesn't change the output of base64 -d).
• It's crucial to use unbuffered sed (-u flag); otherwise one sed could read too much and eventually discard what it thinks is excessive data; then the next sed (consequently the next identify) wouldn't get all the data it should.

Hints:

• The extra line can carry metadata instead of "S", like a filename or so (but beware of newlines in names etc.).
• Because base64 produces bigger output than its input, you may want to use gzip on the both sides, especially if your stream travels via the Internet.

Answer 2

It does sort of work if you pipe the input to while read with the delimiter set to \004 (using the $'...' expansion, since read doesn't interpret backslash escapes)

for x in *.jpg ; do base64 < "$x" ; echo -e '\004';  done | 
    while read -rd $'\004' file ; do 
        echo "$file" | base64 -d | identify - 
    done

It's awfully slow for larger files, though, since the shell can't know if something inside the loop is going to read the pipe, and so read needs to read byte-by-byte. This should probably be implemented in Perl or some other real programming language so that buffering could be controlled exactly.