4

I found bash ignores binary zero on input when reading using the read buildin command. Is there a way around that?

The task is reading from a pipe that delivers binary data chunks of 12 bytes at a time, i.e. 2 ints of 16 bit and 2 ints of 32 bit. Data rate is low, performance no issue. Since bash variables are C-style, the obvious read -N 12 struct does not work, bytes beyond a NUL are not accessible. So I figured I need to read the data byte by byte, using read -N 1 byte. Problems easy to fix are escapes (requires -r), and UTF multi-byte character coding (export LC_ALL=C). The problem I'm so far unable to solve is to deal with zero bytes. I thought they'd show up as empty variable byte, but in fact read -r -N 1 byte does not return at all upon zero (ignores zeros) but returns with the next following non-zero byte in the data stream.

This is what I was attempting to do, which, as long as no zero comes in, works without flaws:

export LC_ALL=C

while true;
  do
     for ((index = 0; index < 12; index++))
       do
          read -r -N 1 byte
          if [ -n "${byte}" ]; then
               struct[${index}]=$(echo -n "${byte}" | od -An -td1)
             else
               struct[${index}]=0
            fi
       done
... # some arithmetics reconstructing the four bitfields and processing them
  done < pipe

It turns out the else branch in the if is never taken. A data chunk of 12 bytes that includes a zero does not make the for loop run 12 times, instead it awaits more data to fill the struct array. I demonstrated the behaviour by feeding the pipe 12 bytes using the command

echo -en "ABCDE\tGH\0JKL" > pipe

Since it is so easy to fool oneself with this, I verified the sending of zeros with

~# mkfifo pipe
~# od -An -td1 <pipe &
[1] 25512
~# echo -en "ABCDE\tGH\0JKL" > pipe
~#    65   66   67   68   69    9   71   72    0   74   75   76

[1]+  Done                    od -An -td1 < /root/pipe

Is there a way to change this behaviour of bash? Or how else can the zero bytes be read?

Improve this question
4
  • Bash simply can't store NUL bytes in variables. You may want to look to other languages. Commented Dec 29, 2020 at 14:40
  • AFAIK, zsh is the only shell that can store 0 bytes in its variables. ksh93 has some helper features to help you base64 encode/decode data. Commented Dec 29, 2020 at 15:04
  • I do not need the NUL byte stored in a variable. As I pointed out, bash uses C-style strings, so in a sense, every variable contains a NUL byte. That's why I expect to receive an empty string from the read builtin. So all I need is that the read builtin returns on every byte, not non-zero ones only. Commented Dec 29, 2020 at 15:29
  • Related: bash: read: how to capture '\n' (newline) character?. See also Fastest way to determine if shebang is present Commented Dec 29, 2020 at 16:07

1 Answer 1

Reset to default
3

bash variables can't store NUL bytes (only zsh does, though see also ksh93's printf %B and typeset -b using base64 encoding). Its read builtin will also skip NUL bytes in input.

However, here, you could use:

LC_ALL=C IFS= read -rd '' -n1 c

That is read up to one byte off a NUL-delimited record. So if $c is empty, that means either EOF (but then read's exit status would be non-zero) or a NUL byte was read.

For both, you can get the numeric value of that byte with:

LC_ALL=C printf -v value %d "'$c"

So:

while
  IFS= LC_ALL=C read -rd '' -n1 c &&
    LC_ALL=C printf -v value %d "'$c"
do
  echo "Got byte with value $value"
done

Would read the input one byte at a time until EOF and support NUL bytes.

Or you could always do:

value=$(dd bs=1 count=1 2> /dev/null | od -An -vtu1)

Or with some od implementations:

value=$(od -N1 -An -vtu1)

Though that implies forking extra processes and run external executables (and if stdin is a terminal device, that will not put it out of icanon mode like read does).

Improve this answer
2
  • 1
    Wow. Using -n, rather than -N as the read builtin argument indeed nailed it. I had not tried -n because -N seemed indeed so much better suited to my purpose. I had tried -d '', though, without any effect, which is understandable given that the specs of -N clearly state that no delimiters are observed. Now, with -n I do have to care about delimiters, but that's easy (as between the lines you point out too). Thanks a lot! Commented Dec 29, 2020 at 16:33
  • I take it you suggest using printf```` instead of od``` because it's a builtin and therefore should be faster, as no new process needs to be forked? Commented Dec 29, 2020 at 16:40

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.