Append N hexadecimal numbers to a binary file with Bash

Question

I have a Bash script that appends bytes written as hexadecimal values. I use echo to write the bytes and it works

hex="1F"
byte="\x$hex"
echo -en $byte >> "output.bin"

At some point I need to pad the file with a byte that could be anything from 00 to FF. I want to specify the byte as a hex value and the total of repetitions.

I tried doing this with a for loop but it just takes too long, especially since I need to add something like 65535 bytes sometimes.

byte="00"
total=65515
for (( i=1; i<=total; i++ ));
do
    echo -en "\x$byte" >> "output.bin"
done

I am looking for a more performant way that does not use a for loop. At the moment I am stuck with something between printf and echo but instead of writing the values as binary it writes them as text

result=$(eval printf "\x$byte%0.s" {1..$total})
echo -en $result >> "output.bin"

The result in the output file is 78 30 30, which is the "x00" text, instead of 00. If I directly use echo -en "\x00" >> "output.bin", it does write one byte holding the 00 value and not the text "x00". This is where I don't know how to proceed in order to make it write the actual binary value

Answer 1

You're running that command substitution with eval, so you get the quotes and escapes processed twice.

Assuming byte=41 total=3, what eval sees is the arguments printf, \x41%0.s, 1, 2, 3. It concatenates them to printf \x41%0.s 1 2 3, which after escape processing runs printf with the four args x41%0.s, 1, 2, 3. (It should print x41x41x41, here.)

You could fix that by adding the correct amount of backslashes, but you probably also want to not use echo -e, since it'd only mess up the result if you were printing backslashes.

---

Anyway,

Unless you want to do it the hard way on purpose, it'd be easier to use some other tools than just the shell. E.g. you could use tr to get the byte you want, but it doesn't support hex escapes, only octal with \000, so there's the trouble of converting.

$ byte=41 count=15
$ printf "%${count}s" "" | tr ' ' "\\$(printf %o $((0x$byte)) )"
AAAAAAAAAAAAAAA

Or, you could use Perl:

$ perl -e 'print chr(hex($ARGV[0])) x $ARGV[1]' 41 15
AAAAAAAAAAAAAAA

Answer 2

Note that, if it's about padding with 0 bytes to align to some boundary, you can use dd with conv=sync for that.

$ echo test | dd bs=128 iflag=fullblock conv=sync | hexdump -Cv
0+1 records in
1+0 records out
128 bytes copied, 5.9459e-05 s, 2.2 MB/s
00000000  74 65 73 74 0a 00 00 00  00 00 00 00 00 00 00 00  |test............|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080

(iflag=fullblock is a GNU extension¹, but is only needed for pipe inputs or more generally when there could be short reads, not for regular files; with GNU dd, you can also add status=none to remove the transfer information).

Or you can use truncate on an already existing file to a larger size, for the end to be filled with NULs (not taking any more space on disk):

$ echo test > file
$ truncate -s 1M file
$ hexdump -C file
00000000  74 65 73 74 0a 00 00 00  00 00 00 00 00 00 00 00  |test............|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00100000

Or the same with fallocate -l in place of truncate -s to allocate the disk space.

Note that among the shells that have copied zsh's {x..y}, bash is the only one where {1..$n} doesn't work, so here, you could also switch to one of those shells (zsh, ksh93, yash -o brace-expand).

In zsh:

$ hex=00; n=100; printf "\x$hex%.0s" {1..$n} | hexdump -Cv
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  00 00 00 00                                       |....|
00000064

zsh also has padding operators and (contrary to bash) can store arbitrary bytes in its variable:

$ string=test n=128 pad=$'\0'; printf %s ${(pr[n][$pad])string} | hexdump -Cv
00000000  74 65 73 74 00 00 00 00  00 00 00 00 00 00 00 00  |test............|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080

(beware, padding is based on number of characters, use set +o multibyte for it to be byte-wise in locales where characters can be made of more than one byte).

---

^{¹ Since added to the 2024 edition of the POSIX standard, but currently with an unfortunate typo as they have iflags=fullblock instead of iflag=fullblock, typo copied from the request to add that GNU extension to the standard.}

Answer 3

GNU/awk has a function to convert hex expressions to a character. It will also create a long string (using a length option to sprintf), and then globally substitute your character.

#! /bin/bash

Awk='{
    #.. Make a long empty string.
    str = sprintf ("%*s", $1, "");
    #.. Make a single char from the hex value.
    hex = sprintf ("%c", strtonum ($2));
    #.. Fill the string with that character.
    gsub (".", hex, str);
    #.. Output the string.
    printf ("%s", str);
}'
    time echo 70552 0x14 | awk "${Awk}" >> doHex.big
    ls -l doHex.big
    od -Ad -t x1a doHex.big

It makes 70 KB in about 1/20th of a second.

$ ./doHex

real    0m0.053s
user    0m0.044s
sys 0m0.009s
-rw-rw-r-- 1 paul paul 70552 Sep  5 00:46 doHex.big
0000000  14  14  14  14  14  14  14  14  14  14  14  14  14  14  14  14
        dc4 dc4 dc4 dc4 dc4 dc4 dc4 dc4 dc4 dc4 dc4 dc4 dc4 dc4 dc4 dc4
*
0070544  14  14  14  14  14  14  14  14
        dc4 dc4 dc4 dc4 dc4 dc4 dc4 dc4
0070552
$