How to get binary representations of strings in Shell?

Question

I'm using openssl dgst -sha1 -binary to get hash values of my strings in binary format.

(I'm using -binary flag because my version of openssl adds "stdout" before each hash value on default output, and -binary helps to avoid it, therefore it's easier to store hash results in binary format for further processing (so I can just use xxd -p when I want hex values instead of manually cutting out that "stdout" from each string)

So, the binary output of openssl dgst -sha1 -binary for a "Hello!" string in Cygwin console will look like this: _▒▒"q▒%▒a▒▒▒▒.&C▒0N▒Q▒▒vH&8i

Now I create a new variable with this result and concatenate it with another variable, which value is not in binary format (i.e. "World"). So my new variable now looks like

_▒▒"q▒%▒a▒▒▒▒.&C▒0N▒Q▒▒vH&8iWorld

Then I generate another hash for this concatenated string and compare it to the one I got using default Java hashing libraries (MessageDigest). On this step, however, hashes obtained via shell and Java don't match (and I need to get exactly the same value as the one generated on Java side).

So I suppose that my "World" string should be in binary format as well to match my Java hash output (because as long as I generate hashes for concatenated binary values all hashes match). However, I don't know how to convert my "World" string to binary format in shell. Any ideas?

Answer 1

You can't store binary data (binary data generally refers to data with arbitrary byte values, not only byte values that form valid characters but is not special otherwise) in bash variables as bash doesn't support storing the 0 byte value in its variables (and remember you can't pass such strings in arguments to commands as those are NUL delimited strings).

You can in zsh though. Also remember that command substitution strips trailing newline characters (0xa bytes, maybe different on Cygwin), so it's probably better to use read here:

$ echo 323 | openssl dgst -sha1 -binary | hd
00000000  3a 8b 03 4a 5d 00 e9 07  b2 9e 0a 61 b3 54 db 45  |:..J]......a.T.E|
00000010  63 4b 37 b0                                       |cK7.|
00000014

See how that contains both a 0 byte and newline character (0xa)

$ echo 323 | openssl dgst -sha1 -binary | IFS= LC_ALL=C read -ru0 -k20 var &&
  var=${var}World
$ printf %s $var | hd
00000000  3a 8b 03 4a 5d 00 e9 07  b2 9e 0a 61 b3 54 db 45  |:..J]......a.T.E|
00000010  63 4b 37 b0 57 6f 72 6c  64                       |cK7.World|
00000019

Note again that you can only pass that variable to builtin commands (printf...).

Now, if all you want is hash it again, then it's just

(echo 323 | openssl dgst -sha1 -binary; printf %s World) |
  openssl dgst -sha1 -binary

no need for a variable.