Why does converting from base64 and back garble the end of this string?

Question

Command:

echo "HelloWorld==" | base64 -d | base64

Output:

HelloWorlQ==

Why is my d now a Q?

Edit:

I am not trying to start with arbitrary data and base64 encode it. My intention is to start with Base64 and end with Base64, having only produced a binary value in the interim.

Edit 2:

I have noticed that it does not happen if the input string has is a multiple of four characters, so I think it is an interaction with the padding somehow:

❯ echo 'abcdefghij==' | base64 -d | base64
abcdefghig==

❯ echo 'abcdefgh' | base64 -d | base64 
abcdefgh

Edit 3

Removed confusing mention of the -i flag, which turned out to have nothing to do with my problem.

Answer 1

HelloWorld== contains information which cannot be decoded, and isn't technically valid Base64 since they should in general be 0 padded. The extra 1s this contains will be ignored and lost when you echo "HelloWorld==" | base64 -d.

To explain...

Base64 Works with groups of 4 characters. Each character represents 6 bits, so each group of 4 decodes into 3 bytes (8 bits per byte). The only exception is the last 4 characters which will vary depending on the number of = signs. Base64 strings will always be divisible by 4.

• 0 decodes to 3 bytes
• 1 = decodes to 2 bytes
• 2 == decodes to 1 byte

In your example Hell and oWor are both valid. But ld== isn't. To understand why see this lookup table: https://en.wikipedia.org/wiki/Base64

ld== should decode to only one byte because it has two = at the end. But ld decodes to: 100101 011101. A byte is only eight 8 bits. So when you decode your string with base64 -d, only 100101 01 will be converted into a byte and the end 1101 will be ignored completely.

Any base 64 string ending in == must only use the first two bits of the last character. That is the only valid endings with == are Q== A== w== g==

Answer 2

Yes, it's an interaction with the padding.

Let's look at your actual encoded data by decoding it, and (since it's not an ASCII string) convert it into binary:

$ base64 -d <<<'HelloWorld==' | xxd -b
00000000: 00011101 11101001 01100101 10100001 01101010 00101011  ..e.j+
00000006: 10010101                                               .

This is the data that HelloWorld== is the base64 encoding of. Philip Couling explains the intricacies of the decoding of the final ld== part and that, in a way, only a third of the data encoded by the d is actually used when decoding the data. Below I'm showing where the Q comes from when you re-encode the data.

Let's repeat that binary:

00011101 11101001 01100101 10100001 01101010 00101011 10010101

In groups of six bits (which is what the base64 encoder will use):

000111 011110 100101 100101 101000 010110 101000 101011 100101 01

This padded with four zero bits at the end to make 10 complete 6-bit codes:

000111 011110 100101 100101 101000 010110 101000 101011 100101 010000

The 010000 is the Q you see when you re-encode the data (see the base64 table of codes).

Answer 3

The piping is not in order. You should encode first, before you decode.

$ echo "Hello World!==" | base64 | base64 -id
Hello World!==

You were decoding an invalid base64 encode format.