utf-8 encoded String to Buffer node.js

Question

On nodejs express server #1, I receive binary stream data from another nodejs Server #2 in HTTP response. I want encode this data to base64. I have problems with encoding. I did it as shown below.

let result = await axios.post(firmwareDownloadURL, {
        id: 'firmware1'
      }, {
        headers: {
          'Content-Type': 'application/json',
        }
      });
let buff1 = new Buffer.from(result.data);
let base64Firmware1 = buff1.toString('base64');

The value of buff1 is not correct and therefore base64Firmware1 is also wrong. I compared it by reading the firmware file from my system with fs.

let buff2 = fs.readFileSync('./f1.bin');
let base64Firmware2 = buff3.toString('base64');

The buff1 and buff2 do not match.

buff <Buffer ef bf bd ef bf bd ef bf bd 00 01 ef bf bd 48 ef bf bd ef bf bd 01 6d 02 08 22 ef bf bd ef bf bd 0a ef bf bd ef bf bd 6e 6d 02 08 ef bf bd 0a ef bf bd ... 612 more bytes>

buff2 <Buffer a7 a3 fe 00 01 8d 48 a7 a6 01 6d 02 08 22 a7 a3 0a a7 a6 6e 6d 02 08 f1 a7 a3 0a a7 a6 f7 6d 02 08 b4 a7 a3 32 a7 a6 14 6e 02 08 39 a7 a3 06 a7 a6 56 ... 307 more bytes>

Interestingly, when I convert buff2 to string and compare with result.data they match.

if (buff2.toString() === result.data && buff2.toString().length ==  result.data.length) {
  console.log('equal');
}

it prints equal on console. Please help me identify, what am I missing?

Answer 1

Your observation that the data matches when stringified is correct, as that is exactly what Axios does under the hood. The problem is that Axios will, by default, stringify the response data. On Node.js, this means it calls Buffer.toString('utf8') which produces what you see. As a result, the bytes are interpreted as UTF-8, which turns invalid code units into the U+FFFD REPLACEMENT CHARACTER Unicode code point, as specified in the documentation for Buffer.toString():

If encoding is 'utf8' and a byte sequence in the input is not valid UTF-8, then each invalid byte is replaced with the replacement character U+FFFD.

The U+FFFD code point is encoded in UTF-8 as the code units EF BF BD, which you see in the data.

Turning the already stringified buffer into the original buffer after the fact is not possible, as all non-UTF8 code units are turned into U+FFFD, which loses information. To fix the encoding issue, you can use the responseType option in Axios to specify that you want the raw buffer instead:

let res = await axios.post(
  firmwareDownloadURL,
  {
    id: "firmware1",
  },
  {
    headers: {
      "Content-Type": "application/json",
    },
    responseType: "arraybuffer",
  }
);
let buff1 = res.data;
let base64Firmware1 = buff1.toString("base64");

Valid options for the responseType are 'arraybuffer', 'document', 'json', 'text', 'stream', and when using Axios in a browser environment, 'blob'.

---

As a side note, if you're interested to learn more about what exactly an Unicode "code point" is, I'd suggest giving this article a read.