As a C programmer, I was surprised to see that wc -c (which count the number of bytes), and wc -m (which counts the number of characters) output very different results for a long, text file of mine. I had always been told that sizeof(char) is 1 byte.
qdii@nomada ~/Documents $ wc -c sentences.csv
102990983 sentences.csv
qdii@nomada ~/Documents $ wc -m sentences.csv
89023123 sentences.csv
Any explanation?
The char type in C is one byte, but it's intended for ASCII characters; there are variable-width encodings like UTF-8 that can take up many bytes per character. wc uses the mbrtowc(3) function to decode multibyte sequences, depending on the locale set by the LC_CTYPE environment variable. If you set the locale properly, you should get the same result for all cases. For example:
qdii@nomada ~/Documents $ LC_CTYPE="C" wc -m sentences.csv
102990983 sentences.csv
At a guess,
Your locale uses UTF-8 encoding, and
About 10% of your file consists of characters which require more than one octet to encode into UTF-8.
By the way, from man wc:
-c, --bytes
print the byte counts
-m, --chars
print the character counts
Minimal example
Consider the Unicode character "é" known as "LATIN SMALL LETTER E WITH ACUTE", which is an 'e' with an acute accent used in several European languages.
Its UTF-8 encoding is two bytes long "0xc3 0xa9".
With that in mind we see:
printf '\xc3\xa9' | LC_CTYPE=en_US.UTF-8 wc -c
printf '\xc3\xa9' | LC_CTYPE=en_US.UTF-8 wc -m
printf '\xc3\xa9' | LC_CTYPE=C wc -c
printf '\xc3\xa9' | LC_CTYPE=C wc -m
outputs:
2
1
2
2
So we understand as explained at https://unix.stackexchange.com/a/51948/32558 that to get the correct UTF-8 count we need both wc -m and LC_CTYPE=en_US.UTF-8.
In my system, the outcome is the same if I use the input method to type a literal é:
printf 'é' | LC_CTYPE=en_US.UTF-8 wc -c
Tested on Ubuntu 21.04.
