uniq -i is does not ignore case-sensitive in non-Ascii characters

Question

I have a txt: a.txt

A
a
B
b
Ş
ş

I try this command and I get false output:

$ uniq -ic a.txt 
      2     A
      2     B
      1     Ş
      1     ş

How can I solve the non-ascii character problem with uniq?

here is my full-code:

function show_authors() {
    id=0;
sqlite3 $db_file "SELECT author FROM books;" > /tmp/.list.txt
sort /tmp/.list.txt | uniq -ic > /tmp/.listed.txt
while IFS=" " read -r count author
do
cat <<EOT
<a href="#$id">$author</a> ($count), 
EOT
id=$(($id + 1))
done < /tmp/.listed.txt
}

cat <<EOT
<div id="author">
$(show_authors)
</div>
EOT

My code is correctly working in shell but not correctly working in a CGI-Bash subshell.

Answer 1

You might need to change the locale (if you haven't already), at least for that command. For a en_US locale, Ş and ş are not related.

LC_ALL=tr_TR will bring in a locale where Ş is the uppercase of ş and it will be considered correctly.

But the command might still not work if you are using UTF-8. For it to work you need to go back to a single byte encoding that uniq can handle and then go back to UTF-8:

So, if this does not do it:

$ LC_ALL=tr_TR uniq -ic a.txt

You can try:

$ LC_ALL=tr_TR iconv -f UTF-8 -t ISO-8859-3 < a.txt | tr '[:upper:]' '[:lower:]' | uniq -c | iconv -f ISO-8859-3 -t UTF-8

The command line goes from UTF-8 (multibyte) to ISO-8859-3 (single byte), then changes everything to uppercase, calls uniq and then moves back to UTF-8.

I know there are other languages and locales using Ş, but I had to choose one them to write the answer. Yours might be different.

Answer 2

How about something like:
cat file.txt | iconv | uniq -i

with the iconv unspecified & before 'sort' or 'uniq'