How do I properly convert the file to UTF-16LE encoding without strange characters appearing in the file?

Question

I'm having some peculiarities with the dictionary file of .dsl format I'm trying to convert. It's essentially a text file with the dictionary pairs. The dictionary software I use is GoldenDict. It requires UTF-16 dictionaries so they render properly.

All the dictionaries I have are UTF-16LE format. There is one standing out however. It has iso-8859-1 encoding. An entry looks like this when I open it with vim:

abandonarse
    [m2][c crimson][b]Sinónimos[/b][/c][/m]
    [m2][i][c green]verbo[/c][/i][/m]
    [m1][trn][b]desanimarse:[/b] <<desanimarse>>, <<abatirse>>, <<tumbarse>>, <<plegarse>>, <<entregarse>>, <<desligarse>>[/trn][/m]

I have to convert it to UTF-16LE because Goldendict renders some Cyrillic characters instead of Spanish accented characters. Then I try:

iconv -f iso-8859-1 -t utf-16le dictionary.dsl -o test.dsl

The new test.dsl dictionary is rendered correctly by Goldendict, however I can see some peculiar things I would love to get rid of. First is that the just converted file's encoding is not recognized as it usually is with the other dictionaries:

    aleksandr@desktop:~/windoc/Dic/Es extra/dictionary.dsl> file dictionary.dsl
dictionary: data

When I open the file test.dsl with vim every character inside has ^@ added to it. Here is the example of the same entry:

    ^@<^@<^@e^@n^@t^@r^@e^@g^@a^@r^@s^@e^@>^@>^@,^@ ^@<^@<^@d^@e^@s^@l^@i^@g^@a^@r^@s^@e^@>^@>^@[^@/^@t^@r^@n^@]^@[^@/^@m^@]^@
^@      ^@[^@m^@2^@]^@[^@c^@ ^@c^@r^@i^@m^@s^@o^@n^@]^@[^@b^@]^@A^@n^@t^@ó^@n^@i^@m^@o^@s^@[^@/^@b^@]^@[^@/^@c^@]^@[^@/^@m^@]^@
^@      ^@[^@m^@2^@]^@[^@i^@]^@[^@c^@ ^@g^@r^@e^@e^@n^@]^@v^@e^@r^@b^@o^@[^@/^@c^@]^@[^@/^@i^@]^@[^@/^@m^@]^@

I tried removing this characters in vim

%s/<Ctrl-V><Ctrl-J>//g

However, then I save the file, it has the encoding iso-8859-1 again. I would like to have this file to be show without ^@ characters, because I may need to edit some headings in the dictionary manually.

The real question here is How do I get VIM to correctly recognize my text file as UTF-16LE (or ucs2-le) when the file does not have a BOM?, iconv not adding BOMs when it is explicitly told the endinanness to use. — JdeBP
– JdeBP, Commented Sep 8, 2020 at 15:54
How do I get file to correctly recognize my text file as UTF-16LE when the file does not have a BOM? is really a separate question. — JdeBP
– JdeBP, Commented Sep 8, 2020 at 16:00

Chris Davies · Accepted Answer · 2020-09-09 16:46:04Z

Your vim hasn't recognised the encoding, and is showing the 16-bit characters as 8-bit characters. The ^@ markers represent the higher order 8-bits, which for common Latin characters will be zero valued.

You can type this after reading in the file to force recognition of UTF-16LE

:e ++enc=utf-16le

^{(Credit: StackOverflow)}

It seems that this would also work, but whether it's ideal is far from clear to me

vim -c 'e ++enc=utf-16le' dictionary.dsl

Finally, from your comment it seems that a BOM would be quite acceptable. You can't use iconv to add a BOM but you can add one yourself

(
    printf "%s" $'\xFF\xFE'
    iconv -f iso-8859-1 -t utf-16le dictionary.dsl
) > dictionary-utf16le.dsl

Confirm with

file dictionary-utf16le.dsl
dictionary-utf16le.dsl: Little-endian UTF-16 Unicode text

Stack Exchange Network

How do I properly convert the file to UTF-16LE encoding without strange characters appearing in the file?

1 Answer 1

You must log in to answer this question.

Hot Network Questions

How do I properly convert the file to UTF-16LE encoding without strange characters appearing in the file?

1 Answer 1

You must log in to answer this question.

Related

Hot Network Questions