Python encoding issue

Question

i m working with some python script, got a raw string with UTF8 encoding. first of all i decoded it to utf8 then some processing is done and at the end i encode it back to utf8 and inserted to DB(mysql) but chars in DB are not presented in real format.

str = '<term>Beiträge</term>'
str = str.decode('utf8')
...
...
...
str = str.encode('utf8')

after that string is found in txt file in its real form but in MYSQL_DB, i found it like this

 <term>"BeitrÃ¤ge</term>

any idea why this happened? :-(

str is not a good variable name. It hides the function str(). Also, use different variable names for different types of a variable. — eumiro
– eumiro, Commented May 17, 2011 at 13:11

marr75 · Accepted Answer · 2011-05-17 13:22:24Z

1

Assuming you are using the MySQLdb library, you need to create connections using the keyword arguments:

use_unicode If True, text-like columns are returned as unicode objects using the connection's character set. Otherwise, text-like columns are returned as strings. columns are returned as normal strings. Unicode objects will always be encoded to the connection's character set regardless of this setting.

&

charset If supplied, the connection character set will be changed to this character set (MySQL-4.1 and newer). This implies use_unicode=True.

You should also check the encoding of your db tables.

answered May 17, 2011 at 13:22

marr75

5,7251 gold badge29 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

shahzad Over a year ago

Connection's char set is utf8, and i m using MySQLdb

marr75 Over a year ago

try utf8_bin (whatever the utf8 binary encoding is called in mysql, I didn't look it up).

Halberdier · Accepted Answer · 2011-05-17 14:48:18Z

0

To make a string a Unicode string you should use the stringprefix 'u'. See also here http://docs.python.org/reference/lexical_analysis.html#literals

Maybe your example works by just adding the prefix in the initial assignment.

answered May 17, 2011 at 14:48

Halberdier

1,2241 gold badge12 silver badges15 bronze badges

Collectives™ on Stack Overflow

Python encoding issue

2 Answers 2

2 Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Related