Python selenium decode iso-8859-1 to UTF-8

Question

Im using selenium to extract data from a web page. Im trying to write this data into a file, but i have some trouble doing so, when i write special char like 'é' it create unreadable char in my file(Ã©). The website im getting the page from is encoded in iso-8859-1 and im using python 2.7.

browser = webdriver.Firefox()  
browser.get(URL_SITE_ENCODED_IN_iso-8859-1)
html = browser.page_source.decode('iso-8859-1') //error

From what i understood i have to decode the page from iso-8859-1 then it will encode it in utf-8, but when i try to an error is raised : UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 246: ordinal not in range(128)

Please provide the code that generates error, as described by: stackoverflow.com/help/mcve — Alastair McCormack
– Alastair McCormack, Commented Dec 17, 2015 at 17:37

Alastair McCormack · Accepted Answer · 2019-10-04 09:01:29Z

4

It's probably because browser.page_source.decode is already decoded Unicode. Check with:

>>> type(browser.page_source.decode)
<type 'unicode'>

When you write this to a file, you need to convert it to an appropriate encoding. In Python 2.x, use io module to create an automatic encoding file wrapper. Try:

browser = webdriver.Firefox()  
browser.get(anysite)

with io.open("myoutfile.txt", "w", encoding="utf-8") as my_file:
    my_file.write(browser.page_source)

edited Oct 4, 2019 at 9:01

answered Dec 18, 2015 at 8:54

Alastair McCormack

28k8 gold badges81 silver badges106 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python selenium decode iso-8859-1 to UTF-8

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related