Selenium/ChromeDriver Timeout Exception when Webscraping

Question

I have built a webscraper that scrapes urls. Its occassionally having an issue (doesn't occur all the time). I get a TimeOutException at this particular part of my code:

tries = 0
while tries<1000:
    try:  
        domain=wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "cite._Rm")))
        domain = domain.text
        break
        except StaleElementReferenceException:
            tries+=1
            self.browser.refresh()

Error:

File "", line 389, in findDomains domain=wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "cite._Rm"))) File "", line 78, in until raise TimeoutException(message, screen, stacktrace) TimeoutException: Message:

The code you show in your question won't run.

Louis
– Louis

2015-09-28 16:40:15 +00:00
Commented Sep 28, 2015 at 16:40 — Louis
– Louis, Commented Sep 28, 2015 at 16:40

alecxe · Accepted Answer · 2015-09-28 16:17:04Z

1

Since you have it wrapped in a while block with multiple attempts, just handle TimeoutException:

try:  
    domain=wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "cite._Rm")))
    domain = domain.text
    break
except (StaleElementReferenceException, TimeoutException):
    tries+=1
    self.browser.refresh()

Note that it's difficult to say more without being able to reproduce the problem, or at least having a link to the target webpage that have occasional issues you are describing.

answered Sep 28, 2015 at 16:17

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

dmr · Accepted Answer · 2015-09-28 16:57:01Z

0

You get timeout exception because element was not found during timeout, try to increase it or refresh page like in a code below

tries = 0
while tries<1000:
    try:  
        domain=WebdriverWait(self.browser, timeout=10).until(
             EC.presence_of_element_located((By.CSS_SELECTOR, "cite._Rm"))
        )
        domain_text = domain.text
        break
    except TimeoutException:
        tries+=1
        self.browser.refresh()

answered Sep 28, 2015 at 16:57

dmr

5562 silver badges9 bronze badges

Collectives™ on Stack Overflow

Selenium/ChromeDriver Timeout Exception when Webscraping

2 Answers 2

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Related