Selenium Explicit waits not working properly in Python

Question

I try to crawl the head banner carousel for practice in python.org. I use WebDriverWait to wait elements visible after clicking the trigger but not working properly. Here is my code.

# ChromeDriver
driver.get("https://www.python.org/")

hBannerNav = driver.find_elements_by_xpath(
    '//ol[@class="flex-control-nav flex-control-paging"]/li/a')

for i in range(len(hBannerNav)):
    print(hBannerNav[i].text)
    hBannerNav[i].click()
    try: 
        self.wait.until(EC.visibility_of_element_located(
            (By.XPATH, '//ul[@class="slides menu"]/li[{}]'.format(i + 1))))
        h1 = driver.find_element_by_xpath(
            '//ul[@class="slides menu"]/li[{}]/div/h1'.format(i + 1))
        print(h1.text) 

        # if add a sleep the crawler will work properly and smoothly,
        # but I want to use WebDriverWait only.
        # sleep(1) 

    except Exception as e:
        print('error', e)

Here are logs:

# without sleep
1
Functions Defined
2
Compound Data Types
3
error Message:

4
Quick & Easy to Learn
5
All the Flow You’d Expect # wait for a long time but still crawl it

# use sleep
1
Functions Defined
2
Compound Data Types
3
Intuitive Interpretation
4
Quick & Easy to Learn
5
All the Flow You’d Expect

Use presence_of_all_elements_located

# the results by using 
h1 = self.wait.until(EC.presence_of_all_elements_located(
    (By.XPATH, '//ul[@class="slides menu"]/li[{}]/div/h1'.format(i + 1))))[0]

1
Functions Defined
2
Compound Data Types
3

4

5

I used presence_of_all_elements_located in my project and it worked fine for me. You might want to try it. — kerwei
– kerwei, Commented Jan 9, 2019 at 3:58
you can try adding a implicit wait, which will poll the dom to find the element till the time given. To do this once the driver is created add driver.implicitly_wait(time to wait in seconds) — SnR
– SnR, Commented Jan 9, 2019 at 4:39
It would be helpful to see the actual error. Try removing the try/except so you can see the full error. — Mike B
– Mike B, Commented Jan 9, 2019 at 6:03
@kerwei It's not work for me and the results is updated on my post. — ovo
– ovo, Commented Jan 9, 2019 at 6:18

Mike B · Accepted Answer · 2019-01-09 07:47:22Z

I loaded up your code and gave it a whirl. You're essentially doing it right; the problem is this slides menu element is a little weird. When switching slides, there's a fade effect that takes a fraction of a second. During this time, the li/h1 of interest is considered "visible" BUT the slide buttons are unresponsive! Try clicking them yourself during the fade effect. Nothing happens.

I often run into these small, unexpected timing problems when using Selenium, and the solution varies from case to case.

Normally I'd check if the buttons are clickable, but clickability is not the issue here.

Here, I got it to work by waiting for invisibility of the previous slide:

for i in range(len(hBannerNav)):
    print(hBannerNav[i].text)
    hBannerNav[i].click()
    # We don't wait if i == 0 because in that case, there's no previous slide
    if i > 0:
        WebDriverWait(driver, 3).until(
            EC.invisibility_of_element_located((By.XPATH, '//ul[@class="slides menu"]/li[{}]'.format(i))))
    h1 = driver.find_element_by_xpath(
        '//ul[@class="slides menu"]/li[{}]/div/h1'.format(i + 1))
    print(h1.text)

There may be other, potentially better ways to get around this timing problem, but hopefully this is enough to get you unstuck.

Todor Minakov · Accepted Answer · 2019-01-09 05:45:13Z

2

Add a second wait - for the /div/h1:

h1 = self.wait.until(EC.visibility_of_element_located(
        (By.XPATH, '//ul[@class="slides menu"]/li[{}]/div/h1'.format(i + 1))))

It m8ght be the case it's put in the html a bit later than its parent.

The WebdriverWait.until() does return the matched element, so h1 will have the desired value.

answered Jan 9, 2019 at 5:45

Todor Minakov

20.2k3 gold badges61 silver badges63 bronze badges

2 Comments

Mike B Over a year ago

Looks good, though I don't think you need the first wait at all anymore.

ovo Over a year ago

It didn't work whether I kept both or left the second only. The results are the same.

Collectives™ on Stack Overflow

Selenium Explicit waits not working properly in Python

2 Answers 2

Comments

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Related