empty element when using selenium webdriver

Question

I am trying to locate the following element using selenium webdriver:

<div class="lv-product__details"><div class="lv-product__details-head"><span class="lv-product__details-sku">
            M40712
          </span> <div class="lv-product-add-to-wishlist"><button aria-label="Add to Wishlist" aria-disabled="false" tabindex="0" class="lv-icon-button lv-product-add-to-wishlist__button"><svg focusable="false" aria-hidden="true" class="lv-icon"><use xlink:href="/_nuxt/icons.svg#sprite-navigation-wishlist-off"></use></svg></button></div></div> <h1 class="lv-product__title">
          Pochette Accessoires
        </h1> <div class="lv-product-variations"><button class="lv-product-variation-selector list-label-l lv-product-variations__selector" aria-expanded="false"><span class="lv-product-variation-selector__title -text-is-medium">
    Material

I have tried:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

url = "https://en.louisvuitton.com/eng-nl/products/pochette-accessoires-monogram-005656"

options = Options()
options.headless = True
driver = webdriver.Chrome('path/to/chromedriver', chrome_options=options)
driver.get(url)

elem = driver.find_element_by_class_name("lv-product__details")

or via Xpath

elem = driver.find_element_by_xpath('//*[@id="__layout"]/div/div[2]/div[2]/div[1]/div[1]/div[2]')

but the elem is returned as an empty list. is there something that I am doing wrong / can do differently to access the contents of the website?

Did it work without headless? If it did check the page_source it might detect your a bot and then you'd have to change options to fix that like adding user agents and so on. — Arundeep Chohan
– Arundeep Chohan, Commented Mar 31, 2021 at 7:17
@ArundeepChohan You're right as soon as I set headless to True - the elem is returned as an empty string. However when False it is able to retrieve the element. — jojo_040
– jojo_040, Commented Mar 31, 2021 at 10:18

Herker · Accepted Answer · 2021-03-31 06:57:58Z

1

I think you have formated your XPath incorrect

Try this

driver.find_element_by_xpath('/div/div[2]/div[2]/div[1]/div[1]/div[2]/div[3]')

Or this

driver.find_element_by_class_name("lv-icon-button lv-product-add-to-wishlist__button")

And try to import time

import time

time.sleep(3) # To make sure everything loads before selenium starts to locate the element

answered Mar 31, 2021 at 6:57

Herker

6029 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Herker Over a year ago

I usually use driver.implicitly_wait(10) too, so try to have that as well

jojo_040 Over a year ago

Thanks for the answer - issue was going headless. Element is able to be retrieved when headless is set to False.

Arundeep Chohan · Accepted Answer · 2021-04-05 01:07:13Z

1

from fake_useragent import UserAgent
ua = UserAgent()
user_agent = ua.random
print(user_agent)
options.add_argument(f'user-agent={user_agent}')

All you had to do was add an user agent.

M40712
POCHETTE ACCESSOIRES
Material
Monogram Canvas
currently selected
600,00€
Always the epitome of iconic style, this interpretation of the Pochette Accessoires can accommodate a Zippy Wallet. In Monogram canvas, it easily carries all the daily necessities.
Detailed features
23.5 x 13.5 x 4 cm
(Length x Height x Width)
Natural cowhide leather trimmings
Zipper closure
Golden color metallic pieces
See More
PRODUCT CARE

answered Apr 5, 2021 at 1:07

Arundeep Chohan

9,9895 gold badges17 silver badges36 bronze badges

2 Comments

jojo_040 Over a year ago

Hi the fake user agent helped however still returned an empty element. I found that it finally worked after I implemented the following options in addition to the fake user agent:

jojo_040 Over a year ago

options.add_argument("--window-size=1920,1080"), options.add_argument('--ignore-certificate-errors'), options.add_argument('--allow-running-insecure-content'), options.add_argument("--disable-extensions"), options.add_argument("--proxy-server='direct://'"), options.add_argument("--proxy-bypass-list=*"), options.add_argument("--start-maximized"), options.add_argument('--disable-gpu'), options.add_argument('--disable-dev-shm-usage'), options.add_argument('--no-sandbox'),

Collectives™ on Stack Overflow

empty element when using selenium webdriver

2 Answers 2

I think you have formated your XPath incorrect

2 Comments

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

I think you have formated your XPath incorrect

2 Comments

2 Comments

Related