Selecting an option from an optgroup with selenium and python

Question

I would like to choose a value from this optgroup which should then give me a dropdown of links.

<div class="searchbar">
    <select id="q" multiple="" tabindex="-1" class="select2-hidden-accessible" aria-hidden="true">
    <option></option>
    <option class="q-all-text" value="al:all">Search all text</option>

    <optgroup label="Business Type">
        <option value="bt:Buyer">Buyer</option>
        <option value="bt:Farmer/Rancher">Farmer/Rancher</option>
        <option value="bt:Farmers Market">Farmers Market</option>
        <option value="bt:Fishery">Fishery</option>
        <option value="bt:Food Bank">Food Bank</option>
    </optgroup>

Here is my code so far:

driver = webdriver.Chrome('/path/to/chromedriver')
driver.get("https://foodmarketmaker.com/main/mmsearch")
iframe = driver.find_element_by_tag_name("iframe")
driver.switch_to.frame(iframe)

select  = Select(driver.find_element_by_class_name("select2-hidden- accessible"))
select.select_by_value("bt:Farmer/Rancher")
links = driver.find_elements_by_tag_name('a')
print(links)
for link in links:
    print(link.get_attribute('href'))

I either get an exception that the element does not exist, or when I access by index, i get an ElementNotVisibleException due to the aria-hidden attribute being true. Is there any way around this?

Either you can simulate some action like hovering or clicking somewhere which makes your elements visible and then select the element . If it get visible in some time then use explicit wait. or you intend to select that element any how then use JavaScript executor to change that aria-hidden to false and select the same — NarendraR
– NarendraR, Commented May 7, 2018 at 16:32

Andersson · Accepted Answer · 2018-05-07 17:03:58Z

select is not the node that you need to handle as it's not visible.

Try below code to get required output:

from selenium import webdriver as web
from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By


driver = web.Chrome('/path/to/chromedriver')
driver.get("https://foodmarketmaker.com/main/mmsearch")
iframe = driver.find_element_by_tag_name("iframe")
driver.switch_to.frame(iframe)

wait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//section[@id="search-right"]//input[@placeholder="start typing to search"]'))).click()
wait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//li[.="Bass"]'))).click()

links = wait(driver, 10).until(EC.presence_of_all_elements_located((By.XPATH, '//section[@id="search-results"]//a[.//*[name()="svg"]]')))

for link in links:
    print(link.get_attribute('href'))

P.S. You need to replace text value of li node in '//li[.="Bass"]' XPath with required option

I am trying to scroll through the results so that I can load all of the links, however it won't scroll. Is there a specific way of scrolling in an iframe? I tried using execute script and scrolling to bottom of page but it did nothing.

undetected Selenium · Accepted Answer · 2018-05-08 09:41:12Z

As per the HTML you have shared to print all the options you have to induce WebDriverWait to switch to the desired frame, then click on the Search Box for the options to be visible as follows :

Code Block :

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get("https://foodmarketmaker.com/main/mmsearch")
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[@src='https://search.foodmarketmaker.com']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@class='searchbar']//input[@class='select2-search__field' and @placeholder='start typing to search']"))).click()
links = driver.find_elements_by_xpath("//li[@class='select2-results__option' and @aria-label='Business Type']/strong//following::ul[1]//li")
for link in links:
    print(link.get_attribute('innerHTML'))

Console Output :

Buyer
Buyer: Baked Goods
Buyer: Beverages
Buyer: Dairy
Buyer: Fish &amp; Seafood
Buyer: Fruit &amp; Nuts
Buyer: Grains &amp; OilSeed
Buyer: Herbs
Buyer: Meat &amp; Poultry
Buyer: Miscellaneous
Buyer: Specialty Products
Buyer: Sugar/Confectionary &amp; Snacks
Buyer: Vegetables
Eating &amp; Drinking Place
Eating &amp; Drinking Place: Cafeteria &amp; Buffet
Eating &amp; Drinking Place: Caterer
Eating &amp; Drinking Place: Drinking Place
Eating &amp; Drinking Place: Food Service Contractor
Eating &amp; Drinking Place: Mobile Food Service
Eating &amp; Drinking Place: Restaurant
Eating &amp; Drinking Place: Snack &amp; Nonalcoholic Beverage Bar
Farmer/Rancher
Farmer/Rancher: Dairy
Farmer/Rancher: Forage
Farmer/Rancher: Fruits &amp; Nuts
Farmer/Rancher: Grains
Farmer/Rancher: Herbs
Farmer/Rancher: Livestock Production (Beef Cattle)
Farmer/Rancher: Meat &amp; Poultry
Farmer/Rancher: Specialty Products
Farmer/Rancher: Vegetables
Farmers Market
Fishery
Fishery: Fish/Shellfish/Seafood
Food Bank
Food Bank: Food Bank
Food Hub
Food Retailer
Food Retailer: Baked Goods Store
Food Retailer: Beer, Wine &amp; Liquor Store
Food Retailer: Candy, Nut &amp; Confectionery
Food Retailer: Fish Market
Food Retailer: Fruit &amp; Vegetable Market
Food Retailer: Grocery/Convenience Store
Food Retailer: Health Supplement Store
Food Retailer: Meat &amp; Poultry Market
Food Retailer: Other Direct Selling Establishments
Food Retailer: Other Specialty Food Store
General Member
Processor/Packing Shed
Processor/Packing Shed: Animal Food Manufacturing
Processor/Packing Shed: Bakery Products
Processor/Packing Shed: Beverages
Processor/Packing Shed: Dairy Products
Processor/Packing Shed: Fish/Shellfish/Seafood Products
Processor/Packing Shed: Fruit &amp; Vegetable Products
Processor/Packing Shed: Grain &amp; Oilseed Milling
Processor/Packing Shed: Meat &amp; Poultry Products
Processor/Packing Shed: Miscellaneous
Processor/Packing Shed: Packing Shed
Processor/Packing Shed: Sugar and Confectionery Products
Tourism
Tourism: Agritourism
Tourism: Fishing Charter
Tourism: Hunting
Wholesaler
Wholesaler: Beer, Wine &amp; Alcoholic Beverages
Wholesaler: Confectionery &amp; Snacks
Wholesaler: Dairy Products
Wholesaler: Farm Products
Wholesaler: Fish/Shellfish/Seafood
Wholesaler: Fruit &amp; Vegetable
Wholesaler: General Grocery
Wholesaler: Meat &amp; Poultry
Wholesaler: Other Grocery
Wholesaler: Packaged Frozen Food
Winery
Winery: Blends or Generic Wine
Winery: Fortified Wine
Winery: Fruit Wine
Winery: Other Wine Products
Winery: Red Wine
Winery: Sparkling Wine
Winery: White Wine

Collectives™ on Stack Overflow

Selecting an option from an optgroup with selenium and python

2 Answers 2

1 Comment

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Linked

Related