Scraping Multiple urls selenium

Question

I'm new to coding but i wrote this code that scraps the page fine but i want to scrape multiple of these urls like 200 how do i do that?

from selenium import webdriver

chrome_path = r"C:\Users\lenovo\Downloads\chromedriver_win32 (5)\chromedriver.exe"

driver = webdriver.Chrome(chrome_path)

driver.get("https://www.kijijiautos.ca/vip/22442312")

driver.find_element_by_xpath('//div[@class="b1yLWE b3zFtQ"]').text

btn = driver.find_element_by_xpath('//button[@class="g1zAe-"]')

btn.click()

driver.find_elements_by_xpath('//span[@class="A2jAym q2jAym"]').text

driver.find_element_by_xpath('//div[@class="b1yLWE b1zAe-"]').text

print(driver.current_url)

I'm really a noob can you please show me what loop thanks man — Alian Nadeem
– Alian Nadeem, Commented Sep 25, 2021 at 15:13
Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. — Community
– Community Bot, Commented Oct 3, 2021 at 20:07

Danish Bansal · Accepted Answer · 2021-09-25 15:34:32Z

Something like below

from selenium import webdriver

chrome_path = r"C:\Users\lenovo\Downloads\chromedriver_win32 (5)\chromedriver.exe"

driver = webdriver.Chrome(chrome_path)


def get_scarping(link):
    driver.get(link)
    driver.find_element_by_xpath('//div[@class="b1yLWE b3zFtQ"]').text
    btn = driver.find_element_by_xpath('//button[@class="g1zAe-"]')
    btn.click()
    driver.find_elements_by_xpath('//span[@class="A2jAym q2jAym"]').text
    driver.find_element_by_xpath('//div[@class="b1yLWE b1zAe-"]').text
    print(driver.current_url)
    return driver.current_url 


links = ["https://www.kijijiautos.ca/vip/22442312", "other_urls"]
scrapings = []
for link in links:
    scrapings.append(get_scarping(link))

Ruslan Shamsiev · Accepted Answer · 2021-09-25 15:46:11Z

Just add for loop

from selenium import webdriver
chrome_path = r"C:\Users\lenovo\Downloads\chromedriver_win32 (5)\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
for x in range(200):
    driver.get("https://www.kijijiautos.ca/vip/22442312")
    driver.find_element_by_xpath('//div[@class="b1yLWE b3zFtQ"]').text
    btn = driver.find_element_by_xpath('//button[@class="g1zAe-"]')
    btn.click()
    driver.find_elements_by_xpath('//span[@class="A2jAym q2jAym"]').text
    driver.find_element_by_xpath('//div[@class="b1yLWE b1zAe-"]').text
    print(driver.current_url)

Collectives™ on Stack Overflow

Scraping Multiple urls selenium

2 Answers 2

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Related