0

I want to get results of this query from Google (html links).

The code runs, but it seems I'm only getting results from the first page(Html links).

What is wrong with my loop how I can fix it to get the links for the (4 pages)? for i in range(counter,4)...

for i in range(counter,4):
    page_no = driver.find_elements_by_xpath("//table[@class='AaVjTc']/tbody/tr/td/a")   
    page_no[i].click()                                                                  
value = driver.find_elements_by_xpath('//div[@class="r"]/a')
headings = driver.find_elements_by_xpath('//div[@class = "g"]')  



for heading in headings:

        title = heading.find_elements_by_tag_name('h3')
        link = heading.find_element_by_css_selector('.yuRUbf>a').get_attribute("href")  
        links.append(link)    



df_da=pd.DataFrame()
df_da['Title']=links

df_da.to_csv('links.csv',encoding='utf-8-sig')      

1 Answer 1

1

You have to put the second loop inside the first one.

Updated code:-

for i in range(counter,4):
    page_no = driver.find_elements_by_xpath("//table[@class='AaVjTc']/tbody/tr/td/a")
    page_no[i].click()

    value = driver.find_elements_by_xpath('//div[@class="r"]/a')
    headings = driver.find_elements_by_xpath('//div[@class = "g"]')  
    for heading in headings:
        title = heading.find_elements_by_tag_name('h3')
        link = heading.find_element_by_css_selector('.yuRUbf>a').get_attribute("href")  
        links.append(link)    

df_da=pd.DataFrame()
df_da['Title']=links

df_da.to_csv('links.csv',encoding='utf-8-sig') 
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.