Scraping through multiple pages when url doesn't change

Question

I need to scrape through all the pages in the (link)[https://mahabocw.in/safety-kit-benefits-distribution/]. But the url doesn't change when I move to the next page. I tried to use selenium, but I am stuck as I don't know how to click the next page. Any help or suggestions would be really appreciated.

I have implemented the following code till now.

from selenium import webdriver
import time

url = "https://mahabocw.in/safety-kit-benefits-distribution/"
driver = webdriver.Chrome()
driver.get(url)

Below is the button element I need to click

<button type="button" class="ag-paging-button">Next</button>

Thanks a lot in advance. [1]: https://mahabocw.in/safety-kit-benefits-distribution/

It's because they use http request to refresh the page information. You have to look for mouse clicks events in selenium documentation or learn abouts requests library and reproduce the browser requests. — João Victor
– João Victor, Commented Jun 23, 2020 at 20:18
How could I use requests library here. Could you share some code. I am a noob in this field. — Suraj
– Suraj, Commented Jun 24, 2020 at 4:26

Lzypenguin · Accepted Answer · 2020-06-24 22:31:47Z

1

You need to tell selenium to click the next button. Add this to your code and see if it works.

next_button = '/html/body/div/div[6]/div/article/div/div/div/div/div[2]/div/div/div[2]/div/div[4]/span[2]/div[3]/button'
click_next = driver.find_element_by_xpath(next_button)
click_next.submit()

You might have to use click_next.click() instead of .submit() depending on the page. Also, to get the 'next_button' you just inspect elements on the page, find the item you want, and click copy as xpath.

edited Jun 24, 2020 at 22:31

answered Jun 23, 2020 at 22:05

Lzypenguin

9652 gold badges7 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Suraj Over a year ago

The code doesn't work. I receive the following error :

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"./ancestor-or-self::form"}

Suraj Over a year ago

Actually click_next seems to retrieve the element correctly, but the last line causes the above mentioned error. I changed it to click_next.click() and it works now. Thanks man.

Suraj Over a year ago

Thank you. And for anyone reading it afterwards, you can use a xpath finder extension in your browser to find the path as in the variable next_button

Lzypenguin Over a year ago

Sorry I am just now seeing this. I believe it depends on the item on the page for how you click it. When I was using this work on a project I had to use .submit() but i know on other pages .click() works also. Glad you got it working!

Suraj Over a year ago

Actually .click() was the one that worked for me. You could edit your answer. Thanks again.

Collectives™ on Stack Overflow

Scraping through multiple pages when url doesn't change

1 Answer 1

5 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Related