This python selenium program does not iterate

Question

I would like to iterate the function using a while loop, but it iterates only once, and after that, the program just stands still.

from selenium import webdriver
from multiprocessing import Process


def browse(url):
    driver = webdriver.Chrome()
    driver.get(url)
    print(driver.page_source)
    driver.__exit__()

Pros = []
urls = open('urls.txt')

if __name__ == '__main__':
    while True:
        for url_item in urls:
            print(url_item)
            p = Process(target=browse, args=(url_item,))
            Pros.append(p)
            p.start()
        for t in Pros:
            t.join()

You are not reading anything from the urls.txt file you're opening, so I don't think the variable urls gets populated properly. — Nastor
– Nastor, Commented Jul 21, 2021 at 9:40
You're appending to Pros array in every iteration, but you're not clearing it before the next step. That might be the problem. Try setting it (Pros = []) inside the while loop. — MrJuicyBacon
– MrJuicyBacon, Commented Jul 21, 2021 at 9:43

Deyuan · Accepted Answer · 2021-07-21 10:00:40Z

1

The main problem relies on how the file's content was read as well as how the Process object has been started. It starting on a temporary Process object instead of the Process object in process list (Pros). In short, it is only a referencing issue and wrong way of utlizing File Object. Here is the working code.

from selenium import webdriver
from multiprocessing import Process


def browse(URL):
    driver = webdriver.Chrome()
    driver.get(URL)
    print(driver.page_source)
    driver.__exit__()

Pros = []
urls = open('urls.txt').readlines()

if __name__ == '__main__':
    while True:
        print("testing")
        for url_item in urls:
            print(url_item)
            p = Process(target=browse, args=(url_item,))
            Pros.append(p)
            Pros[-1].start()
        for t in Pros:
            t.join()
        Pros = []

edited Jul 21, 2021 at 10:00

answered Jul 21, 2021 at 9:42

Deyuan

561 silver badge6 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Praveen Kumar Over a year ago

It's not working. If I use the above code, the multithreading will not work as intended. It runs the code one by one instead of running parallelly.

Deyuan Over a year ago

I just noticed, i have edited the answer, you can try it out again.

Praveen Kumar Over a year ago

No, you can check from your side as well.

Deyuan Over a year ago

From my side, the process actually prints out the page_source fine. After edit, it prints all the page source simultaneously as well.

Deyuan Over a year ago

Can you elaborate more on what the error is about? Show the traceback from the console when the program is running.

|

Collectives™ on Stack Overflow

This python selenium program does not iterate

1 Answer 1

7 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Related