Faster For Loops Python

Question

so I have been working on a simple python program to iterate through each file name on the computer to check for the file type of an image or well any file type.

for i in drives:
    for root, dirs, files in os.walk(str(i.replace("\\","\\\\"))):
        #print(root,dirs)
        for file in files:
            if file.endswith(".jpeg"):
                pictures.append(file)

You see that I used a simple for loop group(i think they call this nesting I don't know) and I was wondering if there is someway to GPU accelerate this operation or can you point me in a direction so that I can improve the efficiency another way. I do note that there was a question on stack about searching filepaths for keyword but I am asking for a way to improve efficiency

The bottleneck here is the disk I/O, you can't do anything about it in software — BlackBear
– BlackBear, Commented Jun 15, 2016 at 17:21

Brian · Accepted Answer · 2016-06-15 17:25:32Z

1

There really isn't any significant improvement you can make due to the limitations of disk I/O (as the comments state). However, instead of using os.walk(...), you can use os.scandir(...) to get a little more speed out of it. See this link: https://www.python.org/dev/peps/pep-0471/

answered Jun 15, 2016 at 17:25

Brian

1,67512 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

waudy Over a year ago

Looked into the os.scandir module but i cannot seem to find any reference of the output of the function, is it similar to the walk function or?

Brian Over a year ago

It returns a generator object. It has a bit of assembly required (ie. you have to make your own recursive function if you want to loop through recursively), but it's significantly faster if you use it. It can be between 2 and 20 times faster, according to the documentation.

giullianomorroni · Accepted Answer · 2016-06-15 17:34:01Z

0

you can use threads and disparate one thread by driver at you first line...then you will get there faster

q = Queue()
for i in drives:
    t = Thread(target=your_method, args=(q,pictures))
    t.daemon = True
    t.start()
q.join()

hope could helped

answered Jun 15, 2016 at 17:34

giullianomorroni

295 bronze badges

1 Comment

Brian Over a year ago

Using this method, he would need to put a mutex on pictures otherwise, you've created a race condition

Collectives™ on Stack Overflow

Faster For Loops Python

2 Answers 2

2 Comments

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Related