using Multi Thread for processing file using python3

Question

I am trying to process a file using the single core of my CPU. But I guess it is not sufficient to use a single core. Instead, if I had get an access to the multiple cores of my system then I can make the process run better and faster.

But unfortunately, I know to process a file using single core only. Here is what I did:

data = open('datafile','r',encoding='ascii',errors='ignore')
for line in data.readlines():
    splitted = line.lower().strip().split()
    check = process(splitted[0],splitted[1])
    if check == '':
         pass
data.close()

I want to know how I can use the complete capacity of the CPU for processing teh process() while taking the line separately and getting the output as desired? Even how I can avoid the deadlock state of the thread while processing as this can be dangerous for the process output.

Please share your view with me.

Answer 1

First of all: you need multiple processes to utilize multiple cores. Not threads. It's a limitation due to GIL.

Now here's an example of how you can implement it with multiprocessing.Pool:

from multiprocessing import Pool, cpu_count

def process(arg1, arg2):
    ...

workers_count = 2*cpu_count()+1  # or whatever you need
pool = Pool(processes=workers_count)

with open('datafile','r',encoding='ascii',errors='ignore') as fo:
    buffer = []
    for line in fo:
        splitted = line.lower().strip().split()
        buffer.append((splitted[0], splitted[1]))
        if len(buffer) == workers_count:
            results = pool.map(process, buffer)
            buffer = []
            # do something with results
    if buffer:
        results = pool.map(process, buffer)
        # do something with results again

So what it does it reads the file line by line and once it gathers enough data it sends it to a multiprocess pool and waits for parallel processing. Note that unless you have SSD then running disk io in parallel will only deteriorate the performance (also it would not be trivial to parallelize line-by-line reads).

What you have to be aware though is that since multiple processes are used you cannot share memory between them, i.e. the process function should not read/write to global variables.