Python Asyncio - Running the same task in parallel

Question

I'm trying to run this code in parallel:

availablePrefix = {"http://URL-to-somewehere.com": "true", "http://URL-to-somewehere-else.com": "true"}
def main():
        while True:
            prefixUrl = getFreePrefix() # Waits until new url is free
            sendRequest("https://stackoverflow.com/", prefixUrl) 

def getFreePrefix():
        while True:
            for prefix in self.availablePrefix.keys(): 
                if availablePrefix.get(prefix) == "true":
                    availablePrefix[prefix] = "false" # Can't be used for another request
                    return prefix 

async def sendRequest(self, prefix, suffix):
        url = prefix + "/" +  suffix
        async with aiohttp.ClientSession() as session:
            async with session.get(url) as resp:
                response = await resp.text()
                availablePrefix[prefix] = "true" # Can be used again
                return json.loads(response)

Basically, I'm trying to run the main() function in parallel. The main() function is stuck until getFreePrefix() returns a new prefix (URL to my server). With the help of this prefix we can access my server and start a request. If this Prefix is used, it is set to false, to indicate that it can't be used for another request right now (If request is completed, it is set to true again).

What I want to achieve is, that every time a new prefix is ready, a new request is run in parallel.

Thanks for helping!

Booboo · Accepted Answer · 2020-10-28 11:30:01Z

With your inconsistent use of self, I can't tell whether whether parts of your code is supposed to be part of a class or not. It also appears that your intention is for function main to run in an infinite loop and as soon as a key of availablePrefix has been processed, it is available for processing again. In your current, non-concurrent code, I believe that this could have been accomplished more simply as:

# simple list:
availablePrefix = ["http://URL-to-somewehere.com", "http://URL-to-somewehere-else.com"]

def main():
    while True:
        for prefixUrl in availablePrefix:
            sendRequest("https://stackoverflow.com/", prefixUrl)

And you get rid of method getFreePrefix and remove the code from sendRequest that updates the heretofore availablePrefix dictionary, which is now a list. The other improvement I would make is to have the aiohttp.ClientSession() instance created only once in main and passed as an argument to whomever needs it.

Moving on. To repeatedly process the prefixes concurrently, the simplest way I know is:

import asyncio
import aiohttp


availablePrefix = ["http://URL-to-somewehere.com", "http://URL-to-somewehere-else.com"]

async def main():
    # create the session instance once and pass it as an argument:
    async with aiohttp.ClientSession() as session:
        while True:
            tasks = {asyncio.create_task(sendRequest(session, "https://stackoverflow.com/", prefixUrl)) for prefixUrl in availablePrefix}
            for task in asyncio.as_completed(tasks):
                result = await task

            
async def sendRequest(session, prefix, suffix):
    url = prefix + "/" +  suffix
    async with session.get(url) as resp:
        response = await resp.text()
        return json.loads(response)


await(main())

Collectives™ on Stack Overflow

Python Asyncio - Running the same task in parallel

1 Answer 1

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Related