Iterating over asyncio.coroutine

Question

I've been mucking around with asyncio recently, and while I'm beginning to get an intuition for how it works, there's something that I've not been able to do. I'm not sure if it's because I've got the construction wrong, or if there's a reason why what I'm trying to do doesn't make sense.

In short, I want to be able to iterate over a yielding asyncio.coroutine. For example, I'd like to be able to do something like:

@asyncio.coroutine
def countdown(n):
    while n > 0:
        yield from asyncio.sleep(1)
        n = n - 1
        yield n

@asyncio.coroutine
def do_work():
    for n in countdown(5):
        print(n)

loop.run_until_complete(do_work())

However, this throws an exception from the bowels of asyncio. I've tried other things, like for n in (yield from countdown(5)): ... but that also gives a similarly opaque runtime exception.

I can't immediately see why you shouldn't be do something like this, but I'm getting to the limits of my ability to understand what's going on.

So:

if it is possible to do this, how can I do it?
if it is not possible, why not?

Let me know if this question's not clear!

Andrew Svetlov · Accepted Answer · 2015-06-23 11:29:06Z

In asyncio coroutines you should to use yield from and never yield. That's by design. Argument for yield from should be another coroutine or asyncio.Future instance only.

Calls of coroutine itself should be used with yield from again like yield from countdown(5).

For your case I recommend using queues:

import asyncio

@asyncio.coroutine
def countdown(n, queue):
    while n > 0:
        yield from asyncio.sleep(1)
        n = n - 1
        yield from queue.put(n)
    yield from queue.put(None)

@asyncio.coroutine
def do_work():
    queue = asyncio.Queue()
    asyncio.async(countdown(5, queue))
    while True:
        v = yield from queue.get()
        if v:
            print(v)
        else:
            break

asyncio.get_event_loop().run_until_complete(do_work())

Well, you can use check for values yielded by countdown, the following example works. But I think it is antipattern:

Too easy to make a mess
You anyway cannot compose countdown calls with, say, itertools functions. I mean something like sum(countdown(5)) or itertools.accumulate(countdown(5)).

Anyway, example with mixing yield and yield from in coroutine:

import asyncio

@asyncio.coroutine
def countdown(n):
    while n > 0:
        yield from asyncio.sleep(1)
        n = n - 1
        yield n

@asyncio.coroutine
def do_work():
    for n in countdown(5):
        if isinstance(n, asyncio.Future):
            yield from n
        else:
            print(n)

asyncio.get_event_loop().run_until_complete(do_work())

Huazuo Gao · Accepted Answer · 2015-09-10 01:39:19Z

In Python 3.5, the async for syntax is introduced. However, asynchronous iterator function syntax is still absent (i.e. yield is prohibited in async functions). Here's a workaround:

import asyncio
import inspect

class escape(object):
    def __init__(self, value):
        self.value = value

class _asynciter(object):
    def __init__(self, iterator):
        self.itr = iterator
    async def __aiter__(self):
        return self
    async def __anext__(self):
        try:
            yielded = next(self.itr)
            while inspect.isawaitable(yielded):
                try:
                    result = await yielded
                except Exception as e:
                    yielded = self.itr.throw(e)
                else:
                    yielded = self.itr.send(result)
            else:
                if isinstance(yielded, escape):
                    return yielded.value
                else:
                    return yielded
        except StopIteration:
            raise StopAsyncIteration

def asynciter(f):
    return lambda *arg, **kwarg: _asynciter(f(*arg, **kwarg))

Then your code could be written as:

@asynciter
def countdown(n):
    while n > 0:
        yield from asyncio.sleep(1)
        #or:
        #yield asyncio.sleep(1)
        n = n - 1
        yield n

async def do_work():
    async for n in countdown(5):
        print(n)

asyncio.get_event_loop().run_until_complete(do_work())

To learn about the new syntax, and how this code works, see PEP 492

This is awesome! Of course we should be using Pyhton 3.6 where asyncgenerators are natively supported, but if you're forced to us Python 3.5 for whatever reason, this is a real gem. It is also very similar in syntax to the way you'd use async generators in 3.6, so transition should be easy when it happens.
How can I create nested async generators this way? In your example, what if do_work itself should yield results? If we decorate it with @asynciter instead of using async def, we obviously cannot use async for anymore. How do we iterate over results of the inner coroutine?
Actually, nevermind, I've discovered there's an awesome package async_generator that does all that already!

Claude · Accepted Answer · 2015-09-14 13:08:29Z

Update: It seems python 3.5 supports this even better natively:

Being stuck with the same problem (and inspired by code in aio-s3), I felt there ought to be a more elegant solution.

import asyncio

def countdown(number):
    @asyncio.coroutine
    def sleep(returnvalue):
        yield from asyncio.sleep(1)
        return returnvalue
    for n in range(number, 0, -1):
        yield sleep(n)

@asyncio.coroutine
def print_countdown():
    for future in countdown(5):
        n = yield from future
        print ("Counting down: %d" % n)

asyncio.get_event_loop().run_until_complete(print_countdown())

Rationale: The countdown method yields futures, each one will resolve after a 1 second sleep to the number provided.

The print_countdown function takes the first future, yield from-ing it (which will pause until it's resolved) and getting the intended result: n.

Collectives™ on Stack Overflow

Iterating over asyncio.coroutine

3 Answers 3

Comments

3 Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

3 Comments

Comments

Related