Python’s Iterators and Generators: Still Not Fully Understood

Question

import random

my_list = [random.randint(3, 100) for i in range(4)]

def is_even(some_list):
    for i in some_list:
        if i % 2 == 0:
            yield True
        else:
            yield False

print(my_list)
print(list(is_even(my_list)))

>>> [94, 53, 27, 42]
>>> [True, False, False, True]

It seems that I still do not fully understand the concept. On each iteration, the is_even function yields True or False accordingly. I do not see how those “yields” are accumulated. (Or, I do not see how they are appended to the final list at the end of each iteration. For example, in the given example, the first iteration yields True, then the second iteration starts. Where is that first True value kept?) What exactly is happening there?

Answer 1

To answer your question: the accumulation magic happens inside the list constructor - in fact, accumulation of items into a list is its very job. When you write:

print(list(is_even(my_list)))

The object returned by is_even(...) has a next() method that provides the next value yielded by the generator, or raises an exception when no more values are available. The list function is the one that accumulates those values into a list, which is finally returned.

You can imagine list being a function defined like this:

def list(iter):
    accum = []
    # the following loop is an approximate expansion of
    # for item in iter: accum.append(item)
    while 1:
        try:
            # get the next yielded value from generator
            item = iter.next()
        except StopIteration:
            # no more values
            break
        accum.append(item)
    return accum

In your case, where list is given a generator, the for loop is exhausting the generator and accumulating individual elements in a temporary list kept under the hood. It is this temporary list, accum in the above implementation, that holds that first True value, and goes on to store the subsequent ones. Once the generator is exhausted, the whole accumulated list is returned to the caller.

Answer 2

I think I understand what you mean:

Each time is_even is run, it checks every number in the list and returns True or false (as i'm sure you're aware). When you use 'print(list(is_even(my_list)))' the process is very simple - you are creating a list of every result returned by the function 'is_even', in the same order the for loop processed them before hand.

Hope this has helped.

Answer 3

You can understand the iterators as a list which has all its elements unfilled until you ask for them. If you instance a list with an iterator as a parameter, like you are doing in this instruction list(is_even(my_list)) you will get all of its elements out to memory immediately, which is not what we want when using iterators because is more efficient to have only the last element to check and not the whole thing, right? so try to iterate over it once at a time either like this:

for item in iterator:

Or using next() method.

Answer 4

Following your example from start:

my_list = [random.randint(3, 100) for i in range(4)]

This is a nice example of list comprehension. It is just a more concise way of writing something like:

my_list = []
for i in range(4):
    my_list.append(random.randint(3, 100))

---

Next, you define a generator function:

def is_even(some_list):
    for i in some_list:
        if i % 2 == 0:
            yield True
        else:
            yield False

A generator function is able to behave like an iterator. Values are "yielded" on demand. You can picture it as a function that streams results.

---

print(list(is_even(my_list)))

A few things are happening at this point, let's split them to get a better overview of what is going on:

>>> g = is_even(my_list)

is_even(my_list) doesn't return a list, as you may expect, but a generator:

>>> g
... <generator object is_even at ...>
>>> type(g)
... generator

The list construct accepts an iterable as argument, an a generator is iterable. Your yielded values are actually accumulated at this step, when the actual list is constructed:

>>> l = list(g)
>>> print(l)
... [True, False, False, True]