Converting iterator to list changes the iterator

Question

The following code produces some very strange results

import re
string = "test test test"
positions = re.finditer("test", string)
print (list(positions))
print (list(positions))

Output:

[<Match object...>, <Match object...>, <Match object...>]
[]

Now, I think I know what's going on here. The first list call "exhausts" the iterator (so it "uses the iterator up", like in a generator, in the process of creating a list from the iterator) so then when the second call to list is made, the iterator is gone and we get an empty list. This seems to be confirmed by the paragraph below, although I am am trying to understand some of the things they are saying here, so I am not entirely happy with this explanation (if it is the right one):

A container object (such as a list) produces a fresh new iterator each time you pass it to the iter() function or use it in a for loop. Attempting this with an iterator will just return the same exhausted iterator object used in the previous iteration pass, making it appear like an empty container.

The above paragraph is from the official documentation.

I do not really understand what they are saying in the first sentence in the above paragraph, especially with regards to passing to the iter() function, and I do not know how they connect usage in a for loop to a list producing a fresh new iterator. The second sentence, though, seems closer to what I first thought was going on in the code above.

If anyone can help me clear up the confusion here, I would appreciate it immensely.

NOTE:

I am using Python 3.5.1

wim · Accepted Answer · 2016-03-18 21:55:18Z

1

This line:

positions = re.finditer("test", string)

It returns a one-shot iterator. Then you called list(positions) twice on the same iterator.

So for the second call, it was already exhausted.

Lists will give you a fresh iterator every time you iterate them, so there is no exhausting behaviour on a list itself. Compare the behaviour below to understand the piece of documentation you quoted:

>>> L = ['a', 'b', 'c']
>>> list(L)
['a', 'b', 'c']
>>> list(L)
['a', 'b', 'c']
>>> iter_L = iter(L)  # calls L.__iter__() and returns you a one-shot iterator
>>> list(iter_L)
['a', 'b', 'c']
>>> list(iter_L)
[]

edited Mar 18, 2016 at 21:55

answered Mar 18, 2016 at 21:48

wim

368k112 gold badges680 silver badges815 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

Konrad Over a year ago

one-shot iterator?

wim Over a year ago

Yes. You can only iterate it once.

Konrad Over a year ago

Ok, but that was pretty much what I said I thought already (that it exhausts the iterator the first time, then the second time it's gone). But nevermind, I think I have figured out what the paragraph I have quoted in my question is saying.

Konrad Over a year ago

Haha, was just about to answer my own question with exactly the latest edit you have made on your answer (with the code starting with L = ['a'...). But I see you beat me to it

wim Over a year ago

Note that the behaviour you see in the original question has nothing to do with list.__iter__, and everything to do with the contract of re.finditer.

|

Amit Gold · Accepted Answer · 2016-03-18 22:09:33Z

When list() runs over an iterator, it basically calls next() on it until it raises the exception StopIteration and then appends each thing that the next() returns to a list, then returns that list. Basically, an implementation of list(iter) could be:

my_list(iter):
    output = []
    try:
        while True:
            output.append(iter.next())
    except StopIteration:
        return output

By the way, a for loop does the exact same thing, but instead of output.append(iter.next()) it does the body of the loop.

Collectives™ on Stack Overflow

Converting iterator to list changes the iterator

2 Answers 2

8 Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

8 Comments

Comments

Related