Python, loops and closures

Question

I'm a fairly experienced C/C++ (and to some degree, Java) programmer. I'm learning python, but I'm baffled at some strange (for my backgroung) behaviors of the language.

I'm learning about nested function and closures (reading "Learning Python", that seems a really good source for me).

I understand that if I nest a def inside a for loop when I call the created function, it looks up the last value of the captured loop variable (as it captures by reference, as a C++ programmer would put it)

funcs = []
for i in range(4):
    def f():
        print(i)
    funcs.append(f)

and running the program the result is

>>> for f in funcs:
      f()


3
3
3
3

Now, I was wrapping my head around this when I stumbled upon this (what to me seems) an inconsistency: if I do

for i in range(4):
  funcs[i]()


0
1
2
3

more baffling, if I do

>>> i = 2
>>> funcs[i]()

2

and now, all functions in list returns 2:

for f in funcs:
  f()


2
2
2
2

there must be some scope related question that I can't grasp

Answer 1

First, this creates a list of four functions.

funcs = []
for i in range(4):
  def f():
    print(i)
  funcs.append(f)

Each of these functions looks up the value of i and then prints it.

This loops through the list of function and calls each one:

>>> for f in funcs:
      f()

As stated above, these functions look up i, which is 3 right now due to the for i in range(4) loop that completed earlier, so you get four printouts of 3.

Now you loop again, using i as the loop variable:

for i in range(4):
  funcs[i]()


0
1
2
3

The first time through the loop, i is 0, so when the function looks up i, it gets 0, and then prints that. Then it changes to 1, then 2, then 3.

The following code simply changes i in yet another way, and calls a function:

>>> i = 2
>>> funcs[i]()

2

You could've called any of those functions and they still would've printed 2, because that's the value of i now. You're just getting lost because you looped over range(4) to create these functions, then you looped over range(4) to index the list of functions, and you keep reusing i, and then you reassign i and also use it to index the list.

If you want each function's printed value of i to be fixed at what it was when you defined the function, the easiest way to do that is with a default argument, as those are evaluated when the function is defined rather than when it's called:

funcs = []
for i in range(4):
  def f(num=i):
    print(num)
  funcs.append(f)

Answer 2

For the sake of completeness, this is an alternate implementation:

def getfunc(i):
    return lambda: print(i)

funcs = []
for i in range(5):
    funcs.append(getfunc(i))

for f in funcs:
    f()    

Another implementation using functools.partial:

from functools import partial

def f(n):
  print(n)
  
funcs = [partial(f, i) for i in range(4)]
  
for f in funcs:
    f()

Answer 3

Your functions

def f():
    print(i)

print the current value of i.

If you write

for i in range(4):
    funcs[i]()

then i is being set to 0,1,2,3 as you go through the loop. That's what for i in range(4) means.

If you write

for f in funcs:
    f()

then i continues with whatever value it already had.

Answer 4

take some time for me to understand, actually in this example,

You will find f.__closure__ is None if you print it, ie. nothing related to closure, it's just about procedure the undefined local var i look for its value:

it can't find its value in local scope, finally find it in global scope (like python MRO)

Answer 5

There is no inconsistency here. The value of i in f() depends on the value of i from the parent scope. After you've run the first for i in range(4) i has the value of the last item in the range, which is 3, and thus all subsequent calls to f() will print 3

If you run

for i in range(4):
  funcs[i]()

you redefine the value of i at each iteration step, and so you get 0,1,2,3 as the values printed by f. Doing

for x in range(4):
   funcs[x]()

will not affect the value of i and so you'll get 3 as the value of i in all function calls