11

I was debugging some code with generators and came to this question. Assume I have a generator function

def f(x):
    yield x

and a function returning a generator:

def g(x):
    return f(x)

They surely return the same thing. Can there be any differences when using them interchangeably in Python code? Is there any way to distinguish the two (without inspect)?

3
  • Can you use next on the result of g(x)? Commented Sep 10, 2016 at 14:25
  • 3
    f is actually also "a function returning a generator", not a "generator" itself. Commented Sep 10, 2016 at 14:30
  • There is absolutely no difference, other than an additional function call, which negligably slows down execution. Commented Sep 10, 2016 at 16:11

4 Answers 4

4

I like turkus answer, however examples shown are mostly theoretical and aren't common in day to day coding.

The main practical difference between generator function (with yield) and function which returns generator is that the generator function is lazily evaluated.

Consider this session:

$ python
Python 3.6.0 
[GCC 6.3.1 20170109] on linux
>>> def a():
...     print('in a')
...     yield 0
... 
>>> def b():
...     print('in b')
...     return iter(range(1))
... 
>>> aa = a()  # Lazy evaluation - nothing printed after this line.
>>> next(aa)
in a
0
>>> next(aa)
Traceback ...
StopIteration
>>> bb = b()  # Eager evaluation - whole function is executed after this.
in b
>>> next(bb)
0
>>> next(bb)
Traceback ...
StopIteration

None of them is a golden bullet.

To give you a real example of where this lazy evaluation makes a huge difference in your code check this example.

def get_numbers(x: int):
    if x < 0:
        raise ValueError("Value cannot be negative")
    for i in range(x):
        yield i

try:
   numbers = get_numbers(-5)
except ValueError:
   pass  # log or something
else:
   print(list(numbers))  # <== ValueError is thrown here!

Here is where lazy evaluation is actually bad for your function. It will throw exception in arguably wrong place because the intention is to make it fail just at the start, not during iteration. With this implementation you're passing responsibility of triggering the generator function and managing exception to its user which is tedious and somewhat ugly:

import itertools

try:
    numbers = get_numbers(-5)
    first = next(numbers)
    numbers = itertools.chain([first], numbers)
except ValueError:
    ...

The best way to solve this is to make a function that returns a generator instead a generator function:

def get_numbers(x: int):
    if x < 0:
        raise ValueError("Value cannot be negative")
    return (i for i in range(x))  # I know it can be just `return range(x)`, but I keep it that way to make a point.

As you can see there is no "best way" to do it, both options are viable. It all depends on how you want things to work.

Sign up to request clarification or add additional context in comments.

2 Comments

This should be the accepted answer for future visitors, as the difference in evaluation is there, even if it is not that obvious in the example of the OP.
Added even better example which I came about, by creating such bad generator function myself. Learn from my mistakes!
3

The best way to check it out is using inspect.isgeneratorfunction, which is quite simple function:

def ismethod(object):
    """Return true if the object is an instance method.

    Instance method objects provide these attributes:
        __doc__         documentation string
        __name__        name with which this method was defined
        im_class        class object in which this method belongs
        im_func         function object containing implementation of method
        im_self         instance to which this method is bound, or None"""
    return isinstance(object, types.MethodType)

def isfunction(object):
    """Return true if the object is a user-defined function.

    Function objects provide these attributes:
        __doc__         documentation string
        __name__        name with which this function was defined
        func_code       code object containing compiled function bytecode
        func_defaults   tuple of any default values for arguments
        func_doc        (same as __doc__)
        func_globals    global namespace in which this function was defined
        func_name       (same as __name__)"""
    return isinstance(object, types.FunctionType)

def isgeneratorfunction(object):
    """Return true if the object is a user-defined generator function.

    Generator function objects provides same attributes as functions.

    See help(isfunction) for attributes listing."""
    return bool((isfunction(object) or ismethod(object)) and
                object.func_code.co_flags & CO_GENERATOR)

Now, if you declared your generator using a syntax like this:

my_generator = (i*i for i in range(1000000))

In that case, you could check its type quite easily, for instance, __class__ will return <type 'generator'>.

2 Comments

Why is this answer downvoted? I guess is because the author mentioned he didn't want to use inspect but that's why I've posted the internals of the inspect method. So, I don't understand.
I don't understand either
2

They will act the same. And about way to distinguish the two (without inspect). In python? Only inspect:

import inspect

print inspect.isgeneratorfunction(g) --> False
print inspect.isgeneratorfunction(f) --> True

Of course you can also check it using dis:

Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> def f(x):
...     yield x
... 
>>> def g(x):
...     return f(x)
... 
>>> import dis
>>> dis.dis(f)
  2           0 LOAD_FAST                0 (x)
              3 YIELD_VALUE         
              4 POP_TOP             
              5 LOAD_CONST               0 (None)
              8 RETURN_VALUE        
>>> dis.dis(g)
  2           0 LOAD_GLOBAL              0 (f)
              3 LOAD_FAST                0 (x)
              6 CALL_FUNCTION            1
              9 RETURN_VALUE  

but inspect is more appropriate.

Comments

1

If you want to identify what is a generator, the answer is simple. f is a generator because it contains the yield statement. g is not a generator because it does not contain the yield statement. (You can have a look at https://docs.python.org/2/reference/simple_stmts.html?highlight=yield#the-yield-statement)

As for what is the difference in using them, they are quite the same. You can store a generator in a variable, then use it in a for statement. In that case, g(x) just acts as a "middle man". Have a look at the following examples:

def f(x):
  for r in range(x):
    yield r

def g(x):
  return f(x)

print "using f(x)"
for i in f(3):
  print i

print "using g(x)"
for j in g(3):
  print j

print "store the iterator f(x) in a variable, then use it in a 'for' statement"
m = f(3)
for k in m:
  print k

print "store the iterator f(x) returned by g(x), then use it in a 'for' statement"
n = g(3)
for k in n:
  print k

These are in python2. Just add parentheses in print statements for python3.

3 Comments

f is not a generator. The way to see what it is is to print f. To see what it returns, print f(x). The second one is a generator.
What is wrong with this answer? It was downvoted more than once. Well, it does not answer the question how to tell the difference without inspect, but the part about yield being present in the function deserves +1, IMHO.
I added print f in the code and got <function f at 0x7f1c010f3578>. I also added print f(3) and got <generator object f at 0x7f1c010e8780><generator object f at 0x7f1c010e8780>. According to "Python Enhancement Proposal PEP 255: "The yield statement may only be used inside functions. A function that contains a yield statement is called a generator function. A generator function is an ordinary function object in all respects, [...]" So a generator is a special kind of function. (n.b. a link to PEP 255 is present in python doc). (Aso @VPfB, thanks for the support!)

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.