40

Am getting this error when using the pool.map(funct, iterable):

AttributeError: __exit__

No Explanation, only stack trace to the pool.py file within the module.

using in this way:

with Pool(processes=2) as pool:
   pool.map(myFunction, mylist)
   pool.map(myfunction2, mylist2)

I suspect there could be a problem with the picklability (python needs to pickle, or transform list data into byte stream) yet I'm not sure if this is true or if it is how to debug.

EDIT: new format of code that produces this error :

def governingFunct(list):
    #some tasks
    def myFunction():
         # function contents
    with closing(Pool(processes=2)) as pool:
         pool.map(myFunction, sublist)
         pool.map(myFunction2, sublist2)

ERROR PRODUCED:

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
Improve this question
2
  • what do you mean by while .. as? did you mean with? Commented Sep 22, 2014 at 7:20
  • Do include the full traceback for errors, and your actual code. while is an obvious syntax error here, for example, but from the exception it is clear you used with really. Commented Sep 22, 2014 at 7:32

2 Answers 2

Reset to default
66

In Python 2.x and 3.0, 3.1 and 3.2, multiprocessing.Pool() objects are not context managers. You cannot use them in a with statement. Only in Python 3.3 and up can you use them as such. From the Python 3 multiprocessing.Pool() documentation:

New in version 3.3: Pool objects now support the context management protocol – see Context Manager Types. __enter__() returns the pool object, and __exit__() calls terminate().

For earlier Python versions, you could use contextlib.closing(), but take into account this'll call pool.close(), not pool.terminate(). Terminate manually in that case:

from contextlib import closing

with closing(Pool(processes=2)) as pool:
    pool.map(myFunction, mylist)
    pool.map(myfunction2, mylist2)
    pool.terminate()

or create your own terminating() context manager:

from contextlib import contextmanager

@contextmanager
def terminating(thing):
    try:
        yield thing
    finally:
        thing.terminate()

with terminating(Pool(processes=2)) as pool:
    pool.map(myFunction, mylist)
    pool.map(myfunction2, mylist2)
Improve this answer
Sign up to request clarification or add additional context in comments.

11 Comments

Well Sir thank you for such a detailed explanation. You're solution has moved the function a step closer to working - now I am getting this error: PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
@sidewaiise: are you trying to use a method on a class perhaps? See Multiprocessing: using Pool.map on a function defined in a class
Heh I was actually just reading that as you posted this. Thanks - I'll have a read and comment shortly.
Not sure I understand the solution on that link - I'm not using classes. It's as simple as I wrote above, except that it is inside another function.... eg: **** Refer to Edited Question ^ **** So not sure.. any ideas?
@paulochf: for some reason the decorator line was missing. Only import it when you actually plan on using it! :-) (and sorry).
|
1

with statement is for object that have __enter__ and __exit__ functions, i.e. Context Manager Types
multiprocessing.Pool is not Context Manager Type. try do the following:

pool = Pool(processes=2)
pool.map(myFunction, mylist)
pool.map(myfunction2, mylist2)
Improve this answer

1 Comment

The question is why the OP thought it was. The answer is that the Python 3 documentation tells you it is. But it also qualifies that as Python 3.3 and up.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.