5

I want my program to print out one line at a time, however it's printing multiple at a time and creating a garbled mess. I cannot seem to find out why the semaphore is not preventing multiple processes from printing over each other.

How can I get it to respect the semaphore?

Here is a simplified version of my code that has the same problem when run it (I'm running on Windows with Python 2.7.11 (this can't be changed)):

import multiprocessing

rightofway = multiprocessing.Semaphore(1)

def writenum(number):
    rightofway.acquire()
    print("[+] - " + str(number))
    rightofway.release()
    return

def main():
    starting = 0
    ending = 50

    list = range(starting, ending)

    pool = multiprocessing.Pool(10)
    pool.map(writenum, list)
    return

#Required for Windows multiprocessing
if __name__ == '__main__':
    main()

Here is an example of garbled output:

[+] - 0
[+] - 1
[+] - 2
[+] - 3
[+] - 4
[+] - 5
[+] - 6
[+] - 7
[[+] - 8
+] - 10[
+] - 9[+] - 11
[+] - 12

[[+] - 13+] - 14

[[+] - 15+] - 16

[[+] - 18+] - 17

[[+] - 19+] - 20

[[+] - 22+] - 21

[[+] - 23+] - 24

[[+] - 26+] - 25

[[+] - 27+] - 28

[[+] - 30+] - 29

[[+] - 31+] - 32

[[+] - 34+] - 33

[[+] - 35+] - 36

[[+] - 38+] - 37

[[+] - 39+] - 40

[[+] - 42+] - 41

[[+] - 43+] - 44

[[+] - 46+] - 45

[[+] - 47+] - 48

[+] - 49

Here's an example of the output I want (note I don't care about the order):

[+] - 0
[+] - 1
[+] - 2
[+] - 3
[+] - 4
[+] - 5
[+] - 6
[+] - 7
[+] - 8
[+] - 9
[+] - 10
[+] - 11
[+] - 12
[+] - 13
[+] - 14
[+] - 15
[+] - 16
[+] - 17
[+] - 18
[+] - 19
[+] - 20
[+] - 21
[+] - 22
[+] - 23
[+] - 24
[+] - 25
[+] - 26
[+] - 27
[+] - 28
[+] - 29
[+] - 30
[+] - 31
[+] - 32
[+] - 33
[+] - 36
[+] - 34
[+] - 35
[+] - 37
[+] - 38
[+] - 40
[+] - 39
[+] - 41
[+] - 42
[+] - 44
[+] - 43
[+] - 45
[+] - 46
[+] - 48
[+] - 47
[+] - 49
Improve this question
8
  • Which operating system are you on? Commented May 16, 2018 at 20:00
  • @tdelaney - Windows. I'll add that to my question. Commented May 16, 2018 at 20:01
  • 1
    I don't have windows handy, but this accepted solution uses an initializer with the pool. its my guess for the fix. a lock should work for you or adapt for semaphore stackoverflow.com/questions/28664720/… Commented May 16, 2018 at 20:07
  • @tdelaney - I attempted it and it resulted in exactly the same kind of garbled output. Lock isn't working either. Commented May 16, 2018 at 20:18
  • @tdelaney - Does the code work for you? Commented May 16, 2018 at 20:23

2 Answers 2

Reset to default
5

You question is similar to this one.

From the multiprocessing programming guidelines.

Explicitly pass resources to child processes

... it is better to pass the object as an argument to the constructor for the child process.

Apart from making the code (potentially) compatible with Windows ...

On Windows, you need to pass the shared objects to the Process constructor list of arguments. Otherwise, the child process will get a brand new copy instead of the parent's one. That's why you get the impression the Semaphore isn't working. The two processes are creating their own distinct Semaphore object instead of sharing the same one.

To pass a Semaphore object to a Pool on Windows you need to struggle a bit but not too much. As you cannot pass the Semaphore object to the writenum function directly, you need to rely on the Pool initializer.

from multiprocessing import Semaphore, Pool

mutex = None

def initializer(semaphore):
    """This function is run at the Pool startup. 
    Use it to set your Semaphore object in the child process.

    """
    global mutex

    mutex = semaphore

def writenum(args):
    with mutex:
        print "[+] - " + str(number)

def main():
    semaphore = Semaphore()
    pool = Pool(initializer=initializer, initargs=[semaphore])

    numbers = range(50)

    pool.map(writenum, numbers)

EDIT: just noticed I wrote about Lock instead of Semaphore. The core reasoning remains the same.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

That worked! And I think I see why the code is that way, but lemme ask just in case. Why do we need to have mutex=None at the top even though it is declared in the initializer function? Also, why is the initializer function not a part of the Pool object?
Can you better explain what do you mean with the question why is the initializer function not a part of the Pool object?
It appears that Pool() is built to take arguments and bring them into the global space of the processes. If so, then why do I have to write part of that functionality myself instead of simply passing the (already global?) semaphore to the processes?
The Pool has two ways of passing data to child processes. The first is via the apply/map function arguments. This requires arguments to be pickleable and Semaphores are not. You can only rely on the second way which is by using the initializer function. As that function cannot return values, only way for your data to persist is by using global variables.
0

To make things a bit easier, Following worked for me. Tested on Win10. TL;DR - Use locks not semaphore

import multiprocessing

rightofway = multiprocessing.Lock()

def writenum(number):

    with rightofway:
        print("[+] - " + str(number))

    return

def main():
    starting = 0
    ending = 50

    list = range(starting, ending)

    pool = multiprocessing.Pool(10)
    pool.map(writenum, list)
    return

#Required for Windows multiprocessing
if __name__ == '__main__':
    main()
Improve this answer

1 Comment

This isn't working for me for some reason. Same jumbled results.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.