3

I want to open a Python script using subprocess in my main python program. I want these two programs to be able to chat with one another as they are both running so I can monitor the activity in the slave script, i.e. I need them to send strings between each other.

The main program will have a function similar to this that will communicate with and monitor the slave script:

Script 1

import subprocess
import pickle
import sys
import time
import os

def communicate(clock_speed, channel_number, frequency):
    p = subprocess.Popen(['C:\\Python27\\pythonw','test.py'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
    data = pickle.dumps([clock_speed, channel_number, frequency]).replace("\n", "\\()")
    print data
    p.stdin.write("Start\n")
    print p.stdout.read()
    p.stdin.write(data + "\n")
    p.poll()
    print p.stdout.readline()
    print "return:" + p.stdout.readline()
    #p.kill()

if __name__ == '__main__':
    print "GO"
    communicate(clock_speed = 400, channel_number = 0, frequency = 5*1e6)

The test.py script looks similar to this:

Script 2

import ctypes
import pickle
import time
import sys

start = raw_input("")
sys.stdout.write("Ready For Data")
data = raw_input("")
data = pickle.loads(data.replace("\\()", "\n"))
sys.stdout.write(str(data))
###BUNCH OF OTHER STUFF###

What I want these scripts to do is the following:

  1. Script 1 to open Script 2 using Popen
  2. Script 1 sends the string "Start\n"
  3. Script 2 reads this string and sends the string "Ready For Data"
  4. Script 1 reads this string and sends the pickled data to Script 2
  5. Then whatever...

The main question is how to do parts 2-4. Then the rest of the communication between the two scripts should follow. As of now, I have only been able to read the strings from Script 2 after it has been terminated.

Any help is greatly appreciated.

UPDATE:

Script 1 must be run using 32-bit Python, while Script 2 must be run using 64-bit Python.

Improve this question
4
  • Could you use the multiprocessing module which is designed to run python in supbrocesses Commented Mar 13, 2013 at 21:36
  • The reason why I am using subprocess is because I need the second script to be run using 64-bit Python, while the main script is run in 32-bit Python. It is a driver issue. I did not think that multiprocessing allowed for this. Commented Mar 13, 2013 at 21:39
  • I think I had a similar problem with a threadpool running in python and each thread "communicating" with some node.js workers -- what worked for me was to flush() the stdout after each write() since otherwise, buffering delayed everything until the streams were closed. Commented Mar 13, 2013 at 21:49
  • It allows calls to other machines so possibly docs.python.org/2/library/… Commented Mar 13, 2013 at 21:53

2 Answers 2

Reset to default
4

The problem with pipes is that if you call a read operation and there is nothing to read, your code is stalled until the other party writes something for you to read. Also if you write too much, your next write operation might block until the other party reads something out of the pipe and frees it.

There are "non-blocking calls" you can make, that will return an error in these cases instead of blocking, but your application will still need to deal with the errors sensibly.

In any case, you need to set up some kind of protocol. Think of HTTP, or any other protocol you know well: there are requests and responses, and while you are reading either of the two the protocol always tells you if there is something else you need to read or not. That way you can always make an informed decision on whether to wait for more data or not.

Here is an example that works. It works because there is the following protocol:

  • p1 sends a single line, ending with '\n';
  • p2 does the same;
  • p1 sends another line;
  • p2 does the same;
  • both are happy and exit.

In order to write a line to the pipe (on either side) and make sure it gets onto the pipe, I call write() and then flush().

In order to read a single line from the pipe (on either side) but not a single byte more, thus blocking my code until the line is ready and no longer than that, I use readline().

There are other calls you could make and other protocols, including ready-made ones, but the single-line protocol works well for simple things and for a demo like this.

p1.py:

import subprocess

p = subprocess.Popen(['python', 'p2.py'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
p.stdin.write("Hello\n")
p.stdin.flush()
print 'got', p.stdout.readline().strip()
p.stdin.write("How are you?\n")
p.stdin.flush()
print 'got', p.stdout.readline().strip()

p2.py:

import sys

data = sys.stdin.readline()
sys.stdout.write("Hm.\n")
sys.stdout.flush()
data = sys.stdin.readline()
sys.stdout.write("Whatever.\n")
sys.stdout.flush()
Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

There might be issues with "\n" on Windows: whether .readline() understands newline characters produced by other end (b"\n" <--> b"\r\n" conversions).
Excellent answer. Thank you. Also, the "\n" works on Windows for me (in case anyone reading this for reference is worried about this issue).
0

I also had a problem similar to this, where there was no way to send general Python objects between different processes without running into the problem of knowing either when the other side hasn't sent an object or is closed. Also trying to use multiprocessing.Queue usually means that the process needs to have been started by the current process which is not always the case when two processes want to collaborate.

To combat this I use the picklepipe module, which defines a generic object serialization pipe interface as well as a pipe that uses the pickle protocol called the PicklePipe (also one that uses the marshal protocol called MarshalPipe). It can send more than just strings, it can send any pickleable object to it's peer.

The pipes are even selectable, meaning they can be used by the selectors module (or selectors2, selectors34) as file objects when a new object is ready to be received. This makes waiting for many different pipes to be ready very efficient.

Supports Python 2.7+ (and probably 2.6) and all major platforms. Can even send objects between two different versions of Python! Check out the project documentation or view the source on Github.

Disclosure: I am the author of picklepipe. I would love to hear your feedback. :)

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.