Python 2.5 subprocess.Popen problem

Question

I'm trying to run the following little Python script:

#!/usr/local/bin/python2.5
import sys, subprocess
child = subprocess.Popen("muscle -stable -FASTA",
                         stdin=sys.stdin,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.PIPE,
                         shell=(sys.platform!="win32"))
print child.stderr.read()
print child.stdout.read()

Which works fine if stdin does not exceed about 3750 characters. Above this limit, the subprocess does no longer produce any stdout or stderr. If I switch them for sys.stdout and sys.stderr respectively, everything works again as expected. The muscle executable writes to both stdout and stderr when called from command-line.

Any suggestions what could be the cause and how to solve this?- best without updating Python, since the installation is on a server I have limited access to.

phooji · Accepted Answer · 2011-03-24 22:54:07Z

3

Use child.communicate(someinput) to get the full output; see the 'warning' boxes in the documentation also. If you really need to read stdin interactively while writing to stdout, then (at least on Windows) you may be out of luck with Python + pipes.

answered Mar 24, 2011 at 22:54

phooji

10.4k3 gold badges41 silver badges46 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Michael Schubert Over a year ago

Thank you, "out, err = child.communicate()" works perfectly. But why I don't understand. For the reason geekosaur explained?

Fred Foo Over a year ago

@user: communicate fires up two threads that read stdout and stderr concurrently. svn.python.org/view/python/trunk/Lib/…

phooji Over a year ago

@user538603: Yes. What's worse, the behavior will probably differ across platforms, etc., so you may not notice deadlock at all (while, say, your client does). In addition, if you launch multiple subprocesses at once, you'll need close_fds=True to ensure that the child processes don't inadvertently start waiting for each other.

geekosaur · Accepted Answer · 2011-03-24 22:54:13Z

3

Your process is blocked on a full pipe, waiting for you to read from it; since you're busy writing, you have a nice little deadlock. If you're going to both read and write to the same subprocess, you should use a select()-based event loop, reading when output is available and writing when there is room for more data in the input pipe.

answered Mar 24, 2011 at 22:54

geekosaur

61.8k13 gold badges127 silver badges115 bronze badges

1 Comment

Michael Schubert Over a year ago

An event-loop seems to be overkill for the thing I'm trying to do. communicate() solved it, but thanks either way.

Collectives™ on Stack Overflow

Python 2.5 subprocess.Popen problem

2 Answers 2

3 Comments

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Related