2

I think I might be repeating the question but I didn't find any of the answers suited to my requirement. Pardon my ignorance.

I have a program running which continuously spits out some binary data from a server.It never stops until it's killed.

I want to wrap it in a python script to read the output and process it as and when it arrives. I tried out few of the subprocess ideas in stack overflow but no use. Please suggest.

p=subprocess.popen(args,stderr=PIPE,stdin=PIPE,stdout=PIPE,shell=FALSE)
#p.communicate#blocks forever as expected
#p.stdout.read/readlines/readline-->blocks
#select(on p.stdout.fileno())-->blocks

what is the best method?

Improve this question
3
  • I would like to add that the data generated is a http stream.Forgot to mention that Commented Nov 14, 2011 at 5:05
  • OK.matter finally resolved.I figured out that the server feeding the data was intermittently failing to stream as in no HTTP error but just hanging in wait state.It was slow sometimes and sometimes failing to authenticate.The solution read/readline by icktoofay is working fine. Commented Nov 14, 2011 at 6:49
  • I was able to solve another related problem using asyncproc module mentioned in this post stackoverflow.com/questions/375427/… though read no more blocks by using select,I need to read the exact bytes the app spits out, otherwise I would have to do a packet re-construction logic which the app already did.To circumvent, that using asyncproc helped.Now doing a read returns the exact bytes(one packet length) the other app spat at a time. Commented Nov 15, 2011 at 23:14

3 Answers 3

Reset to default
3

Read with a length limit:

proc = subprocess.Popen(args, stdin=None, stdout=subprocess.PIPE, stderr=None)
while True:
    chunk = proc.stdout.read(1024)
    # chunk is <= 1024 bytes

This is the code from your comment, slightly modified. It works for me:

import subprocess

class container(object):
    pass

self = container()
args = ['yes', 'test ' * 10]

self.p = subprocess.Popen(args, stdin=None, stderr=None,
                          stdout=subprocess.PIPE, shell=False)
while True:
    chunk = self.p.stdout.read(1024)
    print 'printing chunk'
    print chunk
Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

this didn't work: {self.p=subprocess.Popen(args,stdin=None,stderr=None,stdout=subprocess.PIPE,shell=False) #out,err=p.communicate() print self.p print self.p.stdout while True: chunk=self.p.stdout.read(1024) self.p.stdout.flush() print 'printing chunk' print chunk}
This code looks correct. @rroh can you be more precise about how "it didn't work"?
Mine just waits indefinitely in read() never returns. <subprocess.Popen object at 0x8f4cc4c> <open file '<fdopen>', mode 'rb' at 0x8f0ad30>
I can see the the running process(from popen) using ps.and it does catch my keyboardinterrupt and exits.But doesn't spit out anything while the same command works fine in command line.
0

You could run the other program with its output directed to a file and then use Python's f.readline() to tail the file.

Improve this answer

2 Comments

Hi Raymond,I thought of that.But the program spitting data spits it as and when a packet arrives and I need to capture the entire packet and process/redirect.It would be cumbersome to write a logic for packet reconstruction from readline again.I just have the logic to process the packet and not to assemble.
I was thinking of doing trying one more method../datagenerator | myprogram.py.Any thoughts.I will try this out though.
0

It sounds like you could use an asynchronous version of the subprocess module. For more information, check out the developer's blog.

Improve this answer

1 Comment

Interesting.I will check this out.Thx

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.