0

I'm trying to capture the output stream of executed commands from a python script. All is going well until the command is a python command.

Here are my functions that get the streams with different callbacks on stdout and stderr :

capture_output.py

import sys
import logging
import shlex
import asyncio


async def _read_stream(stream, cb):  
    while True:
        line = await stream.readline()
        if line:
            cb(line.decode())
        else:
            break


async def _stream_subprocess(cmd, stdout_cb, stderr_cb):  
    process = await asyncio.create_subprocess_exec(*cmd,
            stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)

    await asyncio.wait([
        _read_stream(process.stdout, stdout_cb),
        _read_stream(process.stderr, stderr_cb)
    ])
    return await process.wait()


def execute(cmd, stdout_cb, stderr_cb):  
    loop = asyncio.get_event_loop()
    rc = loop.run_until_complete(
        _stream_subprocess(
            cmd,
            stdout_cb,
            stderr_cb,
    ))
    loop.close()
    return rc


def run_command(command,
                output_function=lambda x: print("STDOUT: %s" % x),
                error_handling=lambda x: print("STDERR: %s" % x),
                cwd=None):
    execute(shlex.split(command), output_function, error_handling, )


if __name__ == "__main__":
    "Get the command on 1st position and run it"
    command_str = sys.argv[1]
    run_command(command_str)

Here is an example of external command :

anylongrunning_script.sh

#!/bin/bash
for i in {1..5}
do
   echo "$i"
   sleep 0.5
done

If I run python capture_output.py "sh anylongrunning_script.sh", I see the live stream, everything is going well.

But if run a python command like this python capture_output.py "python anylongrunning_script.py" :

anylongrunning_script.py

import time

for i in range(1, 6):
    print(i)
    time.sleep(0.5)

The output will be rendered in 1 block at the end. Do you know what is the difference between a shell "echo" and a python "print" in this case ?

2

1 Answer 1

1

Disable output buffering by running your command with the -u option. Try python capture_output.py "python -u anylongrunning_script.py"

You can make sure all python processes use unbuffered mode by setting the PYTHONUNBUFFERED env variable for your subprocesses.

Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.