Unable to read stdout from spawned subprocess on Windows with Python

Question

I am trying to build a GUI frontend for a very simple windows terminal application (binary available here https://github.com/00-matt/randomx-stress/releases/download/200109/randomx-stress-windows-200109.zip), and start the terminal application as a subprocess using popen and feed the output into a queue, then read the queue and put the output into the tkinter GUI. No matter what I try though I am unable to get absolutely anything from stdout or stderr from the spawned subprocess. This is a stripped down version of my code:

from tkinter import *
import subprocess
import threading
import queue
import os
from os import system
from re import sub
import sys

os.environ["PYTHONUNBUFFERED"] = "1"

def enqueue_output(p, q):
    while True:
        out = p.stdout.readline()
        if out == '' and p.poll() is not None:
            break
        if out:
            print(out.strip(), flush=True)
            q.put_nowait(out.strip())


class Window(Frame):

    def __init__(self, master=None):
        Frame.__init__(self, master)
        self.master = master
        self.started = False
        self.p = None
        self.q = queue.Queue()
        self.threads = 1
        self.init_window()

    def init_window(self):
        self.master.title("RandomX Stress Tester")
        self.pack(fill=BOTH, expand=1)

        self.hashrateLabel = Label(self, text="Hashrate: ")
        self.hashrateLabel.after(2000, self.refresh_hashrate)

        self.startButton = Button(self, text="Start", background="green", command=self.startstop)
        self.quitButton = Button(self, text="Quit", command=self.client_exit)

        self.hashrateLabel.place(x=50, y=220)
        self.startButton.place(x=50, y=270)
        self.quitButton.place(x=220, y=270)

    def startstop(self):
        if not self.started:
            self.p = subprocess.Popen([r"randomx-stress.exe", "-t", str(self.threads)],
                        stdout=subprocess.PIPE,
                        stderr=subprocess.STDOUT,
                        shell=True,
                        encoding='utf-8',
                        errors='replace')
            self.t = threading.Thread(target=enqueue_output, args=(self.p, self.q))
            self.t.daemon = True
            self.t.start()
            self.started = True
            self.startButton.config(text="Stop", background="red")
        elif self.started:
            system("taskkill /im randomx-stress.exe /f")
            self.p.kill()
            self.t.join()
            self.started = False
            self.startButton.config(text="Start", background="green")

    def refresh_hashrate(self):
        print("checking hashrate if running")
        if not self.started:
            pass
        elif self.started:
            print("its running")
            try:
                line = self.q.get_nowait()
                print(line)
                if 'H/s' in line:
                    hashrate = line.split(' ')[0]
                    self.hashrateLabel.text = "Hashrate: {0:.2f} h/s".format(hashrate)
            except:
                print("error")
        self.hashrateLabel.after(2000, self.refresh_hashrate)

    def client_exit(self):
        exit()


root = Tk()
root.geometry("400x300")

app = Window(root)
root.mainloop()

My suspicion now is that the terminal application is buffering the output and I am unable to bypass that on Windows. If someone could confirm that is the issue and if possible offer a workaround I would appreciate it!

Is randomx-stress.exe a long running process? Because it is not run directly from the console, the program output is buffered and flushed to your script on block boundaries. On unix-like systems you can fix this with a pseudo-tty but I don't know any way to fix it on windows. — tdelaney
– tdelaney, Commented May 11, 2020 at 15:51
Yes, it's a long-running process that prints to the terminal every five seconds or so. The thing is I've made similar GUIs in the past that worked on Windows: github.com/jwinterm/kivy-cpuminer/releases , so is it just a difference of whether or how the spawned subprocess buffers its output? — jwinterm
– jwinterm, Commented May 11, 2020 at 16:01
This is line in the subprocess program that is outputting via cout: github.com/00-matt/randomx-stress/blob/master/main.cc#L137 . Is there a way to tell how that would get buffered or how I could unbuffer it? — jwinterm
– jwinterm, Commented May 11, 2020 at 16:09
Yes, it is. If you control the .exe you can flush() after write and it works. I don't remember its exact name, but there is a "is_a_tty" flag in the C libraries that control flush on write (Microsoft has published its clib implementation and you can dig it up). On unix, this is controlled by whether stdout is a tty or pipe. On windows, the console is not a stream - its a 2 dimensional grid of characters and a streaming console is emulated on top of that. But Microsoft never added a way to control this bit, as far as I am aware. — tdelaney
– tdelaney, Commented May 11, 2020 at 16:10
In fact, if memory serves, endl flushes so std::cout << hashes / difference.count() << " H/s" << endl; instead of putting \n in the stream. — tdelaney
– tdelaney, Commented May 11, 2020 at 16:18

tdelaney · Accepted Answer · 2020-05-11 19:46:12Z

Programs based on C libraries treat stdout differently depending on whether its a terminal (assumes user wants data on a line by line basis) or a pipe (block buffer for efficiency). On unix-like systems, a program can be tricked into thinking it is running against a terminal with a pseudo-tty device. On Windows, Microsoft has never implemented an equivalent technology. Subprocesses will block buffer.

You can get around that in the called program by flushing stdout after write. This is done automatically in C++ with endl. In your case that would be:

std::cout << hashes / difference.count() << " H/s" << endl;

Collectives™ on Stack Overflow

Unable to read stdout from spawned subprocess on Windows with Python

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related