Python 3 subprocess.PIPE output error

Question

I have a simple script that I use to automate CLI calls to our software (the Moab Workload Manager) in testing, to avoid having to use the '--xml' flag to get xml output and then pipe it through tidy so it's easily readable. It uses a subprocess.Popen call to run the command, then uses str.strip() and str.replace() to do a minor cleanup on the returned xml to make it easy to visually inspect. The code in question is here:

---

cmdString = "%s --xml" % cmd
cmdList = cmdString.split()

cmdRun = subprocess.Popen(cmdList,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE)

crOut,crErr = cmdRun.communicate()

xmlOutput = crOut.strip().replace("><",">\n<").replace("\" ","\"\n")

---

When I run this (I recently upgraded my Python to Python 3.1.2) I now get the following error:

---

Traceback (most recent call last):
  File "/usr/local/bin/xmlcmd", line 50, in <module>
    runXMLCmd(getCmd())
  File "/usr/local/bin/xmlcmd", line 45, in runXMLCmd
    xmlOutput = crOut.strip().replace("><",">\n<")
TypeError: expected an object with the buffer interface

---

It appears that the communicate() call is returning byte arrays, but in the python interpreter, if I do dir(bytes) I can still see the strip() and replace() functions. Anybody know how to make this right?

Thanks.

Answer 1

bytes.replace() expects bytes as arguments:

crOut.strip().replace(b"><", b">\n<").replace(b"\" ", b"\"\n")

Though in general it could be preferable to decode the input to unicode text as early as possible. And the transformations to be performed on the text (not bytes).

Answer 2

You need to use crOut.decode("utf-8") and do the .replace in the returned string.

Answer 3

In python 3.2, using decode('ascii') fixed some unicode and type errors which were difficult to trace. Regardless of byte or bytearray the decode will convert to the string as desired.

pipe = subprocess.Popen("cmd", 1024, stdout=subprocess.PIPE)
while pipe.returncode == None:
    lines = pipe.communicate()[0]
    lines = lines.decode('ascii')
    for line in lines.splitlines():
        if (re.search('^---', line)):
            pass # do stuff

From the manual,

bytes.decode(encoding="utf-8", errors="strict") 
bytearray.decode(encoding="utf-8", errors="strict") 
Return a string decoded from the given bytes. 
Default encoding is 'utf-8'. errors may be given to set a 
different error handling scheme.