Python: Executing a shell command

Question

I need to do this:

paste file1 file2 file3 > result

I have the following in my python script:

from subprocess import call

// other code here.

// Here is how I call the shell command

call ["paste", "file1", "file2", "file3", ">", "result"])

Unfortunately I get this error:

paste: >: No such file or directory.

Any help with this will be great!

Answer 1

You need to implement the redirection yourself, if you're wisely deciding not to use a shell.

The docs at https://docs.python.org/2/library/subprocess.html warn you not to use a pipe -- but, you don't need to:

import subprocess
with open('result', 'w') as out:
    subprocess.call(["paste", "file1", "file2", "file3"], stdout=out)

should be just fine.

Answer 2

There are two approaches to this.

1. Use shell=True:

   call("paste file1 file2 file3 >result", shell=True)
   

   Redirection, >, is a shell feature. Consequently, you can only access it when using a shell: shell=True.

2. Keep shell=False and use python to perform the redirection:

   with open('results', 'w') as f:
       subprocess.call(["paste", "file1", "file2", "file3"], stdout=f)
   

The second is normally preferred as it avoids the vagaries of the shell.

Discussion

When the shell is not used, > is just another character on the command line. Thus, consider the error message:

paste: >: No such file or directory. 

This indicates that paste had received > as an argument and was trying to open a file by that name. No such file exists. Therefore the message.

As the shell command line, one can create a file by that name:

touch '>'

If such a file had existed, paste, when called by subprocess with shell=False, would have used that file for input.

Answer 3

If you don't mind adding an additional dependency in your code base you might consider installing the sh Python module (from PyPI:sh using pip, of course).

This is a rather clever wrapper around Python's subprocess module's functionality. Using sh your code would look something like:

#!/usr/bin/python
from sh import paste
paste('file1', 'file2', 'file3', _out='result')

... although I think you'd want some exception handling around that so you could use something like:

#!/usr/bin/python
from __future__ import print_function
import sys
from sh import paste
from tempfile import TemporaryFile
with tempfile.TemporaryFile() as err:
    try:
        paste('file1', 'file2', 'file3', _out='result', _err=err)
    except (EnvironmentError, sh.ErrorReturnCode) as e:
        err.seek(0)
        print("Caught Error: %s" % err.read(), file=sys.stderr)

sh makes such things almost trivially easy although there are some tricks as you get more advanced. You also have to note the difference between _out= and other keyword arguments of that form, vs. sh's magic for most other keyword arguments.

All that sh magic make confuse anyone else who ever reads your code. You might also find that using Python modules with sh code interlaced into it makes you complacent about portability issues. Python code is generally fairly portable while Unix command line utilities can vary considerably from one OS to another and even from one Linux distribution or version to another. Having lots of shell utilities interlaced with your Python code in such a transparent way may make that problem less visible.