Calling bash command inside Python returns error, but works in terminal

Question

Here is the except of my code related to this:

def grd_commands(directory):
    for filename in os.listdir(directory)[1:]:
        print filename
        new_filename = ''
        first_letter = ''
        second_letter = ''
        bash_command = 'gmt grdinfo ' + filename + ' -I-'
        print bash_command
        coordinates = Popen(bash_command, stdout=PIPE, shell=True)
        coordinates = coordinates.communicate()
        latlong = re.findall(r'^\D*?([-+]?\d+)\D*?[-+]?\d+\D*?([-+]?\d+)', coordinates)
        if '-' in latlong[1]:
            first_letter = 'S'
        else:
            first_letter = 'N'
        if '-' in latlong[0]:
            second_letter = 'W'
        else:
            second_letter = 'E'

        new_filename = first_letter + str(latlong[1]) + second_letter + str(latlong[0]) + '.grd'
        Popen('gmt grdconvert ' + str(filename) + ' ' + new_filename, shell=True)

filenameis the name of the file that is is being passed to the function. When I run my code, I am receiving this error:

/bin/sh: gmt: command not found
Traceback (most recent call last):
  File "/Users/student/Desktop/Code/grd_commands.py", line 38, in <module>
    main()
  File "/Users/student/Desktop/Code/grd_commands.py", line 10, in main
    grd_commands(directory)
  File "/Users/student/Desktop/Code/grd_commands.py", line 23, in grd_commands
    latlong = re.findall(r'^\D*?([-+]?\d+)\D*?[-+]?\d+\D*?([-+]?\d+)', coordinates).split('\n')
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/re.py", line 177, in findall
    return _compile(pattern, flags).findall(string)
TypeError: expected string or buffer

If I print out the string bash_command and try entering it into terminal it fully functions. Why doesn't it work when being called by my Python script?

Most likely this is a path issue; in which directory is gmt found? — chepner
– chepner, Commented Jan 3, 2017 at 20:16
If you pass a single string to Popen it will do different things on Unix vs Windows and vs typing that string at the command prompt. You should pass a list with the first element the path name to the command. You may also have a different path from within Python vs at the Bash prompt. Add the path to gmt in the first element of the list. Popen docs — dawg
– dawg, Commented Jan 3, 2017 at 20:16
You are running sh, not bash - /bin/sh: gmt: command not found suggests that the directory that gmt resides in is not part of your PATH environment variable. Try giving the full pathname of the gmtprogram. — cdarke
– cdarke, Commented Jan 3, 2017 at 20:17
@cdarke I gave it the full path of gmt. I think it might be that i'm not handling storing the output properly. Here's what the output of the command is in the shell pastebin.com/v2941QCJ — Harrison
– Harrison, Commented Jan 3, 2017 at 20:19
Print coordinates. It's a tuple of stdout and stderr. You want coordinates[0]. — tdelaney
– tdelaney, Commented Jan 3, 2017 at 20:26

chepner · Accepted Answer · 2017-01-03 20:12:19Z

2

The entire command line is being treated as a single command name. You need to either use shell=True to have the shell parse it as a command line:

coordinates = Popen(bash_command, stdout=PIPE, shell=True)

or preferably store the command name and its arguments as separate elements of a list:

bash_command = ['gmt', 'grdinfo', filename, '-I-']
coordinates = Popen(bash_command, stdout=PIPE)

answered Jan 3, 2017 at 20:12

chepner

538k77 gold badges594 silver badges746 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Harrison Over a year ago

Thank you for your answer. I quickly realized I needed to add shell = True to my parameters after posting the question, and updated the question to handle the actual error I was getting after doing that.

chepner Over a year ago

I still strongly suggest using the list argument, though. (I'll delete this answer in a bit.)

Bill Schumacher · Accepted Answer · 2017-01-03 23:50:13Z

Popen takes a list of arguments. There is a warning for using shell=True

Passing shell=True can be a security hazard if combined with untrusted input.

Try this:

from subprocess import Popen, PIPE
bash_command = 'gmt grdinfo ' + filename + ' -I-'
print(bash_command)
coordinates = Popen(bash_command.split(), stdout=PIPE)
print(coordinates.communicate()[0])

Ensure gmt is installed in a location specified by PATH in your /etc/environment file:

PATH=$PATH:/path/to/gmt

Alternatively, specify the path to gmt in bash_command:

bash_command = '/path/to/gmt grdinfo ' + filename + ' -I-'

You should be able to find the path with:

which gmt

As other people have suggested, an actual list would be the best approach instead of a string. Additionally, you must escape spaces with a '\' in order to actually access the file if there is a space in it.

for filename in os.listdir(directory)[1:]:
    bash_command = ['gmt', 'grdinfo', filename.replace(" ", "\ "), '-I-']

Simply splitting the string on whitespace is not a good idea; suppose the file name is foo bar.txt?

Collectives™ on Stack Overflow

Calling bash command inside Python returns error, but works in terminal

2 Answers 2

2 Comments

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Related