3

I need to change which program is called by a Python application. Unfortunately I cannot change the Python code. I can only change the calling environment (in particular, PATH). But unfortunately Python’s subprocess module seems to ignore PATH (at least under certain circumstances).

How can I force Python to respect PATH when searching which binary to invoke?

To illustrate the problem, here’s an MVCE. The actual Python application is using subprocess.check_output(['nvidia-smi', '-L']), but the following simplified code shows the same behaviour.

Create test.py:

import os
from subprocess import run

run(['which', 'whoami'])
run(['/usr/bin/env', 'whoami'])
run(['whoami'])

os.execvp('whoami', ['whoami'])

Now create a local whoami script and execute test.py:

echo 'echo foobar' >whoami
chmod +x whoami
PATH=.:$PATH python3 test.py

On my system1 this prints:

./whoami
foobar
konrad
konrad

I expect this code to always print foobar instead of konrad.

My MVCE includes the os.execvp call because the subprocess documentation states that

On POSIX, the class uses os.execvp()-like behavior to execute the child program.

Needless to say, the actual execvp POSIX API, called from C, does respect PATH, so this is a Python specific issue.

1 Ubuntu 18.04.2 LTS, Python 3.6.9.

Improve this question
6
  • 2
    if I put #!/bin/sh at the beginning of your whoami script I get your expected behaviour, is omitting that deliberate? running it without gives ENOEXEC (Exec format error) on the script Commented Jul 14, 2020 at 11:17
  • 1
    I can reproduce the problem and adding #!/bin/sh fixes it. Which makes sense - in strace I see execve("./whoami", ...) = -1 ENOEXEC (Exec format error) because of the missing shebang kernel can't execute the file - so /usr/bin/whoami is chosen. Commented Jul 14, 2020 at 11:21
  • 1
    @KamilCuk note that strace uses execve while OP appears to be deliberately using execvp. the latter which should attempt to interpret using /bin/sh if it gets ENOEXEC when executing directly Commented Jul 14, 2020 at 11:31
  • os.py#L608 _execvpe() I guess a except (ENOEXEC): exec("sh", file...) is missing Commented Jul 14, 2020 at 11:37
  • @SamMason It wasn’t deliberate, but I thought it shouldn’t be necessary (and isn’t for POSIX’ execvp!). But that indeed fixes my issue, and if you write it up as an answer I can accept that! I’m still curious why it’s required here when it isn’t for the POSIX function (though thinking about it now I’m more surprised that the POSIX function doesn’t require it). Commented Jul 14, 2020 at 11:48

2 Answers 2

Reset to default
2

as per my comment, this is due to Python's implementation of execvp not being consistent with POSIX execvp semantics. in particular Python doesn't respond to ENOEXEC errors by interpreting the file as a shell script and needs an explicit shebang.

creating the file as:

printf '#!/bin/sh\necho foobar\n' > ./whoami

causes things to work as expected

note that this has been known about for a while: https://bugs.python.org/issue19948

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

Your

echo 'echo foobar' >whoami
chmod +x whoami

is not running correctly.

Python is not picking it up a executable, even though though the execute bit is set, it doesn't know it needs to run bash first, to execute, thus is skips in on the path and runs the original whoami which is pathed /usr/bin/whoami

adding shebang

echo "#!/bin/sh" > whoami
echo 'echo foobar' >> whoami
chmod +x whoami

On Unix-style systems (including Linux/OS X), the shebang line—as it’s called—tells the loader (or kernel, or occasionally shell) which program to use to run the file. At its most basic, you’d specify a path to the python interpreter.

i suspect if you ./whoami (with execute permission set), the shell is doing some extra magic so you don't have to type /bin/sh $PWD/whoami

if you do

chmod -x whoami

you can use the special

. ./whoami (tell the shell to execute it as a shell script).

note that execvp should use /bin/sh rather than bash. also . ./whoami will depend on what shell you happen to be using, most will "source" the file rather than executing it in another process (i.e. changes to environment, working directory, etc will remain)

If no shebang or executable header, the shell simply uses itself as the default interpreter (but only when invoked via ./whoami; . ./whoami is different, it sources the file, regardless of whether it’s executable).

The confusing nature of execvp (POSIX, not Python) apparently also does this. Python fails in this case because os.execvp actually does not call execvp under the hood, it similarities are just in name.

Improve this answer

3 Comments

note that execvp should use /bin/sh rather than bash. also . ./whoami will depend on what shell you happen to be using, most will "source" the file rather than executing it in another process (i.e. changes to environment, working directory, etc will remain)
Yep, the shell simply uses itself as the default interpreter if no shebang or executable header is given (but only when invoked via ./whoami; . ./whoami is different, it sources the file, regardless of whether it’s executable). What confused me is that execvp (POSIX, not Python) apparently also does this. And Python fails because os.execvp actually does not call execvp.
Great comments, i will add that to my answer.. thanks @SamMason and Konrad RuDolph

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.