running bash commands in Python script, can not find other Python modules

Question

It is sort of difficult to explain in the title, please see below:

A bash script I used to make calls to caffe functions, this specific example trains a model using solver prototxt:

#!/bin/bash

TOOLS=../../build/tools

export HDF5_DISABLE_VERSION_CHECK=1
export PYTHONPATH=.
#for debugging python layer
GLOG_logtostderr=1  $TOOLS/caffe train -solver    lstm_solver_flow.prototxt -weights single_frame_all_layers_hyb_flow_iter_50000.caffemodel  
echo "Done."

I have worked with this many times, no problems. What it does is to use caffe framework's built in functions, such as "train" and pass arguments. The train code is mainly built in C++ but it calls a Python script for a custom data layer. With the shell, everything runs smootly.

Now, I am calling these exact commands in a python script using subprocess.call() with the Shell=True

import subprocess

subprocess.call("export HDF5_DISABLE_VERSION_CHECK=1",shell=True))
subprocess.call("export PYTHONPATH=.",shell=True))
#for debugging python layer
subprocess.call("GLOG_logtostderr=1  sampleexact/samplepath/build/tools/caffe train -solver    lstm_solver_flow.prototxt -weights single_frame_all_layers_hyb_flow_iter_50000.caffemodel",shell=True))

When running the bash commands from within a python script (init),it is able to start the train process, however the train process makes a call to another python module for a custom layer, and can not find it. Both the init and custom layer modules are in the same folder.

How can I fix this problem? I really need to run it from Python so that I can debug. Is there a way to make -any- python module in the project reachable to any calls from others?

ShadowRanger · Accepted Answer · 2017-03-23 12:52:23Z

Each shell=True subprocess command is invoked in a separate shell. What you're doing is configuring a new shell, throwing it away, then starting over fresh with a new shell, over and over. You must perform all the configuration in a single subprocess, not many.

That said, most of what you're doing doesn't need the shell. For example, setting environment variables in the subprocess can be done in Python, without special exports. Example:

# Make a copy of the current environment, then add a few additional variables
env = os.environ.copy()
env['HDF5_DISABLE_VERSION_CHECK'] = '1'
env['PYTHONPATH'] = '.'
env['GLOG_logtostderr'] = '1'

# Pass the augmented environment to the subprocess
subprocess.call("sampleexact/samplepath/build/tools/caffe train -solver    lstm_solver_flow.prototxt -weights single_frame_all_layers_hyb_flow_iter_50000.caffemodel", env=env, shell=True)

Odd are, you don't even need shell=True at this point, and avoiding it is a good idea in general for safety reasons (and a minor performance benefit), so you could just do:

subprocess.call([
    "sampleexact/samplepath/build/tools/caffe", "train", "-solver",
    "lstm_solver_flow.prototxt", "-weights",
    "single_frame_all_layers_hyb_flow_iter_50000.caffemodel"], env=env)

Collectives™ on Stack Overflow

running bash commands in Python script, can not find other Python modules

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related