16

Is there a way to specify conditional installs in a pip requirements.txt file that are based on the value of an environment variable?

I've been able to control most of what I need with environment markers, but all the markers I know of work only with specific values that are essentially pre-defined by python.

For example, I want to be able to control package installation for RHEL 5.3 vs. RHEL 6.3 vs. RHEL 6.6, etc. Also based on other criteria.

It would be perfect if I could specify in the results.txt file an environment variable that would be checked against a value I set before running pip. This seems like it would be desirable and straight forward functionality. I haven't had much luck so far finding comprehensive discussions of environment markers, so I'm hoping I've just missed some key bit of information :-)

Thanks much.

Improve this question

3 Answers 3

Reset to default
7

There's not really a way to do it with environment variables. Pip requirements files are basically just a list of pip install arguments listed in a file. So if your requirements file looks like this:

Foo==1.1.0
Bar==0.1.0
Baz==2.0.1

Logically, pip is doing this:

pip install Foo==1.1.0
pip install Bar==0.1.0
pip install Baz==2.0.1

Unfortunately, in that context there's no way to apply environment variables.

There are a couple solutions to this problem.

One, you could have a base requirements file, say requirements.txt, that lists common dependencies for all platforms. Then, you could have individual requirements files that are named, say, requirements.rhel53.txt, requirements.rhel63.txt, etc. The top of each of these files could have this as the first line:

-r requirements.txt

And then additional special dependencies listing. Then, in each environment, you could set an env var, let's call it $PLATFORM, and run a command like this to install dependencies:

$ pip install -r requirements.$PLATFORM.txt

Or, you could use constraints files. Your requirements.txt would just list dependencies without versions:

Foo
Bar
Baz

And then you could have a constraints file, again for each platform, that would list specific version requirements. For example, you could have constraints.rhel53.txt that had this:

Foo==1.1.0
Bar==0.1.0
Baz==2.0.1

And again, you set an env var and then run a command like this:

$ pip install -r requirements.txt -c constraints.$PLATFORM.txt

It's a cumbersome solution, but that would be one way of dealing with it. Unfortunately, pip does not have a native solution.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Thanks for the feedback mipadi. I was afraid of that. Since, as I understand it, the pip environment markers are really just equivalents to associate functionality in python (like platform_system is really platform.system() in python,) I was hoping there was a way to have a marker equivalent to something like os.environ.get('MYVARIABLE'). The pain I have is that there are multiple dimensions to my requirement criteria, so using multiple requirement files gets a bit messy to maintain. Using the constraints files is a good idea. I'll try to at least reduce the complexity that way.
Using different constraint files could work around my problem fairly smoothly except that I can't find a way to make a constraint file completely disallow a package. Is there a way? I tried working around this by specifying a non existent version (e.g. six <= 0.1) but then pip errors because it can't find an acceptable version to install. Is there a way to tell pip not to error out, or to warn rather than error?
I also tried using an environment marker that evaluates to false (e.g. six; sys_platform == 'win32' in a linux environment). But instead of constraining the installation of the package, pip constrains (ignores) this line of the constraint file and installs the latest version of the package (because it's unconstrained in the requirements.txt file.)
1

I ended up with a scripted solution similar to one I found in: Is there a way to have a conditional requirements.txt file for my Python application based on platform?

Basically, I did something like the following (partial) example, which allows me to define packages as needed in the script, and still pass in requirements.txt and constraints.txt files. I'm new to Python, so please forgive (or comment on) any less-than-optimal coding practices:

import pip, sys, getopt

def install(options, packages):
    pipargs = options + packages
    pip.main(pipargs)

constraints_file = ''
requirements_file = ''

try:
    opts, args = getopt.getopt(sys.argv[1:],"hr:c:",["help","requirements-file=","constraints-file="])
except getopt.GetoptError:
    print_usage()
    sys.exit(2)
for opt, arg in opts:
    if opt in ("-h", "--help"):
       print_usage()
       sys.exit()
    elif opt in ("-r", "--requirements-file"):
        requirements_file = arg
    elif opt in ("-c", "--constraints-file"):
        constraints_file = arg

base_pkgs = [
    "nose>=1.3.4",
    "wheel",
    "pytz"
]

foo_pkgs = [
    "Pygments; python_version == '2.7'",
    "rednose; python_version == '3.5'",
]

bar_pkgs = [
    "six",
]

if __name__ == '__main__':

    from os import environ as env

    myvar = env.get('MYVAR')
    if(myvar == 'foo'):
        mypkgs = foo_pkgs
    elif(myvar == 'bar'):
        mypkgs = bar_pkgs
    else:
        mypkgs = ''

    pkglist = base_pkgs + mypkgs

    pipoptions = [  'install',
                    '--upgrade',
                    '--force-reinstall'
                 ]

    if(constraints_file):
        pipoptions += ['-c', constraints_file]
    if(requirements_file):
        pipoptions += ['-r', requirements_file]

    install(pipoptions, pkglist)
Improve this answer

1 Comment

This was extreemely helpful for me. Was not aware packages could be installed using pip within a python script. neat!
1

This is an old question, but to complete the set of answers ever since v10.0 pip supports variable expansion in environment files as documented here:

pip supports the use of environment variables inside the requirements file.

You have to use the POSIX format for variable names including brackets around the uppercase name as shown in this example: ${API_TOKEN}. pip will attempt to find the corresponding environment variable defined on the host system at runtime.

Note: There is no support for other variable expansion syntaxes such as $VARIABLE and %VARIABLE%.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.