2

I wonder how to automatically install all the pip packages used by a Python script. I know one can run:

pip install pipreqs
pipreqs .
pip install -r requirements.txt

But that sometimes fails to capture all packages and all proper packages versions.

Instead, I'd like some more solid solution that tries to run the Python script, catch missing package errors and incorrect package versions such as:

ImportError: peft>=0.17.0 is required for a normal functioning of this module, but found peft==0.14.0.

install these packages accordingly and retry run the Python script until it works or caught in a loop.

I use Ubuntu.

Improve this question
4
  • 1
    This is in general impossible, since imports can be conditional. Commented Oct 17 at 3:08
  • @muru Thanks good point, assume the script is deterministic and takes no user input. Commented Oct 17 at 3:09
  • 1
    Even so, import errors can be caught just like any other exception, and can depend on any number of external factors, so there's no guarantee that a script not raising an import error that the user can see in N executions will continue to work fine in the N+1th execution. Commented Oct 17 at 3:29
  • @muru true, you're right, let's ignore these cases Commented Oct 17 at 3:32

1 Answer 1

Reset to default
5 
pip install pipreqs
pipreqs .

This is a measure of last resort. Whoever wrote that script should have written down the requirements – as you can see, the pipreqs program is not very good at that (which is pretty much impossible to be good at, because there's no way to know which pypi module includes which python module, so chances are you'll be wrong all the time).

So, reality here is that you're trying to automate one of the core jobs of the developer of whatever software you're trying to use.

Seeing peft: The machine learning crowd is typically pretty OK about this; make sure you're not just blindly taking random files from complete modules that do properly declare their dependencies.

But that sometimes fails to capture all packages and all proper packages versions.

That means the requirements.txt (or whatever specifies the requirements for the different tools) is incomplete.

Nothing you can really fix in the general case – typically, this happens when a project imports something in a specific case and the developer forgets to include that in the requirements.txt (or other appropriate metadata). Knowing all specific cases would require developer knowledge, or running all possible code paths (which is infeasible).

ImportError: peft>=0.17.0 is required for a normal functioning of this module, but found peft==0.14.0.

That's a very specific error. I don't think it's generally worth trying to automate this:

XKCD #1319, "Automation"

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.