ImportError: No module named … error in Python - set PYTHONPATH still not working

Question

I have a problem similar to that described here How to fix "ImportError: No module named ..." error in Python? but I cannot fix it with the suggestion to set PYTHONPATH.

my directory looks like:

- project
    - python
        - src
          - ml
            - __init__.py
            - classifier_dnn.py
            - util.py           
            - vectorizer
              - fv_davison.py
              - __init__.py

And I am running classifier_dnn.py at the project folder path:

~project&PYTHONPATH=/home/project/
~project$python3 /home/project/python/src/ml/classifier_dnn.py /home/project/data/labeled_data_all.csv /home/project/output

But an error is generated when classifier_dn imports ml.util:

Traceback (most recent call last):
  File "/home/project/chase/python/src/ml/classifier_dnn.py", line 5, in <module>
    from ml import util
ImportError: No module named 'ml'

I have also tried setting PYTHONPATH=/home/project/python or PYTHONPATH=/home/project/src but the same error happens.

When I test this in PyCharm, it works if set python/src to become source root, regardless what working directory is. But I cannot figure out how to set this properly when I run this from command line.

Any help please

Thanks

Answer 1

I have a writeup on this but I'll copy the relevant text inline.

You have two problems:

• You need to export PYTHONPATH (export PYTHONPATH=/full/path/to/src)
• You should run with python -m module.path instead of python path/to/file.py

The core problem is python path/to/file.py puts path/to on the beginning of the PYTHONPATH (sys.path)

This causes imports to start from (in your case) src/ml instead of the expected src.

By using -m, you avoid this path munging and it will correctly keep src as the beginning of your sys.path.

When I test this in PyCharm, it works if set python/src to become source root, regardless what working directory is. But I cannot figure out how to set this properly when I run this from command line.

You can do this by cding into the src directory

Answer 2

You should not put executable scripts into a library structure.

Only library modules should go into the package structure. Executables should stay outside it. This is important for installation and day-to-day use. Once you have installed your package (using distutils for example) using it would be complicated, if you leave the executable in the library. You would have to give the complete installation path into the python library (and don't forget about -m!) every time or create shortcuts or other workarounds.

Seperate executables and library. Executables are installed to something like usr/local/bin and are callable without path, -m or other problems, and your library is installed into the python library.