Python Not Finding Module

Question

Given the following python project, created in PyDev:

├── algorithms
│   ├── __init__.py
│   └── neighborhood
│       ├── __init__.py
│       ├── neighbor
│       │   ├── connector.py
│       │   ├── __init__.py
│       │   ├── manager.py
│       │   └── references.py
│       ├── neighborhood.py
│       ├── tests
│       │   ├── fixtures
│       │   │   └── neighborhood
│       │   ├── __init__.py
│       └── web
│           ├── __init__.py
│           └── service.py
├── configuration
│   ├── Config.py
│   └── __init__.py
├── __init__.py
└── webtrack
    |- teste.py
    ├── .gitignore
    ├── __init__.py
    ├── manager
        ├── Data.py
        ├── ImportFile.py
        └── __init__.py

We've been trying with no success to import modules from one folder to another, such as:

from algorithms.neighborhood.neighbor.connector import NeighborhoodConnector

Which yields the result:

Traceback (most recent call last):
File "teste.py", line 49, in <module>
from algorithms.neighborhood.neighbor.connector import NeighborhoodConnector
ImportError: No module named algorithms.neighborhood.neighbor.connector

We tried to append its path to the sys.path variable but with no success.

We also tried to use os.walk to insert all paths into PATH variable but still we get the same error, even though we checked PATH does contain the path to find the modules.

We are using Python 2.7 on Linux Ubuntu 13.10.

Is there anything we could be doing wrong?

Thanks in advance,

Answer 1

Getting imports right when running a script that lives within a package is tricky. You can read this section of the (sadly deferred) PEP 395 for a description of a bunch of ways that don't work to run such a script.

Give a file system hierarchy like:

top_level/
    my_package/
        __init__.py
        sub_package/
            __init__.py
            module_a.py
            module_b.py
            sub_sub_package/
                __init__.py
                module_c.py
        scripts/
            __init__.py
            my_script.py
            script_subpackage/
                 __init__.py
                 script_module.py

There are only a few ways to make running my_script.py work right.

1. The first would be to put the top_level folder into the PYTHONPATH environment variable, or use a .pth file to achieve the same thing. Or, once the interpreter is running, insert that folder into sys.path (but this can get ugly).

   Note that you're adding top_level to the path, not my_package! I suspect this is what you've got messed up in your current attempts at this solution. Its very easy to get wrong.

   Then, absolute imports like import my_package.sub_package.module_a will mostly work correctly. (Just don't try importing package.scripts.my_script itself while it is running as the __main__ module, or you'll get a weird duplicate copy of the module.)

   However, absolute imports will always be more verbose than relative imports, since you always need to specify the full path, even if you're importing a sibling module (or "niece" module, like module_c from module_a). With absolute imports, the way to get module_c is always the big, ugly mouthful of code from my_package.sub_package.sub_sub_package import module_c regardless of what module is doing the importing.

2. For that reason, using relative imports is often more elegant. Alas, they're hard to get to work from a script. The only ways are:

   1. Run my_script from the top_level folder with the -m flag (e.g. python -m my_package.scripts.my_script) and never by filename.

      It won't work if you're in a different folder, or if you use a different method to run the script (like pressing F5 in an IDE). This is somewhat inflexible, but there's not really any way to make it easier (until PEP 395 gets undeferred and implemented).

   2. Set up sys.path like for absolute imports (e.g. add top_level to PYTHONPATH or something), then use a PEP 366 __package__ string to tell Python what the expected package of your script is. That is, in my_script.py you'd want to put something like this above all your relative imports:

      if __name__ == "__main__" and __package__ is None:
          __package__ = "my_package.my_scripts"
      

      This will require updating if you reorganize your file organization and move the script to a different package (but that's probably less work than updating lots of absolute imports).

   Once you've implemented one of those soutions, your imports can get simpler. Importing module_c from module_a becomes from .sub_sub_package import module_c. In my_script, relative imports like from ..subpackage import module_a will just work.

Answer 2

I know this is an old post but still I am going to post my solution.

Had a similar issue. Just added the paths with the following line before importing the package:

sys.path.append(os.path.join(os.path.dirname(__file__), '..')) 
from lib import create_graph

Answer 3

The way imports work is slightly different in Python 2 and 3. First Python 3 and the sane way (which you seem to expect). In Python 3, all imports are relative to the folders in sys.path (see here for more about the module search path). Python doesn't use $PATH, by the way.

So you can import anything from anywhere without worrying too much.

In Python 2, imports are relative and sometimes absolute. The document about packages contains an example layout and some import statements which might be useful for you.

The section "Intra-package References" contains information about how to import between packages.

From all the above, I think that your sys.path is wrong. Make sure the folder which contains algorithms (i.e. not algorithms itself but it's parent) needs to be in sys.path

Answer 4

Just set __package__ = None in every .py file. It will setup all the package hierarchy automatically.

After that you may freely use absolute module names for import.

from algorithms.neighborhood.neighbor.connector import NeighborhoodConnector