2

I have a python module here /root/python/foo.py. I have a bunch of other modules here in the folder /root/lib/ which looks like this


    lib/
    |
    ├─ module1/
    |  ├─ __init__.py
    |  └─ bar.py
    |
    └─ module2/
       ├─ __init__.py
       └─ bar.py

I would like to import /root/lib/module1 and /root/lib/module2 from foo.py. I would like to not have to add /root/lib/ to the python system path. This stack overflow answer tells you how to use either imp.load_source, importlib.machinery.SourceFileLoader, or the importlib.util class to load a module from a file (depending on the python version). I think these only work if the module is a single file. If I try something like this in Python 3.4

from importlib.machinery import SourceFileLoader
problem_module = SourceFileLoader('test_mod', '/root/lib/module1').load_module()

I get an IsADirectoryError

My question is whether there is a similar way to load a module (given its full path) if it is a directory, without adding the whole lib/ folder to the system path?

Improve this question
4
  • add ' after module1 Commented Jul 17, 2016 at 18:51
  • You need to use sys.path to reliably import modules. See this question. Commented Jul 17, 2016 at 18:52
  • 2
    "I would like to not have to add /root/lib/ to the python system path." - Why not? It is by far the easiest way to accomplish this. Commented Jul 17, 2016 at 18:55
  • I didn't want to add the directory to the system path because I thought I might have hundreds of modules in the library and I thought it would be nicer to target just the folder I wanted. Maybe adding the lib folder to the system path isn't as bad as I thought. Commented Jul 17, 2016 at 19:30

2 Answers 2

Reset to default
2

try:

from importlib.machinery import SourceFileLoader
problem_module = SourceFileLoader('test_mod', '/root/lib/module1/__init__.py').load_module()

the __init__.py should take care about the modules in the same package:

add from . import bar to make bar.py part of the package.

Some corrections:

  • module1 is a package not a module.
  • bar.py is a module part of the package module1
Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

That sort of worked. I wanted to use module1 as a module name, e.g. I wanted to also import bar.py from module1 once it was loaded. Targeting the init.py alone didn't let me do that.
from . import bar in __init__.py
alternative solution: from .bar import *
I didn't want to have to import the over modules through the init.py file. I wanted to do something like from problem_module import bar but I guess if I had to do this it wouldn't be too bad, I would just have to change the way I work.
This was a huge help to me, thank you! I ran into an issue trying to import a file foo.bar.py, and importlib interpreted foo, bar, and py as modules.
|
2

Python does not give us an easy way to load files that cannot be referenced by sys.path, most usual solution will do one of the following things:

  1. Add desired paths to sys.path or
  2. Restructure your modules so that the correct path is already on sys.path

Nearly all other solutions that does not do either will be work arounds (using non-intended methods to get the job done) and some can cause quite a headache.

However python does give us a mechanic that lets us simulate a package that is spread out across folders that are not held on sys.path, you can do this by specifying a __path__ special name in a module:

__path__ = ["/root/lib"]

Put this line in a file called lib.py and place it in the same folder as foo.py to be imported by it (so in root/python/ in your case) then from foo.py you can do this as you would expect:

import lib.module1
#or
from lib import module1

This indicates to python that the .module1 subpackage is located somewhere on the specified __path__ and will be loaded from that directory (or multiple directories) using the intended import mechanisms and keeping your sys.path unaltered.

Improve this answer

5 Comments

I think you're right and I might just have to end up adding the lib folder to the system.path. But if this is the right way to go then I'm wondering why python lets you import a module if it's just a file using SourceFileLoader (in python 3.4). If they had that functionality you'd think it wouldn't matter if it's a file or a directory.
Read the name... Source File Loader, you didn't specify a file, and loading a single module from source is not as simple as the whole import mechanism so I wouldn't count on it working 100% in any case.
Yeah, I was just wondering why there wasn't a SourceDirectoryLoader method when there is a SourceFileLoader. But I guess you're right that it's not as simple as just using the import mechanism.
@pspencer I just found an actual solution using intended mechanisms, I think it is exactly what you need.
@Tadhq McDonald-Jensen Was the solution you were talking about the edit you made above about the __path__ special name? That is an really interesting feature of python that I didn't know about but it was exactly what I was looking for. I was looking for a method like SourceFileLoader but for directories. I think what I've decided from this discussion is that there isn't a good pythonic way to do what I want except to just add the lib directory to the system path.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.