Importing python modules from multiple directories

Question

I have a python 2.6 Django app which has a folder structure like this:

/foo/bar/__init__.py

I have another couple directories on the filesystem full of python modules like this:

/modules/__init__.py
/modules/module1/__init__.py
/other_modules/module2/__init__.py
/other_modules/module2/file.py

Each module __init__ has a class. For example module1Class() and module2Class() respectively. In module2, file.py contains a class called myFileClass().

What I would like to do is put some code in /foo/bar/__init__.py so I can import in my Django project like this:

from foo.bar.module1 import module1Class
from foo.bar.module2 import module2Class
from foo.bar.module2.file import myFileClass

The list of directories which have modules is contained in a tuple in a Django config which looks like this:

module_list = ("/modules", "/other_modules",)

I've tried using __import__ and vars() to dynamically generate variables like this:

import os
import sys 

for m in module_list:
    sys.path.insert(0, m)
    for d in os.listdir(m):
        if os.path.isdir(d):
            vars()[d] = getattr(__import__(m.split("/")[-1], fromlist=[d], d)

But that doesn't seem to work. Is there any way to do this?

Thanks!

The design is such that you configure a list of directories to find modules, and when you start your web server all of the modules are immediately available to you. For ease of installation of the core app, the modules, development of new modules, and separation of internally and externally developed modules, we wanted to keep this separate. It makes it easy to git clone modules to any directory of your choosing, and then point at it instead of having to mess around in the core project. — mgoffin
– mgoffin, Commented Jun 26, 2013 at 12:46

Aya · Accepted Answer · 2013-06-26 14:48:03Z

I can see at least one problem with your code. The line...

if os.path.isdir(d):

...won't work, because os.listdir() returns relative pathnames, so you'll need to convert them to absolute pathnames, otherwise the os.path.isdir() will return False because the path doesn't exist (relative to the current working directory), rather than raising an exception (which would make more sense, IMO).

The following code works for me...

import sys
import os

# Directories to search for packages
root_path_list = ("/modules", "/other_modules",)

# Make a backup of sys.path
old_sys_path = sys.path[:]

# Add all paths to sys.path first, in case one package imports from another
for root_path in root_path_list:
    sys.path.insert(0, root_path)

# Add new packages to current scope
for root_path in root_path_list:
    filenames = os.listdir(root_path)
    for filename in filenames:
        full_path = os.path.join(root_path, filename)
        if os.path.isdir(full_path):
            locals()[filename] = __import__(filename)

# Restore sys.path
sys.path[:] = old_sys_path

# Clean up locals
del sys, os, root_path_list, old_sys_path, root_path, filenames, filename, full_path

Update

Thinking about it, it might be safer to check for the presence of __init__.py, rather than using os.path.isdir() in case you have subdirectories which don't contain such a file, otherwise the __import__() will fail.

So you could change the lines...

        full_path = os.path.join(root_path, filename)
        if os.path.isdir(full_path):
            locals()[filename] = __import__(filename)

...to...

        full_path = os.path.join(root_path, filename, '__init__.py')
        if os.path.exists(full_path):
            locals()[filename] = __import__(filename)

...but it might be unnecessary.

I'll give this a shot when I get a chance and let you know if that worked for me as well! Thanks!
This doesn't seem to work at all. The code runs without error, but you still can't do any importing.
@mgoffin Hmm. It loads the package into the namespace, but won't import any submodules of the package. You might be able to do something with import hooks, but it seems like a lot of hassle. Why exactly do you need to scope the external modules inside the foo.bar package? What's wrong with sticking the packages' parent directories into sys.path, then doing from module1 import module1Class, rather than from foo.bar.module1 import module1Class?
Each module should be structured to have certain things the main project expects. For example, a standard views.py, urls.py, etc. That means there would be foo.bar.module1.urls and foo.bar.module2.urls. In order to programmatically loop over them all it would be tough to use 'from' to import and not have collisions.
@mgoffin Given from foo.bar.module1 import urls; from foo.bar.module2 import urls and from module1 import urls; from module2 import urls you'd get a collision on the symbol urls either way. If these modules are Django apps, should they not go in INSTALLED_APPS in settings.py?

mgoffin · Accepted Answer · 2013-07-05 01:45:34Z

We wound up biting the bullet and changing how we do things. Now the list of directories to find modules is passed in the Django config and each one is added to sys.path (similar to a comment Aya mentioned and something I did before but wasn't too happy with). Then for each module inside of it, we check for an __init__.py and if it exists, attempt to treat it as a module to use inside of the app without using the foo.bar piece.

This required some adjustment on how we interact with the modules and how developers code their modules (they now need to use relative imports within their module instead of the full path imports they used before) but I think this will be an easier design for developers to use long-term.

We didn't add these to INSTALLED_APPS because we do some exception handling where if we cannot import a module due to dependency issues or bad code our software will continue running just without that module. If they were in INSTALLED_APPS we wouldn't be able to leverage that flexibility on when/how to deal with those exceptions.

Thanks for all of the help!

Collectives™ on Stack Overflow

Importing python modules from multiple directories

2 Answers 2

6 Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Related