Import local function from a module housed in another directory with relative imports in Jupyter Notebook using Python 3

I use this project structure:

my_project
├── nb
│   ├── 20170609-Examine_Database_Requirements.ipynb
│   └── 20170609-Initial_Database_Connection.ipynb
└── my_package
    ├── __init__.py
    └── postgres.py

And from a notebook:

    In [1]: import os
            os.chdir(os.path.expanduser("~/location_of/my_project")

    In [2]: from my_package.postgres import database_connection

All other answers here depends on adding code the the notebook(!)

In my opinion is bad practice to hardcode a specific path into the notebook code, or otherwise depend on the location, since this makes it really hard to refactor you code later on. Instead I would recommend you to add the root project folder to PYTHONPATH when starting up your Jupyter notebook server, either directly from the project folder like so

env PYTHONPATH=`pwd` jupyter notebook

or if you are starting it up from somewhere else, use the absolute path like so

env PYTHONPATH=/Users/foo/bar/project/ jupyter notebook

So far, the accepted answer has worked best for me. However, my concern has always been that there is a likely scenario where I might refactor the notebooks directory into subdirectories, requiring to change the module_path in every notebook. I decided to add a python file within each notebook directory to import the required modules.

Thus, having the following project structure:

project
|__notebooks
   |__explore
      |__ notebook1.ipynb
      |__ notebook2.ipynb
      |__ project_path.py
   |__ explain
       |__notebook1.ipynb
       |__project_path.py
|__lib
   |__ __init__.py
   |__ module.py

I added the file project_path.py in each notebook subdirectory (notebooks/explore and notebooks/explain). This file contains the code for relative imports (from @metakermit):

import sys
import os

module_path = os.path.abspath(os.path.join(os.pardir, os.pardir))
if module_path not in sys.path:
    sys.path.append(module_path)

This way, I just need to do relative imports within the project_path.py file, and not in the notebooks. The notebooks files would then just need to import project_path before importing lib. For example in 0.0-notebook.ipynb:

import project_path
import lib

The caveat here is that reversing the imports would not work. THIS DOES NOT WORK:

import lib
import project_path

Thus care must be taken during imports.

I have just found this pretty solution:

import sys; sys.path.insert(0, '..') # add parent folder path where lib folder is
import lib.store_load # store_load is a file on my library folder

You just want some functions of that file

from lib.store_load import your_function_name

If python version >= 3.3 you do not need init.py file in the folder

If you are working via the Jupyter extension in VS Code, I found that you can now set this path in your settings.json file.

{
    "jupyter.notebookFileRoot": "/path/to/your/module/root"
}

Edit: Or, to set it at your workspace root more generically:

{
    "jupyter.notebookFileRoot": "${workspaceFolder}"
}

I have found that python-dotenv helps solve this issue pretty effectively. Your project structure ends up changing slightly, but the code in your notebook is a bit simpler and consistent across notebooks.

For your project, do a little install.

pipenv install python-dotenv

Then, project changes to:

├── .env (this can be empty)
├── ipynb
│   ├── 20170609-Examine_Database_Requirements.ipynb
│   └── 20170609-Initial_Database_Connection.ipynb
└── lib
    ├── __init__.py
    └── postgres.py

And finally, your import changes to:

import os
import sys

from dotenv import find_dotenv


sys.path.append(os.path.dirname(find_dotenv()))

A +1 for this package is that your notebooks can be several directories deep. python-dotenv will find the closest one in a parent directory and use it. A +2 for this approach is that jupyter will load environment variables from the .env file on startup. Double whammy.

Researching this topic myself and having read the answers I recommend using the path.py library since it provides a context manager for changing the current working directory.

You then have something like

import path
if path.Path('../lib').isdir():
    with path.Path('..'):
        import lib

Although, you might just omit the isdir statement.

Here I'll add print statements to make it easy to follow what's happening

import path
import pandas

print(path.Path.getcwd())
print(path.Path('../lib').isdir())
if path.Path('../lib').isdir():
    with path.Path('..'):
        print(path.Path.getcwd())
        import lib
        print('Success!')
print(path.Path.getcwd())

which outputs in this example (where lib is at /home/jovyan/shared/notebooks/by-team/data-vis/demos/lib):

/home/jovyan/shared/notebooks/by-team/data-vis/demos/custom-chart
/home/jovyan/shared/notebooks/by-team/data-vis/demos
/home/jovyan/shared/notebooks/by-team/data-vis/demos/custom-chart

Since the solution uses a context manager, you are guaranteed to go back to your previous working directory, no matter what state your kernel was in before the cell and no matter what exceptions are thrown by importing your library code.

If you are using jupyter in vsCode, you can do this by adding this to settings.json

    "jupyter.notebookFileRoot": "${workspaceFolder}"

Then import from globally. For example,

from project1.lib.module import function

For those who don't understand the solution as I didn't, you have to go as deep to the directory as you need for your specific problem. Solution for this errorr:

No module named 'your_folder'

My notebook that I work with is in:

C:\Users\vojte\projects\predicta\EDA\apartments\EDA.ipynb

I want to import this notebook:

C:\Users\vojte\projects\functions\functions.ipynb

I had to modify the solution above by @metakermit to this solution:

import os
import sys
module_path = os.path.abspath(os.path.join('..\..\..'))
if module_path not in sys.path:
    sys.path.append(module_path)

from functions import functions as f

You can check the sys.path for appended directory. In this example this line was appended:

C:\\Users\\vojte\\projects

Improving @joshua-cook answer in using cd.., to make sure that you did not re-run the cell and messing your directory in using run all without restart, use this code instead:

if 'NOTEBOOK_INITIATED_FLAG' not in globals():
    NOTEBOOK_INITIATED_FLAG = True
    %cd ..
%pwd

NOTEBOOK_INITIATED_FLAG is used as a placeholder that mark that the kernel is already running, thus changing directory is not needed.

And this is the super convoluted boilerplate if you want to work with jupytext and run your .py file from parent folder:

import os
import sys

if 'NOTEBOOK_INITIATED_FLAG' not in globals():
    NOTEBOOK_INITIATED_FLAG = True
    
    try:
        # not in notebook
        module_path = os.path.join(os.path.dirname(__file__), os.pardir)
    except:
        # in notebook
        module_path = os.path.abspath(os.path.join('..'))
        %cd ..
        %pwd

    if module_path not in sys.path:
        sys.path.append(module_path)

Here's my 2 cents:

import sys
# map the path where the module file is located. In my case it was the desktop
sys.path.append('/Users/John/Desktop')

# Either import the whole mapping module BUT then you have to use the .notation to map the classes like mapping.Shipping()

import mapping #mapping.py is the name of my module file

shipit = mapping.Shipment() #Shipment is the name of the class I need to use in the mapping module

# Or import the specific class from the mapping module
from mapping import Mapping

shipit = Shipment() #Now you don't have to use the .notation

Here's a generic solution based on this answer that neither requires specifying the parent folder name, nor requires changing the current working directory.

Just update root_parent_level with the number of parent levels where relative imports start, and make sure that __init__.py exists in each subpackage.

if "PKG" not in globals(): # `PKG` is used just to avoid re-excuting the cell more than once
  root_parent_level = 2
  import importlib, sys, pathlib
  PKG = %pwd
  PKG = pathlib.Path(PKG)
  root = PKG
  full_pkg = f"{root.name}"
  for _ in range(root_parent_level):
    root = root.parent
    full_pkg = f"{root.name}.{full_pkg}"
    MODULE_PATH = f"{root}{pathlib.os.path.sep}__init__.py"
    MODULE_NAME = f"{root.name}"
    spec = importlib.util.spec_from_file_location(MODULE_NAME, MODULE_PATH)
    module = importlib.util.module_from_spec(spec)
    sys.modules[spec.name] = module 
    spec.loader.exec_module(module)
  __package__ = full_pkg

This is a adapted version of my answer in the linked question. However, to use my solution as it is at least Python 3.8 (released 14 Oct 2019) is required for the walrus operator.

Using a known top-level package

Suppose your code some_notebook.ipynb resides somewhere within a directory structure which looks like:

project_folder
|-- subdir
    |-- ab
        |-- cd
            |-- some_notebook.ipynb
            |-- script1.py
        |-- script2.py
    |-- script3.py
|-- script4.py
|-- other_folder
    |-- script5.py

And you need your package to be subdir.ab.cd without knowing ab or cd or even the number of nested levels (as long as none of the intermediate levels are called "subdir" as well). Then you could use the following:

import os
import sys

__file__ = globals().get("__vsc_ipynb_file__") or globals().get("__session__")
__package__ = ((lambda p, d:
                (".".join(p[-(n := p[::-1].index(d) + 1):]),
                 sys.path.insert(0, os.sep.join(p[:-n])))[0])(
                     os.path.realpath(__file__).split(os.sep)[:-1], "subdir"))

from .script1 import *
from ..script2 import *
from ...script3 import *
from subdir.script3 import *
from script4 import *
from other_folder.script5 import *

Using a known number of sub-levels

Suppose your code some_notebook.ipynb resides somewhere within a directory structure which looks like:

project_folder
|-- ab
    |-- cd
        |-- some_notebook.ipynb
        |-- script1.py
    |-- script2.py
|-- script3.py
|-- other_folder
    |-- script4.py

And you need your package to be ab.cd without knowing ab or cd but the depth of the package is guaranteed to be 2. Then you could use the following:

import os
import sys

__file__ = globals().get("__vsc_ipynb_file__") or globals().get("__session__")
__package__ = (
    (lambda p, n:
     (".".join(p[-n:]), sys.path.insert(0, os.sep.join(p[:-n])))[0])(
         os.path.realpath(__file__).split(os.sep)[:-1], 2))

from .script1 import *
from ..script2 import *
from script3 import *
from other_folder.script4 import *

Relative imports and absolute imports all work

With the sys.path including the project folder, you also of course do any absolute imports from there. With __package__ correctly computed, one can now do relative imports as well. A relative import of .other_script will look for other_script.py in the same folder as some_notebook.ipynb. It is important to have one additional level in the package hierarchy as compared to the highest ancestor reached by the relative path, because all the packages traversed by the ".."/"..."/etc will need to be a Python package with a proper name.

I got annoyed so I wrote a package.

pip install package_root and then you can import package_root # into your PYTHONPATH.

This works by looking through the call stack to determine which file imported package_root and then walking the up directories to find the first directory that is missing an __init__.py file.

So in practice, you'll need to add __init__.py files to your notebook directories just like you would to get Python packages to work.