Effect of using sys.path.insert(0, path) and sys.path.append(path) when loading modules

Question

I was recently having a problem with a python ImportError, where the module was found when running on my local computer but not found on the CI server. I solved this problem by swapping sys.path.append(path) in my script with sys.path.insert(0, path) where path is the string module location.

Since this is my module and not an installed package (related question), why does the order of paths fix this problem?

Answer 1

I'm quite a beginner in Python and I found the answer of Anand was very good but quite complicated to me, so I try to reformulate :

1) insert and append methods are not specific to sys.path and as in other languages they add an item into a list or array and :
* append(item) add item to the end of the list,
* insert(n, item) inserts the item at the nth position in the list (0 at the beginning, 1 after the first element, etc ...).

2) As Anand said, python search the import files in each directory of the path in the order of the path, so :
* If you have no file name collisions, the order of the path has no impact,
* If you look after a function already defined in the path and you use append to add your path, you will not get your function but the predefined one.

But I think that it is better to use append and not insert to not overload the standard behaviour of Python, and use non-ambiguous names for your files and methods.

Answer 2

Because Python checks in the directories in sequential order, starting at the first directory in the sys.path list, until it finds the .py file it was looking for.

Normally, the current directory or the directory of the script is the first element in the list, unless you modify it (like you did). From documentation:

As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter. If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first. Notice that the script directory is inserted before the entries inserted as a result of PYTHONPATH.

So, most probably, you had a .py file with the same name as the module you were trying to import from, in the current directory (where the script was being run from).

Also, something to note about ImportErrors - Let's say the import error says: ImportError: No module named main. This doesn't mean main.py is overwritten - if it was overwritten it would still exist and we would not have issues trying to read it. It's some module above this that got overwritten with a .py or some other file.

For example, consider a directory structure that looks like this:

 - test
    - shared
       - __init__.py
       - phtest.py
    - testmain.py

Now from testmain.py, I call from shared import phtest, which works fine.

Now let's say I introduce a shared.py in test directory:

 - test
    - shared
       - __init__.py
       - phtest.py
    - testmain.py 
    - shared.py

Now when I try to do from shared import phtest from testmain.py, I get the error:

ImportError: cannot import name 'phtest'

As you can see above, the file that introduced the issue is shared.py, not phtest.py.