There is a how-to-fix summary below, but please read to ensure you’re aware of the implications.
Python has a fairly complex mechanism for allowing multiple versions of both the Python binary and it’s libraries and dependencies to co-exist on a given system. This can vary from system to system so as a point of reference, in this instance I’m talking about modern versions of Ubuntu.
When you install Python3 via the apt tool it should install it’s libraries in /usr/lib/python3/dist-packages/, this is generally referred to as the system python and should only really be used by other packages that are supplied by Ubuntu. So if you install any Python application from a third party, the safe and generally recommended mechanism is to use Python Virtual Environments. (or these days, as mentioned at the end (for security) - containers)
If you subsequently install packages using pip as the root user, it will then overlay these packages and use them in preference to the default system versions. So, installing (or more to the point, updating) specific Python modules, which parts of the system may rely upon being, say an older version, can cause problems. So although you can run pip as root, it’s generally not a good idea. (same applies to sudo pip)
These overrides when installed as root typically reside in /usr/local/lib/python(version)/dist-packages/ and will show up in pip list --local. Because pip will load requested packages and their dependencies, although you may only request one package with pip, it may end up installing many packages as a result. Just as an example, I have actually installed some packages as root for reasons that don’t need to be mentioned at this point ( 
) so what I see as a user is this;
$ sudo pip list --local
Package Version
------------------ ---------
certifi 2022.6.15
charset-normalizer 2.1.0
distlib 0.3.5
filelock 3.7.1
pipenv 2022.7.4
platformdirs 2.5.2
requests 2.28.1
setuptools 63.2.0
six 1.16.0
virtualenv 20.15.1
virtualenv-clone 0.5.7
Then is I look at the actual file location I see;
$ ls /usr/local/lib/python3.10/dist-packages/
certifi distlib-0.3.5.dist-info pipenv-2022.7.4.dist-info requests-2.28.1.dist-info virtualenv-20.15.1.dist-info
certifi-2022.6.15.dist-info _distutils_hack pkg_resources setuptools virtualenv_clone-0.5.7.dist-info
charset_normalizer distutils-precedence.pth platformdirs setuptools-63.2.0.dist-info
charset_normalizer-2.1.0.dist-info filelock platformdirs-2.5.2.dist-info six-1.16.0.dist-info
clonevirtualenv.py filelock-3.7.1.dist-info __pycache__ six.py
distlib pipenv requests virtualenv
You’ll see by eye that this list of libraries is relatively short compared to the full list, and the names are similar to the files list in the /usr/local folder.
Fix # 1
To attempt to correct this I would use the following command;
pip uninstall `sudo pip list --local --format freeze|cut -d"=" -f1`
This should cycle through each of the root level pip installed modules and ask me whether I would like to delete the module. (which will fail because the module was installed as root and I’m doing this as a user). If I run this command prefixed with ‘sudo’, saying Y to each question should remove the packages in question.
Caveat; there are some packages that really need to be installed as root (like virtualenv and pyenv) however these can easily be re-added later.
Once you have cleaned your system environment, you may need to look at pip packages you have installed as a user. These will typically live a folder relative to the version of Python you’re using,
$ python -V
Python 3.10.12
$ ls ~/.local/lib/python3.10/site-packages
Now, you may (or may not) want all of these libraries, which either you (or something else on your behalf) has installed into your file-tree. These libraries should override the default system libraries, and they should override any libraries that may have been installed globally via pip as the root user. You can remove these with a slightly modified version of the command above;
Fix # 2
$ pip uninstall `pip list --local --user --format freeze|cut -d"=" -f1`
Found existing installation: certifi 2022.6.15
Uninstalling certifi-2022.6.15:
Would remove:
/home/gareth/.local/lib/python3.10/site-packages/certifi-2022.6.15.dist-info/*
/home/gareth/.local/lib/python3.10/site-packages/certifi/*
Proceed (Y/n)?
i.e. we’ve added “–user” to tell it to look for user installed packages only, and we don’t need to use sudo because they’re installed in our local file-tree to which we already have access. In order to avoid having to hit Y for each package, we can add -y just after uninstall and it should rip through the whole lot.
Corrective summary
- Be aware that these fixes will potentially break locally installed Python applications that you may have installed that rely on non-standard dependencies and that you will need to re-install them later to correct the situation.
- Remove overridden root level pip installed packages with Fix # 1
- Remove overridden user level pip installed packages with Fix # 2
Note that if you’re not using cut/paste, in the example bash commands I’m using back ticks [these are not quotes] (to the left of numeric 1 on a UK keyboard) which tells bash to evaluate the contained expression.
Installing new packages
- Install the Python Virtual Management tool of your choice (which will typically be installed as a root level local pip package), for example;
curl https://pyenv.run | bash # https://github.com/pyenv/pyenv
- Create a dedicated virtual environment for your application, for example;
$ pyenv virtualenv application
- Activate the environment, for example;
$ pyenv activate application
(application) gareth:~$
-
Install your applications, libraries, use pip etc, and everything will go into your virtual environment (in this instance called application) without affecting anything else. It effectively captures everything you do with regards to installing python dependencies.
-
When you subsequently want to run the application, whether you are calling it from the command line or activating it as a system service, just prefix the command with pyenv activate, so for example;
pyenv activate application && python3 -m my_application
# '&&' is important, don't use ';'
So now when my_application runs, it will pick up all the dependencies in terms of libraries, tools etc, that you have installed in the application virtual environment in preference to any installed as root as a part of the system installation.
How to avoid all this and boost security
- Run foreign applications that have the potential to do undesirable things to your data and or your system in a container, for example using LXD, or;
- Choose applications that are supplied as a part of your distribution which hopefully should be subject to a degree of scrutiny / checking (for example I use tinc as my VPN software of choice)
When running inside a container, you can effectively dedicate the entire container (or ‘system’) to the application of your choice, hence virtual environments or considerations about overlapping applications or Python versions are moot.
