Use of PIP has stopped Proton VPN working

I was unable recently to start Proton VPN. Submitting requested info to Proton produced the following reply:

According to the output you are receiving: AttributeError: module 'lib' has no attribute 'OpenSSL_add_all_algorithms' it appears that you might be experiencing some Python misconfiguration probably as a result of pip installing some packages recently.
Could you please let us know if you have recently installed any packages via the pip command? If so, please note that this is something we discourage as packages installed in this way may override official Linux distribution packages and can break other Python software that relies on them, including the Proton VPN app and CLI.>

Unfortunately I can’t remember what I installed with PIP. Is there any way that I can find it - via a log of some kind, perhaps?


Using the command:

tail -f /var/log/installer/debug

results in:

keith@E5570:~$ tail -f /var/log/installer/debug
/usr/lib/ubiquity/ubiquity/frontend/gtk_components/ Warning: Source ID 4713 was not found when attempting to remove it
/usr/lib/ubiquity/ubiquity/frontend/gtk_components/ Warning: Source ID 7716 was not found when attempting to remove it
/usr/lib/ubiquity/ubiquity/frontend/gtk_components/ Warning: Source ID 4176 was not found when attempting to remove it

(ubiquity:1770): Gtk-CRITICAL **: 12:42:11.728: gtk_box_gadget_distribute: assertion ‘size >= 0’ failed in GtkScrollbar

(ubiquity:1770): Gtk-CRITICAL **: 12:42:14.650: gtk_box_gadget_distribute: assertion ‘size >= 0’ failed in GtkScrollbar

Does that help at all?

Ok, so if you have pip installed then pip list will list all the Python packages that the default Python installation can see. On a clean system, everything in this list will be part of the system installation. I would argue (!) that no application should be installed that relies on the state of the system installation for reasons which may now be obvious.

To see what you may have installed locally, which would override the system installation, try pip list --local and this should give you a much shorter list (depending on what you’ve installed!) and removing each one with pip uninstall (package name) should revert you to the default installation / state which it sounds like Proton is relying on.

Generally (!) all Python applications should either run in their own dedicated user space (i.e. their own user-id) or, within a python virtual environment in your user space. There are many different tools for managing Python virtual environments, my current personal choice is pyenv. (Python virtual environments, not to be confused with virtual machines or containers) Virtual environments maintain different sets of installed tools and libraries for different applications, so each application can have it’s own unique set of dependencies which won’t collide or compete with other applications or indeed the system installation.

hth :slight_smile:

Many thanks for this info.
As I understand it, pip is installed automatically with Python >3

The commands pip list and pip list --local each produce the same list (is this expected?) of 131 entries. The only ones that I recognise are pip itself, four proton files, img2pdf (part of imagemagick) and duplicity.

I know that I installed imagemagick and protonVPN and both have worked without problems (except recently for this VPN difficulty). I did not install duplicity as I have written my own backup code, so one wonders why the remaining 100+ files are there. In particular, if I remove all of them using pip uninstall I am guessing that things will be broken. Indeed, as pip is listed, that would be removed as well!

What do you think? I’m not averse to re-installing Ubuntu if necessary, so I’m willing to have a go.

Locally installed files (via pip) will override system installed versions (via apt) which is the issue. I would remove all the --local libraries (as a user) then reinstall your required apps (proton) inside a python virtual environment.

Install pyenv , it’s a one line curl command. Use pyenv create to create an env, then penv activate, then you can pip install what you want. pyenv deactivate to exit the virtual env.

Virtual environments are persistent, so second time around, just pyenv activate, then run your app.

There is a pyenv flag for running a command inside a virtual env, i forget what it is just for a sec.

Thanks for that. It’s looking a bit advanced for me. In particular the use of a virtual environment to run Proton. I had hopes of running Proton VPN automatically using the Settings/Network facility but I won’t be able to do that in a virtual environment.

I shall remove everything in the pip local list, as you suggest, and see if Proton then works as before - then report back.

Ok, I can give you more detail but I’ve been away all week and just got back. If proton has experienced the issue once, it will almost certainly happen again at some point. You will be able to run proton in a virtenv transparently, that’s kinda the point of virtenv’s :wink:

After backing up my personal files I created a file called list from the first column of the output from pip list --local then:

keith@E5570:~$ for name in ${list[@]}; do pip uninstall $name; done
WARNING: Skipping Package as it is not installed.

pip uninstall [options] …
pip uninstall [options] -r …

no such option: ----------------------
Found existing installation: apturl 0.5.2
ERROR: Cannot uninstall ‘apturl’. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
Found existing installation: Babel 2.8.0
Uninstalling Babel-2.8.0:
Would remove:
Proceed (Y/n)? y
ERROR: Exception:
Traceback (most recent call last):
File “/usr/lib/python3.10/”, line 816, in move
os.rename(src, real_dst)
PermissionError: [Errno 13] Permission denied: ‘/usr/lib/python3/dist-packages/Babel-2.8.0.egg-info’ → ‘/tmp/pip-uninstall-xnt4y8to’

etc etc & pages of errors for the next “name”. So I stopped the process. If you think a fresh install of Ubuntu would be the easiest solution, I would be happy with that as I really don’t know what I am doing.


Mm, so, did you run pip as root when you were installing things?
(you should never need to re-install Ubuntu…)

There is a how-to-fix summary below, but please read to ensure you’re aware of the implications.

Python has a fairly complex mechanism for allowing multiple versions of both the Python binary and it’s libraries and dependencies to co-exist on a given system. This can vary from system to system so as a point of reference, in this instance I’m talking about modern versions of Ubuntu.

When you install Python3 via the apt tool it should install it’s libraries in /usr/lib/python3/dist-packages/, this is generally referred to as the system python and should only really be used by other packages that are supplied by Ubuntu. So if you install any Python application from a third party, the safe and generally recommended mechanism is to use Python Virtual Environments. (or these days, as mentioned at the end (for security) - containers)

If you subsequently install packages using pip as the root user, it will then overlay these packages and use them in preference to the default system versions. So, installing (or more to the point, updating) specific Python modules, which parts of the system may rely upon being, say an older version, can cause problems. So although you can run pip as root, it’s generally not a good idea. (same applies to sudo pip)

These overrides when installed as root typically reside in /usr/local/lib/python(version)/dist-packages/ and will show up in pip list --local. Because pip will load requested packages and their dependencies, although you may only request one package with pip, it may end up installing many packages as a result. Just as an example, I have actually installed some packages as root for reasons that don’t need to be mentioned at this point ( :wink: ) so what I see as a user is this;

$ sudo pip list --local
Package            Version
------------------ ---------
certifi            2022.6.15
charset-normalizer 2.1.0
distlib            0.3.5
filelock           3.7.1
pipenv             2022.7.4
platformdirs       2.5.2
requests           2.28.1
setuptools         63.2.0
six                1.16.0
virtualenv         20.15.1
virtualenv-clone   0.5.7

Then is I look at the actual file location I see;

$ ls /usr/local/lib/python3.10/dist-packages/
certifi                             distlib-0.3.5.dist-info   pipenv-2022.7.4.dist-info     requests-2.28.1.dist-info    virtualenv-20.15.1.dist-info
certifi-2022.6.15.dist-info         _distutils_hack           pkg_resources                 setuptools                   virtualenv_clone-0.5.7.dist-info
charset_normalizer                  distutils-precedence.pth  platformdirs                  setuptools-63.2.0.dist-info
charset_normalizer-2.1.0.dist-info  filelock                  platformdirs-2.5.2.dist-info  six-1.16.0.dist-info                  filelock-3.7.1.dist-info  __pycache__         
distlib                             pipenv                    requests                      virtualenv

You’ll see by eye that this list of libraries is relatively short compared to the full list, and the names are similar to the files list in the /usr/local folder.

Fix # 1

To attempt to correct this I would use the following command;

pip uninstall `sudo pip list --local --format freeze|cut -d"=" -f1`

This should cycle through each of the root level pip installed modules and ask me whether I would like to delete the module. (which will fail because the module was installed as root and I’m doing this as a user). If I run this command prefixed with ‘sudo’, saying Y to each question should remove the packages in question.

Caveat; there are some packages that really need to be installed as root (like virtualenv and pyenv) however these can easily be re-added later.

Once you have cleaned your system environment, you may need to look at pip packages you have installed as a user. These will typically live a folder relative to the version of Python you’re using,

$ python -V
Python 3.10.12
$ ls ~/.local/lib/python3.10/site-packages

Now, you may (or may not) want all of these libraries, which either you (or something else on your behalf) has installed into your file-tree. These libraries should override the default system libraries, and they should override any libraries that may have been installed globally via pip as the root user. You can remove these with a slightly modified version of the command above;

Fix # 2

$ pip uninstall `pip list --local --user --format freeze|cut -d"=" -f1`
Found existing installation: certifi 2022.6.15
Uninstalling certifi-2022.6.15:
  Would remove:
Proceed (Y/n)?

i.e. we’ve added “–user” to tell it to look for user installed packages only, and we don’t need to use sudo because they’re installed in our local file-tree to which we already have access. In order to avoid having to hit Y for each package, we can add -y just after uninstall and it should rip through the whole lot.

Corrective summary

  • Be aware that these fixes will potentially break locally installed Python applications that you may have installed that rely on non-standard dependencies and that you will need to re-install them later to correct the situation.
  • Remove overridden root level pip installed packages with Fix # 1
  • Remove overridden user level pip installed packages with Fix # 2

Note that if you’re not using cut/paste, in the example bash commands I’m using back ticks [these are not quotes] (to the left of numeric 1 on a UK keyboard) which tells bash to evaluate the contained expression.

Installing new packages

  • Install the Python Virtual Management tool of your choice (which will typically be installed as a root level local pip package), for example;
curl | bash    #
  • Create a dedicated virtual environment for your application, for example;
$ pyenv virtualenv application
  • Activate the environment, for example;
$ pyenv activate application 
(application) gareth:~$
  • Install your applications, libraries, use pip etc, and everything will go into your virtual environment (in this instance called application) without affecting anything else. It effectively captures everything you do with regards to installing python dependencies.

  • When you subsequently want to run the application, whether you are calling it from the command line or activating it as a system service, just prefix the command with pyenv activate, so for example;

pyenv activate application && python3 -m my_application 
# '&&' is important, don't use ';'

So now when my_application runs, it will pick up all the dependencies in terms of libraries, tools etc, that you have installed in the application virtual environment in preference to any installed as root as a part of the system installation.

How to avoid all this and boost security

  • Run foreign applications that have the potential to do undesirable things to your data and or your system in a container, for example using LXD, or;
  • Choose applications that are supplied as a part of your distribution which hopefully should be subject to a degree of scrutiny / checking (for example I use tinc as my VPN software of choice)

When running inside a container, you can effectively dedicate the entire container (or ‘system’) to the application of your choice, hence virtual environments or considerations about overlapping applications or Python versions are moot.

This is getting beyond me.
Question. In fix2 you say:

But I don’t see a “-user” and “sudo” does appear in the commands. Have I missed something?

Sorry, typo on my part, have corrected. I had “–local” twice instead of “–local --user”. To clear packages you have installed as a user, you won’t need sudo.

Fix # 1 should remove packages installed when you ran pip as root.
Fix # 2 should remove packages installed when you wan pip as yourself.

pyenv is a way to put up a partition between different potentially competing Python applications that might want either different versions of Python, or possibly different versions of the same Python library. (the latter being the likely cause of your problem)

I thought that might be the case. But…

keith@E5570:~$ pip uninstall sudo pip list --local --format freeze|cut -d"=" -f1
[sudo] password for keith:
Found existing installation: apturl 0.5.2
ERROR: Cannot uninstall ‘apturl’. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
keith@E5570:~$ pip uninstall sudo pip list --local --user --format freeze|cut -d"=" -f1
ERROR: You must give at least one requirement to uninstall (see “pip help uninstall”)

Hmm? Curiouser & curiouser.

Sorry, for fix # 1, you are removing system level packages, so you need to prefix the pip uninstall with “sudo”.

For fix # 2, you’re working within your own user area, no sudo’s are required.

So how did you end up with pip installed packages, is this something the Proton installer has done, or was this just experimentation gone awry?

No experiment as I had no idea what pip was.

I downloaded and installed the VPN application according to these instructions. And it all went well with no mention of pip.
However, I do seem to remember using pip to install something else, although I don’t remember what it was, and I was following insructions blindly. The pip list contains a wide variety of disparate things that don’t seem to be related, although the appearance of four proton-related names seems odd, unless the proton installation inserted them unbeknownst to me:

keith@E5570:~$ pip list --local | grep proton
proton-client 0.7.1
protonvpn-cli 3.13.0
protonvpn-gui 1.12.0
protonvpn-nm-lib 3.16.0

and the Fix 2 pip uninstall commands are still not working…as three messages ago.

Fix 1:

keith@E5570:~$ pip uninstall sudo pip list --local --format freeze|cut -d"=" -f1
[sudo] password for keith:
Found existing installation: apturl 0.5.2
ERROR: Cannot uninstall ‘apturl’. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

Well, it would seem that the proton install has used pip to install (parts of) itself, which may well (probably) have loaded in “other” things. If it’s just “apturl” that’s failing (I’m not familiar with that library) I wouldn’t worry about it too much.

After cleaning with fix 1 and 2, if you follow the same Proton instructions I would expect it to work. Note; you may need to follow the “how to remove” first, otherwise it’s liable to complain it’s already installed.

If you’ve not specifically installed anything else that needed pip, then nothing else should be broken, but bear in mind;

  • if you need to use a.n.other pip based application, using it outside of a virtualenv might break Proton
  • whenever you upgrade Ubuntu, it might break Proton

Typically packages are tightly coupled with specific versions of specific libraries. The danger of installing a package published by one vendor (Proton) who relies on libraries published by another vendor (Ubuntu) is that they may not always be in-sync. (which can result in problems like the one you are experiencing)

Note the Proton instructions don’t seem to be compatible with using virtual env’s so I’m not sure how you would protect against the same happening again.

Thanks for this.
After using Fix & Fix2, all the items are still listed in pip list so nothing has been removed.
I shall remove proton vpn and try again. [EDIT] removed using this although the menu item remains.

By the way: would I be better off installing the proton VPN CLI rather than the GUI? I used to use CLI before the GUI was available.

Erm, in which case you need to tweak the list of items and remove the one it’s failing on. Just removing Proton is unlikely to make a difference. The pip uninstall command you have pasted in is missing it’s backticks (so won’t work as listed), that aside, of you try;

sudo pip uninstall `sudo pip list --local --format freeze|cut -d"=" -f1|grep -v apturl`

pip uninstall (the following list);

  • list all local packages (pip list …)
  • for each item cut the line based on “=” and pick the first item (cut …)
  • filter and pass everything not containing “apturl” (grep …)

A long hand alternative to break it down a little might be;

sudo pip list --local --format freeze|cut -d"=" -f1 > to_delete.txt

Edit “to_delete.txt” and remove the problems (apturl)

sudo pip uninstall `cat to_delete.txt`

By the way: would I be better off installing the proton VPN CLI rather than the GUI? I used to use CLI before the GUI was available.

I’m afraid I don’t really know enough about Proton, other than to say that the packaging mechanism they seem to be providing, i.e. a .deb that calls pip, is not one I would deploy and not something “I” would view as “good practice”. If the CLI uses a different approach, then maybe, however given they’ve gone down this route for the GUI, I suspect it may be the same.

Typically I would expect this kind of package to be compiled into an executable using pyinstaller or similar. Then have the executable packaged into a .deb. In this format you get one (runable) program with everything (libraries, virtual environment etc) all packaged up inside, so the user doesn’t need to worry about anything Python specific. In this instance it can’t interfere with the ‘system’ and doesn’t get broken by other installs or system upgrades. (and doesn’t use pip)

Actually, I did include the back ticks - they just don’t show when I cut from the terminal and paste here (how did you manage it?).

OK. Here’s all the output:

keith@E5570:~$ sudo pip uninstall sudo pip list --local --format freeze|cut -d"=" -f1|grep -v apturl
Found existing installation: Babel 2.8.0
Uninstalling Babel-2.8.0:
Would remove:
Proceed (Y/n)? y
Successfully uninstalled Babel-2.8.0
Found existing installation: bcrypt 3.2.0
Uninstalling bcrypt-3.2.0:
Would remove:
Proceed (Y/n)? y
Successfully uninstalled bcrypt-3.2.0
Found existing installation: beautifulsoup4 4.10.0
Uninstalling beautifulsoup4-4.10.0:
Would remove:
Proceed (Y/n)? y
Successfully uninstalled beautifulsoup4-4.10.0
Found existing installation: blinker 1.4
ERROR: Cannot uninstall ‘blinker’. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
keith@E5570:~$ sudo pip list --local --format freeze|cut -d"=" -f1 > to_delete.txt
keith@E5570:~$ sudo pip uninstall cat to_delete.txt
Found existing installation: apturl 0.5.2
ERROR: Cannot uninstall ‘apturl’. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

…but the pip list is still there.
Ah! the pip list does not have any “=” symbols. Here are the last few lines:

webencodings 0.5.1
wheel 0.37.1
xcffib 0.11.1
xdg 5
xkit 0.0.0
zipp 1.0.0

I’ll try " " as I used in my script… No, the pip list is still there.