Ok, so it looks like it can’t uninstall blinker and distutils. So at this point there should only be two packages left … which should be Ok.
I think maybe a re-read of the above would help.
sudo will reference packages installed by the root user with pip.
If you delete packages installed by the root user with pip, then do “pip list”, you are listing packages installed by your user id, with pip. This will be a different list. I would hope that just correcting the root level packages would solve the problem, but to be safe, clearing both would be advisable.
Fix # 1 clears the root packages.
Fix # 2 clears user level packages.
With regards to why you are not seeing “=” in the list;
sudo means it will access as root
–local means it will access packages installed with pip (as root)
–format freeze lists packages, with no header in the form “package name==version”
The freeze option is just an easy way to suppress the headers from “pip list” and use a format that’s easier for “cut” to parse.
If you pipe this through cut -d"=" -f1, this is cut the line using = as the delimiter, and take the first field (i.e. the bit before the =) - which should yield a raw list of packages that pip uninstall can understand.
OK, I’m getting there: –freeze not only removes headers but inserts “=” that one uses as delimiter.
If I’ve understood you, deleting the packages does not remove the names from the list. In which case, as I have done both fix1 and fix2, I have deleted the packages. Although it still confuses me that the list is still populated when I pip list.
If I do manual uninstall of one entry, such as cffi it finds the item but just produces lots of errors:
keith@E5570:~$ pip uninstall cffi Found existing installation: cffi 1.15.1
Uninstalling cffi-1.15.1:
Would remove:
/usr/local/lib/python3.10/dist-packages/_cffi_backend.cpython-310-x86_64-linux-gnu.so
/usr/local/lib/python3.10/dist-packages/cffi-1.15.1.dist-info/*
/usr/local/lib/python3.10/dist-packages/cffi/* Proceed (Y/n)? y
ERROR: Exception:
Traceback (most recent call last):
File “/usr/lib/python3.10/shutil.py”, line 816, in move
os.rename(src, real_dst)
PermissionError: [Errno 13] Permission denied: ‘/usr/local/lib/python3.10/dist-packages/_cffi_backend.cpython-310-x86_64-linux-gnu.so’ → ‘/tmp/pip-uninstall-ut34ga1f/_cffi_backend.cpython-310-x86_64-linux-gnu.so’
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/usr/local/lib/python3.10/dist-packages/pip/_internal/cli/base_command.py”, line 180, in exc_logging_wrapper
status = run_func(*args)
File "/usr/local/lib/py…etc.
Uninstalling packages will uninstall packages. pip list lists installed packages. When you uninstall them, they will no longer appear on the list. The issue is that there are multiple lists and you are removing them from one list, then looking at a different list.
The primary problem with what you’ve just posted is the “permission denied”. This is happening because you are trying to uninstall a package that the root user has installed, and you don’t have the permission to do that. (because you’ve omitted the sudo)
If you do pip list it will show all the packages you have access to. This will include packages installed by root, AND those installed by you.
If you focus on the packages installed by root using pip first, you need to apply fix # 1. Fix # 1 should remove packages installed by user root with pip. To check the results, you need to list the remaining packages installed by root with pip, you need to do;
sudo pip list --local
If you omit the sudo and the --local, it will list all packages, which is a different list. You can double check whether they have been removed by looking in the folder locations they are installed to, as above, ls /usr/local/lib/python(version)/dist-packages/. Just bear in mind, if the package was installed by root, you will need “sudo” top uninstall it. If you see “permission denied”, you’re either trying to delete something you shouldn’t, or you are missing a sudo.
You have been exceptionally patient, for which I am very grateful. I still can’t get my head around all this as I can’t form a coherent picture in my head of all the various lists and their significance.
As far as I understand it, performing these commands should have done the trick:
Fix # 1 should remove packages installed when you ran pip as root. pip uninstall sudo pip list --local --format freeze|cut -d"=" -f1
Fix # 2 should remove packages installed when you ran pip as yourself. pip uninstall pip list --local --user --format freeze|cut -d"=" -f1
Now, if I do:
keith@E5570:~$ pip list --local | wc -l
128
keith@E5570:~$ sudo pip list --local | wc -l
128
keith@E5570:~$ ls /usr/local/lib/python3.10/dist-packages/ | wc -l
68
The list in “packages” contains lots of pairs of “linked” names so the number of actual packages is less than half the 68 shown. All the actual packages listed appear also in (both) the pip lists.
So it does look as though I have deleted packages except those that I mustn’t (?).
I started out with 131 entries in both the pip lists so the three missing will be the ones I removed manually. Not sure what that means. Anyway; re-installing Proton VPN doesn’t work so I shall consider a fresh install of Ubuntu - or perhaps a different VPN. But I shan’t use containers or virtual environments as all that is beyond me.
Ok, I’ll try to come back with a diagram re; visualisation, but in the meantime it looks like you have a bunch of packages installed with the system via apt, which would be expected, then a bunch installed with pip, as root, then nothing installed with pip as a user. This sounds about right (?)
So, you don’t need to worry about fix # 2, and fix # 1 isn’t working for you.
To try to debug this, can you do;
$ sudo pip list --local
# then ..
$ sudo pip uninstall (the first package listed above)
Does this actually work and delete the package? Check;
ls /usr/local/lib/python3.10/dist-packages/ | wc -l
And I would expect it to have decreased by 1, whereas the pip listing may not change. (I will attempt to show why on the diagram)
Essentially the different pip “lists” are overlays or overlapping layers with priority given to the layer the furthest away from the system. So when Python imports, it will look for the module in an active virtual environment first, then in user installed pips, then in root installed pips, then in system installed packages, only failing if it can’t find it anywhere. If the same package is installed at multiple levels, it will take the highest priority on this basis. (where priority [from the diagram] is D → C → B → A)
If you have the package installed with apt (A) and installed with pip by root (B), and you delete it from (B), the package will still be available, but it will be the version installed in (A). The listing of packages available to the user (pip --list) in this instance won’t change in terms of the number of packages.
Does that make more sense?
In the context of the problem, it looks like you’ve installed Proton which relies on packages installed in A. Subsequently some packages have been installed via pip, these have gone into B which take precedence over those in A. It looks like at least one of those packages in B seems to be incompatible with Proton, so the easy fix is to remove everything from B, hence Proton will then only take from A (as it expects) and hopefully it should work.
My point re; virtual environments is that Ubuntu may decide to update (A) (well, it will at some point) and this update will be independent of Proton. At this point, Proton and Ubuntu will be out of sync and it might be that something new in A is incompatible with Proton. Indeed you might find that after clearing B, Proton still refuses to work, in which case the likely cause of the problem would be a recent Ubuntu update. (and the fix would be a new version of Proton, to match the Ubuntu update).
Note: can I just highlight that this issue is not an design flaw or architectural issue with Python. It’s caused specifically by the way that some people choose to package their software. There are (many people would argue) better ways, indeed the likes of snap and flatpak packaging make all these issues vanish.
keith@E5570:~$ sudo pip uninstall blinker
Found existing installation: blinker 1.4
ERROR: Cannot uninstall ‘blinker’. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
keith@E5570:~$ sudo pip uninstall soupsieve
Found existing installation: soupsieve 2.3.1
Uninstalling soupsieve-2.3.1:
Would remove:
/usr/lib/python3/dist-packages/soupsieve
/usr/lib/python3/dist-packages/soupsieve-2.3.1.egg-info
Proceed (Y/n)? y
Successfully uninstalled soupsieve-2.3.1
WARNING: Running pip as the ‘root’ user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: 12. Virtual Environments and Packages — Python 3.12.1 documentation
keith@E5570:~$ ls /usr/local/lib/python3.10/dist-packages/ | wc -l
68
keith@E5570:~$
which is the same number as before but and soupsieve-2.3.1 is no longer listed in packages. Perhaps it wasn’t there before, so:
keith@E5570:~$ sudo pip uninstall numpy
Found existing installation: numpy 1.25.2
Uninstalling numpy-1.25.2:
Would remove:
/usr/local/bin/f2py
/usr/local/bin/f2py3
/usr/local/bin/f2py3.10
/usr/local/lib/python3.10/dist-packages/numpy-1.25.2.dist-info/*
/usr/local/lib/python3.10/dist-packages/numpy.libs/libgfortran-040039e1.so.5.0.0
/usr/local/lib/python3.10/dist-packages/numpy.libs/libopenblas64_p-r0-5007b62f.3.23.dev.so
/usr/local/lib/python3.10/dist-packages/numpy.libs/libquadmath-96973f99.so.0.0.0
/usr/local/lib/python3.10/dist-packages/numpy/*
Proceed (Y/n)? y
Successfully uninstalled numpy-1.25.2
WARNING: Running pip as the ‘root’ user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: 12. Virtual Environments and Packages — Python 3.12.1 documentation
keith@E5570:~$ ls /usr/local/lib/python3.10/dist-packages/numpy
ls: cannot access ‘/usr/local/lib/python3.10/dist-packages/numpy’: No such file or directory
keith@E5570:~$
so it’s working. Now:
keith@E5570:~$ ls /usr/local/lib/python3.10/dist-packages/ | wc -l
65
keith@E5570:~$
OK. 68-65=3 because I tried to remove two other packages which it refused to do because " It is a distutils installed project". But still confusing!
Ok, see my previous post + diagram for an explanation.
Next would be to try to uninstall everything from (B) using the method you just used to see how low you can get the count.
The “cannot uninstall” error is typically because (for whatever reason) it is trying to uninstall something with pip that was actually installed with apt. (i.e. it’s trying to use pip on a package in (A))
Your diagrammatic explanation is excellent - thank you. It will take a while to sink in but it’s becoming clearer.
I’ll go ahead and try the manual method to remove as much as I can, and report back - might take a while.
I may have screwed up (no surprise there, then).
I went through the list produced by sudo pip list rather than sudo pip list --local and successfully deleted everything that would delete. Now I get:
keith@E5570:~$ sudo pip list | wc -l
63
keith@E5570:~$ sudo pip list --local | wc -l
0
keith@E5570:~$ ls /usr/local/lib/python3.10/dist-packages/ | wc -l
0
BUT… five (or six) Proton entries were not removed e.g.:
keith@E5570:~$ sudo pip uninstall proton-client
Found existing installation: proton-client 0.7.1
Not uninstalling proton-client at /usr/lib/python3/dist-packages, outside environment /usr
Can’t uninstall ‘proton-client’. No files were found to uninstall.
WARNING: Running pip as the ‘root’ user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: 12. Virtual Environments and Packages — Python 3.12.1 documentation
keith@E5570:~$
Ok, Proton was installed with apt, so have you tried uninstalling it with apt? It may not have installed with pip, it may just have added to or overwritten python packages in (A) … which means you would need apt to remove them.
apt search proton , or similar should turn up what it can see …
Yes to sudo, but search will pick all packages with proton in the name. You will need to select the ones you actually want to remove. From an earlier listing you put up I think there were 5, although removing the main one may remove the rest automatically.
keith@E5570:~$ sudo apt remove proton-client
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
E: Unable to locate package proton-client
keith@E5570:~$ sudo apt remove protonvpn-cli
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
The following packages were automatically installed and are no longer required:
libsecret-tools protonvpn-gui python-babel-localedata python3-babel python3-dialog python3-dnspython python3-gnupg
python3-jinja2 python3-proton-client python3-protonvpn-nm-lib python3-psutil python3-requests-toolbelt
Use ‘sudo apt autoremove’ to remove them.
The following packages will be REMOVED
protonvpn-cli
0 to upgrade, 0 to newly install, 1 to remove and 2 not to upgrade.
After this operation, 125 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database … 249794 files and directories currently installed.)
Removing protonvpn-cli (3.13.0-1) …
keith@E5570:~$
menu icon still there so:
keith@E5570:~$ sudo apt remove protonvpn-gui
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
The following packages were automatically installed and are no longer required:
libsecret-tools python-babel-localedata python3-babel python3-dialog python3-dnspython python3-gnupg python3-jinja2
python3-proton-client python3-protonvpn-nm-lib python3-psutil python3-requests-toolbelt
Use ‘sudo apt autoremove’ to remove them.
The following packages will be REMOVED
protonvpn-gui
0 to upgrade, 0 to newly install, 1 to remove and 2 not to upgrade.
After this operation, 7,814 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database … 249776 files and directories currently installed.)
Removing protonvpn-gui (1.12.0-1) …
Processing triggers for desktop-file-utils (0.26-1ubuntu3) …
Processing triggers for hicolor-icon-theme (0.17-2) …
Processing triggers for gnome-menus (3.36.0-1ubuntu3) …
Processing triggers for mailcap (3.70+nmu1ubuntu1) …
Processing triggers for bamfdaemon (0.5.6+22.04.20220217-0ubuntu1) …
Rebuilding /usr/share/applications/bamf-2.index…
keith@E5570:~$
menu icon gone.
can’t remove (unable to locate):
proton-client 0.7.1 (which I imagine is important to remove)
protonvpn-nm-lib 3.16.0
I’m not sure how to proceed now as I would like to have a VPN but don’t want all this hassle again. But neither do I want to venture into Containers or Virtual environments. I’ve read you How-Tos on these subjects but don’t really understand them.
Very many thanks for helping me - especially with the diagram which was very useful.
Ok, so you could re-install … Proton isn’t something I’d choose personally.
What was on my list to look at was;
Now, I’ve not tried it, so I can’t say whether it’s subject to the same issue (or others) but it is Cloudflare, it appears to do the same thing, and seems to be a free service.
I will look at WARP and maybe do a review, as I understand it you can run it as a local proxy (in a container) that you can point any of your browsers at.
Security Note any software you install on your computer will have access to things on your computer. If that software (or any sub-component, library etc) is compromised in any way, “it” will have access to things on your computer. “Things” may include passwords, keys etc. There are instances where there is little choice but to trust the author and their chosen supply chain, but (IMO) the days of running any software not in a container are numbered. When you install packages via apt they have full access to your system. When you install via snap they run in a container. These days I always prefer snap based software if/when available.
