Is there a way to sort "pip list" packages which only shows --userinstalled packages?

I want a command which works in this way but in case of python packages:

dnf repoquery --userinstalled

I am looking for a command which looks something like this:

pip list --userinstalled

I’m in the cleaning process where I’m removing packages that are idle (stale packages which are no longer used/required). I’m looking for ways that help me to see all pip packages which are installed manually by me in the past for testing propose.

You really should be using virtual environments when installing packages using pip so that they don’t interfere with your system packages. When you don’t need a virtual environment, you just delete it and whatever was installed in there is gone. No need to find a list using pip and all that.

https://docs.python.org/3/library/venv.html

If you haven’t been using virtual environments and have been using pip but without sudo, pip will generally install user packages in ~/.local/ (see man pip-install). If you’ve been using sudo, pip has installed stuff in the system locations: /usr/lib/pythonX.Y and /usr/lib64/pythonX.Y, and there’s no easy way of “undoing” any changes it has made to packages you installed using dnf. You’ll have to manually find the packages, then remove them using pip, and if that doesn’t break dnf, reinstall the system packages using dnf. If it breaks dnf, the simplest thing to do is to backup your data and reinstall afresh.

Without virtual environments pip knows about all packages installed in both the default system and user locations, and it cannot tell the difference between stuff you installed using pip and stuff you installed using dnf. dnf, however, only installs packages in system wide locations. So, for packages installed with pip using --user, you can use the corresponding command to get a complete list:

pip freeze --user

This will not include system wide packages (ones installed with sudo and without --user).

If you did install packages system wide with pip, you can get the list of python packages installed by pip and dnf and then compare the two:

 # get all Python packages pip can find in the default paths
$ pip freeze | cut -d "=" -f1  | sort -h | uniq
aiodns       
aiofiles       
aiohttp         
aiohttp-cors   
aiohttp-socks           
airspeed      
...

# get list of all dnf installed python packages
$ rpm --qf '%{NAME}\n' -qa python3\* | cut -d "-" -f2 | sort -h | uniq
abrt
aiodns
aiofiles
aiohttp
airspeed
apipkg
appdirs
argcomplete
argon2
astroid
async
atomicwrites
attrs
audit
augeas
...

I’d just save these to files (add > filename.txt at the end of the commands to save to filename.txt) and then use diff to see what’s in one and not in the other:

diff file1 file2
1 Like

If you’ve been using sudo, pip has installed stuff in the system locations

usr/lib/python2.7/site-packages🔒
 ⟹  ls
usr/lib/python3.9/site-packages🔒 
 ⟹  ls
chardet                             py_swtpm_localca       pygments  
swtpm_localca-0.5.2-py3.9.egg-info  torbrowser_launcher
packaging                           py_swtpm_setup    
requests                            swtpm_setup-0.5.2-py3.9.egg-info

Python2 doesn’t have any packages installed from pip. But Python3 does seem to have some of the packages. Are they okay/normal packages, or should I uninstall them all?

[I will post a follow-up-comment about your comment’s second part later in few hours by sorting the packages names.]

is this in the system packages or in your user packages (in ~/.local/lib)?

dnf will never install anything into your ~/, so these will all be from pip. you can remove them without any worries. dnf only installs in system locations.

I was looking at the pip freeze man page, and saw that it also has a --user option. That should tell you what’s installed by pip in your user directories.

pip freeze --user

The sorting etc. commands above is only required if you’ve used sudo to install in the system wide locations. (I’ve also edited my initial answer to note this now).

1 Like

That –user option is a gem. Thanks a lot.

But why do pip freeze --user when we can do pip list --user which makes more sense!

Anyway thanks.

is this in the system packages or in your user packages

Abovels command output is used inside “usr/lib/python2.7/site-packages” and “usr/lib/python3.9” directory. So they are system directories. I forget to put usr path which I have fixed it now.

A new thing that I didn’t see is these massive packages inside python3.10 directory:

  ⟹  ls
__pycache__                               firewall                             pyenchant-3.2.2-py3.10.egg-info
_distutils_hack                           fros-1.1-py3.10.egg-info             pyfros
_version.py                               gi                                   pygments
abrt3.pth                                 gtweak                               Pygments-2.9.0.dist-info
abrt_exception_handler3.py                html2text                            pygtkcompat
abrtcli                                   html2text-2020.1.16-py3.10.egg-info  pyOpenSSL-21.0.0-py3.10.egg-info
aiodns                                    html5lib                             pyparsing-2.4.7.dist-info
aiodns-3.0.0-py3.10.egg-info              html5lib-1.1-py3.10.egg-info         pyparsing.py
aiofiles                                  humanize                             PySocks-1.7.1-py3.10.egg-info
aiofiles-0.7.0.dist-info                  humanize-0.5.1-py3.10.egg-info       python_augeas-1.1.0-py3.10.egg-info
aiohttp_cors                              idna                                 python_dateutil-2.8.1-py3.10.egg-info
aiohttp_cors-0.7.0-py3.10.egg-info        idna-3.2-py3.10.egg-info             python_utils
argcomplete                               ifaddr                               python_utils-2.5.6-py3.10.egg-info
argcomplete-1.12.3-py3.10.egg-info        ifaddr-0.1.7-py3.10.egg-info         pytz
async_timeout                             init_calibre.py                      pytz-2021.3-py3.10.egg-info
async_timeout-3.0.1-py3.10.egg-info       jeepney                              pyxdg-0.27-py3.10.egg-info
attr                                      jeepney-0.7.1-py3.10.egg-info        raven
attrs-21.2.0-py3.10.egg-info              jinja2                               raven-6.10.0-py3.10.egg-info
augeas                                    Jinja2-3.0.1-py3.10.egg-info         requests
augeas.py                                 jsonschema                           requests-2.26.0.dist-info
babel                                     jsonschema-3.2.0-py3.10.egg-info     requests_gssapi
Babel-2.9.1-py3.10.egg-info               koji                                 requests_gssapi-1.2.3-py3.10.egg-info
beaker                                    koji-1.26.1.dist-info                rpmautospec
Beaker-1.10.0-py3.10.egg-info             koji_cli                             rpmautospec-0.2.5-py3.10.egg-info
beautifulsoup4-4.9.3-py3.10.egg-info      louis                                rpmconf
bs4                                       mako                                 rpmconf-1.1.4-py3.10.egg-info
chardet                                   Mako-1.1.4-py3.10.egg-info           seobject.py
chardet-4.0.0.dist-info                   markdown                             sepolgen
charset_normalizer                        Markdown-3.3.4-py3.10.egg-info       sepolicy
charset_normalizer-2.0.4.dist-info        mechanize                            sepolicy-3.3-py3.10.egg-info
chrome_gnome_shell-0.0.0-py3.10.egg-info  mechanize-0.4.5-py3.10.egg-info      setuptools
click                                     nftables                             setuptools-57.4.0.dist-info
click-8.0.1.dist-info                     nftables-0.1-py3.10.egg-info         sgmllib.py
configobj-5.0.6-py3.10.egg-info           odf                                  sgmllib3k-1.0.0-py3.10.egg-info
configobj.py                              odfpy-1.4.1-py3.10.egg-info          six-1.16.0.dist-info
cpuinfo                                   olefile                              six.py
css_parser                                olefile-0.46-py3.10.egg-info         socks.py
css_parser-1.0.6-py3.10.egg-info          OpenSSL                              sockshandler.py
cupshelpers                               orca                                 sos
cupshelpers-1.0-py3.10.egg-info           packaging                            sos-4.2-py3.10.egg-info
dateutil                                  packaging-21.0.dist-info             soupsieve
decorator-5.0.9-py3.10.egg-info           paste                                soupsieve-2.3.1-py3.10.egg-info
decorator.py                              Paste-3.5.0-py3.10-nspkg.pth         tempita
defusedxml                                Paste-3.5.0-py3.10.egg-info          Tempita-0.5.2-py3.10.egg-info
defusedxml-0.7.1-py3.10.egg-info          pexpect                              test
distro-1.6.0-py3.10.egg-info              pexpect-4.8.0-py3.10.egg-info        torbrowser_launcher
distro.py                                 pip                                  torbrowser_launcher-0.3.5-py3.10.egg-info
distutils-precedence.pth                  pip-21.2.3.dist-info                 typing_extensions-3.7.4.3-py3.10.egg-info
dnf                                       pkg_resources                        typing_extensions.py
dnf-plugins                               ply                                  urllib3
dnfpluginscore                            ply-3.11-py3.10.egg-info             urllib3-1.26.6-py3.10.egg-info
dnfpluginsextras                          progressbar                          validate.py
dns                                       progressbar2-3.53.2-py3.10.egg-info  webencodings
dnspython-2.1.0-py3.10.egg-info           ptyprocess                           webencodings-0.5.1-py3.10.egg-info
enchant                                   ptyprocess-0.6.0-py3.10.egg-info     xdg
fedora_third_party                        py_cpuinfo-8.0.0-py3.10.egg-info     zeroconf
fedora_third_party-0.8-py3.10.egg-info    pyatspi                              zeroconf-0.36.9-py3.10.egg-info
feedparser                                pycparser
feedparser-6.0.6-py3.10.egg-info          pycparser-2.20-py3.10.egg-info
  ⟹

But none of these packages are installed by me. Because Python3.10 came with the release of Fedora 35, AKAICS.

Another good thing is there is no user-installed packages that I see when I type:

  ⟹  pip list --user
~
  ⟹

Which seems a bit weird. But it is what it is.

pip list --user lists only packages installed in your user space, that is, the packages in $HOME/.local/lib/python3.x/site-packages. The fact that you have none means that you haven’t used pip install --user to install any packages. But that’s a fairly different thing than how your question started out.

As far as packages in /usr/lib*/python3.10/site-packages/, while you can use pip to examine those, you shouldn’t use pip to manage them at all, because those packages are under the control of rpm. (By the same token, you should never sudo pip install ____ anything, as you’ll be messing up the system-controlled package collection.)

Pip does have a --not-required flag to list, which will exclude packages that are required by another package. However, most of the packages in /usr/lib*/python3.x/site-packages/ will show up as not-required, because they aren’t required by another Python package… that doesn’t mean they aren’t required by one or more RPMs. Again, pip isn’t really the tool for managing the system package collection.

Note, also, that dnf repoquery --userinstalled doesn’t completely mean what it appears to. Packages are automatically marked as “user installed” by dnf if they were installed by direct request of the user, and not marked if they were pulled in as dependencies. But any package can be marked as user-installed by running sudo dnf mark install <package>, and any package can be marked as non-user-installed by running sudo dnf mark remove <package>.

Those flags are used by the dnf autoremove system, which ignores any package listed as user-installed (along with any package that’s a dependency of another package) when computing which packages can be autoremoved.

If you wanted to remove all packages that aren’t required by another package from /usr/lib*/python3.x/site-packages, the easiest way would be to run sudo dnf mark remove python3-\* – that will unmark ALL Python RPMs as being user-installed, which means that any which aren’t required by another RPM will now be on the autoremove chopping block. (You can see what’s auto-removable by running sudo dnf list autoremove, and remove them with sudo dnf autoremove.)

Pip doesn’t have a concept of “user-installed” vs. not, because every package is user-installed as far as pip is concerned. The “system” package collection is totally a construct of the RPM packaging system, which is why those packages should be managed the same way.

2 Likes