User-site-installed python packages, and PATH modification

Python makes it apparently easy to install packages. Just use pip, or any of the other more-or-less old and deprecated ways to install them, right? (heh)

The first difficulty is that maybe your system's Python needs sudo to install those packages, and you don't want (or even can't) use it.

The definitive solution is to use virtual environments, but that can feel like going too far in the "local" direction. You might just want to have something at the user level, without having the risks of using sudo, but still global for everything that the user does.

Well, turns out that PEP 370 allows you to have user-local installs of packages. You're supposed to run pip --user install whatever.

But now you have to remember to always use the --user flag! Kinda breaks the purpose of having something "global for everything that the user does".


If you want to do that by default, so any future "pip install" does install to the user-site, you can add a pip configuration file telling pip to do so. In MacOS, the file must be located at ~/Library/Application Support/pip/pip.conf, and contains

[global]
user = true

(BEWARE! If you use virtual environments, note that thanks to yet-another-example of Python brain damage/unfinished business, as of Python 3.5.3 and pip 9 the --user argument causes problems with installs in virtual environments of both venv and virtualenv varieties... and looks like the problem comes rather from the argument, not from the functionality! A Quick&Dirty fix is to do something like export PIP_CONFIG_FILE=/dev/null inside the virtual environment, so in that case pip will ignore the pip.conf file. This could be further automated with hooks in virtualenvwrapper, but it starts to feel like a pile-up of hacks...)

OK, so now you have user-local installs by default. But those user-local installs won't have their commands in your PATH, so you need to set that too.

You could go the naïve way and just add something like

PATH="~/Library/Python/3.5/bin/:$PATH"

to your .profile. But, note: here we're hardwiring python 3.5 (or any other version). Is that what you want? Is the rest of your system so hardwired? Unless you specifically did so, the answer probably is: no, it isn't. So you might end up mixing Python installs/environments.


I couldn't find an official solution (or even acknowledgement) to this problem. However, Ruby does offer a very nice FAQ example for their case [1]. Following their example, I ended up with this Python snippet:

PATH="$(python -c 'import site; print(site.USER_BASE)')/bin:$PATH"

 ... which asks the current python interpreter for its user-site directory.

But this is not enough yet, because if you use something like MacPorts you can perfectly be in a situation where you have activated different Python versions for the interpreter and for PIP (try port select --show python and port select --show pip). So this can cause a situation where (for example) your "python" command runs python 2.7, while "pip" runs the pip from python 3.5, and any pip-installed packages/commands are therefore for 3.5. Confusing!

So, again following Ruby's example, you can add some guarding to make sure that at least both versions are the same.

PYTHON_WISHED_VERSION=35
if port select --show python | grep $PYTHON_WISHED_VERSION > /dev/null &&
   port select --show pip | grep $PYTHON_WISHED_VERSION > /dev/null ; then
    PATH="$(python -c 'import site; print(site.USER_BASE)')/bin:$PATH"
else
    echo Python and PIP not at version $PYTHON_WISHED_VERSION: not adding the user-site PATH
fi

Of course here we're hardwiring Python 3.5 again :P. However, at least now it's self-consistent and self-checking: if there is a version mismatch it will complain, instead of going on obliviously. And, well, there's a single number to change to make it work with any other version.

I guess that instead of mandating a version number you could make it auto-detect the python version, check that pip is at the same version, and then use the corresponding path; but my use case doesn't involve such frequent/flexible version changes.

Anyway, perhaps this could be done more elegantly in other ways, but again Python is not my workhorse and I only care here about CLI environment sanity – inside and outside of Python. If you can improve this, let me know!




[1] Note to self: it's interesting how every time I peek a bit into Ruby it looks so much nicer than Python. I really need to get into it.