This article is to make a note of confusing phenomena when dealing with Python, virtualenv, pip, and Archlinux.

Background

The background isn't important here (- though what is important, is to always read the date of writing of an article that talks about Python!). What I was trying to figure out, was really what happens if one attempts to require specific versions of python, or pip, within a specific virtualenv environment. And I set off to trial and error.

Naive first tries

First, I need to at least get one verison of everything going.

sudo pacman -S python python-pip
sudo pip install virtualenv
cd some/path/somewhere
virtualenv venv

Now, seems like virtualenv would be working fine. And it is. In this state, the versions of the stuff would be:

  • python: 3
  • pip: 3
  • virtualenv: python inside the virtual environment gives python3.

Different pip's install "different" virtualenv's

What happens if I have multiple versions of different things?

sudo pacman -S python2 python2-pip

Now things starts to get weird. We now have:

  • python,
  • python2,
  • pip,
  • pip2,

but no virtualenv yet. What would be the difference between using

sudo pip install virtualenv

and using

sudo pip2 install virtualenv

? Well, I gave both a few tries. As it turns out:

  • If I use pip install, then some stuff gets installed into /usr/lib/python3.3/site-packages;
  • If I use pip2 install, then some stuff gets installed into /usr/lib/python2.7/site-packages.

In both cases, I always get a virtualenv command (which is always /usr/bin/virtualenv).

  • If I use pip2 install, if I use virtualenv venv then activate, then which python gives me venv/bin/python, which is a python2.
  • If I use pip install, I get python3 in the venv instead.

update: pip2 install gives all of python, python2, python3, pip, pip2, pip3 in the virtual environment, while pip install only gives python, python3, pip, pip3.

$ sudo pip2 install virtualenv
$ virtualenv venv/
New python executable in venv/bin/python2
Not overwriting existing python script venv/bin/python (you must use venv/bin/python2)
Installing setuptools, pip...done.

...
$ sudo pip2 uninstall virtualenv
...

$ sudo pip install virtualenv
$ virtualenv venv
Using base prefix '/usr'
New python executable in venv/bin/python
Installing setuptools, pip...done.

I learned from Stackoverflow that on creation of the virtual environment, one of the versions of python installed on the computer can be specified to be used in this virtual environment. Because it's confusing enough, I didn't care much about it. I tried it, and it worked; that's all I cared about it at this point.

I've noticed that the output of virtualenv installed by different pip's are different. If installed with pip2, the output will contain

  -p PYTHON_EXE, --python=PYTHON_EXE
                        The Python interpreter to use, e.g.,
                        --python=python2.5 will use the python2.5 interpreter
                        to create the new environment.  The default is the
                        interpreter that virtualenv was installed with
                        (/usr/bin/python2)

Notice the "/usr/bin/python2" there. If I installed virtualenv with pip install, that text will be "/usr/bin/python".

However, if I first did pip2 install, then did pip install, I guess the latter command didn't overwrite the virtualenv command in /usr/bin, because the output of virtualenv still contained "/use/bin/python2" rather than without the "2". (update: - and the two virtualenv command behave differently as well. See the "update" above.)

However again, I do think that virtualenv is installed twice, because pip list and pip2 list gives results that correspond reasonably with the state of whether and from where I have installed virtualenv.

I also think that "the" virtualenv that was installed twice, is indeed the same virtualenv, because I tried using the "--download-cache" command line option to point both pip and pip2 to the same cache directory when installing virtualenv, and they both worked. (update: They're the same virtualenv, but some configuration is different - so they behave differently. See previous "update".)

As extra info: I realised that Archlinux provided two virtualenv packages, one for python2 and the other for python3, named python-virtualenv and python2-virtualenv:

$ pacman -Ss virtualenv
extra/python-virtualenv 1.11.2-1
    Virtual Python Environment builder
extra/python2-virtualenv 1.11.2-1
    Virtual Python Environment builder

I didn't even bother.

"Different" virtualenv's generate the same pip

Seen above, the same virtualenv (1.11.2) installed by different pip's (into different locations) will have different default values of which python each generated virtual environment will give you. (update: - and they behave differently. See "update" above.)

However, the pip command inside the virtual environment is always the same one: not pip2 - regardless of whether virtualenv has been installed using pip or pip2. Inside the virtual environment, pip and pip3 always gave the same thing, which is inside the venv/ directory, while pip2 always gives the system's global environment. In other words, pip list always gives two clean lines (pip and setuptools), and pip2 list always gives a long list (full of packages installed in the system environment).

Conclusion

There is no conceptal conclusion.

The factual conclusion is:

  • Different pip's give you the same virtualenv installed in different locations and which uses different default values for the python command inside the virtual environment.
  • Regardless of which version of python you get inside the virtual environment, the pip in there is never pip2.

Just use sudo pip2 install virtualenv, and never use pip install. Both version 2 & 3 of both Python and pip works inside there.