How to Use Python Virtualenv

Install Python

Install Python 2.7, 3.5 and 3.6.

If you're using MacPorts (recommended on macOS).

$ sudo port install python27 py27-pip py27-virtualenv
$ sudo port install python35 py35-pip py35-virtualenv
$ sudo port install python36 py36-pip py36-virtualenv

Apologies to my Linux using friends. I don't have instructions on how to install different versions of Python because each distro has its own process and policy. Refer to the distro's docs for more information.

Verify all versions of Python have pip and virtualenv modules installed.

$ python2.7 -m pip --version
$ python2.7 -m virtualenv --version
$ python3.5 -m pip --version
$ python3.5 -m virtualenv --version
$ python3.6 -m pip --version
$ python3.6 -m virtualenv --version

Create Virtualenv

Create virtualenvs py27, py35, and py36. These names could be anything. In this case they're more descriptive than something else I could have picked.

$ python2.7 -m virtualenv ~/virtualenvs/py27
$ python3.5 -m virtualenv ~/virtualenvs/py35
$ python3.6 -m virtualenv ~/virtualenvs/py36

In my daily work I create separate virtualenvs for each project I work with. For example, I have different virtualens for Ansible under Python 2.7 (called ansible-py27) and Ansible under Python 3.6 (called ansible-py36).

You can (and should) create as many virtualenvs as needed. The only considerations are remembering which ones to use for which purpose.

Basic Management

The easiest way to use a virtualenv is to activate it first. This step modifies your PATH environment variable and other things to make it so when you run python it runs the interpreter from the virtualenv.

For example, these are my system-wide installs of various Python interpreters.

$ which -a python
/usr/bin/python

$ /usr/bin/python --version
Python 2.7.10



$ which -a python2.7
/opt/local/bin/python2.7
/usr/bin/python2.7

$ /opt/local/bin/python2.7 --version
Python 2.7.14

$ /usr/bin/python2.7 --version
Python 2.7.10



$ which -a python3
/opt/local/bin/python3

$ /opt/local/bin/python3 --version
Python 3.5.4



$ which -a python3.5
/opt/local/bin/python3.5

$ /opt/local/bin/python3.5 --version
Python 3.5.4



$ which -a python3.6
/opt/local/bin/python3.6

$ /opt/local/bin/python3.6 --version
Python 3.6.3

Let's activate Python 3.6 virtualenv called py36 above.

$ source ~/virtualenvs/py36/bin/activate

The above step prepends the name of the virtualenv to the prompt. This is visual information on whether a virtualenv is currently active or not. For example, in my case the prompt becomes.

(py36) $

Now let's re-run the which commands above with the virtualenv py36 activated.

(py36) $ which -a python
/Users/codeghar/virtualenvs/py36/bin/python
/usr/bin/python

(py36) $ /Users/codeghar/virtualenvs/py36/bin/python --version
Python 3.6.3

(py36) $ /usr/bin/python --version
Python 2.7.10



(py36) $ which -a python2.7
/opt/local/bin/python2.7
/usr/bin/python2.7

(py36) $ /opt/local/bin/python2.7 --version
Python 2.7.14

(py36) $ /usr/bin/python2.7 --version
Python 2.7.10



(py36) $ which -a python3
/Users/codeghar/virtualenvs/py36/bin/python3
/opt/local/bin/python3

(py36) $ /Users/codeghar/virtualenvs/py36/bin/python3 --version
Python 3.6.3

(py36) $ /opt/local/bin/python3 --version
Python 3.5.4



(py36) $ which -a python3.5
/opt/local/bin/python3.5

(py36) $ /opt/local/bin/python3.5 --version
Python 3.5.4



(py36) $ which -a python3.6
/Users/codeghar/virtualenvs/py36/bin/python3.6
/opt/local/bin/python3.6

(py36) $ /Users/codeghar/virtualenvs/py36/bin/python3.6 --version
Python 3.6.3

(py36) $ /opt/local/bin/python3.6 --version
Python 3.6.3

Notice how which -a python, which -a python3, and which -a python3.6 have additional paths. By activating the virtualenv, PATH was modified to include the virtualenv's bin directory as the first path to search.

To revert the changes made by activating the virtualenv run deactivate.

(py36) $ deactivate

The prompt also returns to your regular prompt to indicate the virtualenv has been deactivated.

$

Install Packages

You install packages using pip. Although your system may have pip, pip-2.7, pip-3.5, or pip-3.6 or other variants installed, it's better to invoke pip using the specific Python version you want to use. For example,

$ python3.6 -m pip

instead of

$ pip-3.6

These should be equivalent invocations of pip but it's a good habit to invoke it from the interpreter especially when you have lots of versions of Python installed and have many virtualenvs available as well.

To install packages system-wide you need root permission. This is highly discouraged since we want to use virtualenvs to manage our environments. Nevertheless, the way to do this is.

$ sudo /opt/local/bin/python3.6 -m pip install pytest

pip has the ability to install packages in your home directory when the --user flag is used.. For this you don't need root permissions.

$ /opt/local/bin/python3.6 -m pip install --user pytest

In my environment the system-wide packages installed for use with the Python 3.6 interpreter are.

$ /opt/local/bin/python3.6 -m pip list
appdirs (1.4.3)
certifi (2017.1.23)
packaging (16.8)
pip (9.0.1)
pyparsing (2.2.0)
readline (6.2.4.1)
setuptools (34.3.1)
six (1.10.0)
virtualenv (15.1.0)

For the same interpreter I have no packages installed in my home directory.

$ /opt/local/bin/python3.6 -m pip list --user
<empty line>

Let's install pytest in the home directory.

$ /opt/local/bin/python3.6 -m pip install --user pytest
<SNIP>
Successfully installed py-1.4.34 pytest-3.2.3

Now two packages show up as installed in my home directory under the Python 3.6 interpreter.

$ /opt/local/bin/python3.6 -m pip list --user
py (1.4.34)
pytest (3.2.3)

Let's import pytest in Python3.6.

$ /opt/local/bin/python3.6
Python 3.6.3 (default, Oct  5 2017, 23:34:28)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytest
>>>

Let's import pytest in Python 3.5. This fails because pytest was installed under Python 3.6 and not under Python 3.5.

$ /opt/local/bin/python3.5
Python 3.5.4 (default, Sep 22 2017, 08:33:07)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytest
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'pytest'
>>>

The lesson learned here is each package is installed under a specific interpreter. This is the functionality that powers virtualenvs. Since we activate a virtualenv we use an independent interpreter which manages its own packages and environment.

Let's install pytest in our py36 virtualenv.

$ source ~/virtualenvs/py36/bin/activate


(py36) $ pip list
appdirs (1.4.3)
packaging (16.8)
pip (9.0.1)
pyparsing (2.2.0)
setuptools (34.3.2)
six (1.10.0)
wheel (0.29.0)



(py36) $ pip install pytest
<SNIP>
Successfully installed py-1.4.34 pytest-3.2.3



(py36) $ python3.6
Python 3.6.3 (default, Oct  5 2017, 23:34:28)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytest
>>>



(py36) $ python
Python 3.6.3 (default, Oct  5 2017, 23:34:28)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytest
>>>

Notice we didn't use --user flag or sudo here. It's because system-wide in the context of a virtualenv is restricted to the virtualenv directory, which in this case is ~/virtualenvs/py36/.

This point bears repeating: anything we install in a virtualenv does not pollute anything outside of the virtualenv.

While a virtualenv is activated I can still use other Python interpreters or packages they have installed.

Let's keep the py36 virtualenv activated and install fabric3 outside of the virtualenv.

(py36) $ /opt/local/bin/python3.6 -m pip install --user fabric3
<SNIP>
Successfully installed asn1crypto-0.22.0 cffi-1.9.1 cryptography-1.8.1 fabric3-1.13.1.post1 idna-2.5 paramiko-2.1.2 pyasn1-0.2.3 pycparser-2.17


(py36) $ /opt/local/bin/python3.6 -m pip list
appdirs (1.4.3)
asn1crypto (0.22.0)
certifi (2017.1.23)
cffi (1.9.1)
cryptography (1.8.1)
Fabric3 (1.13.1.post1)
idna (2.5)
packaging (16.8)
paramiko (2.1.2)
pip (9.0.1)
py (1.4.32)
pyasn1 (0.2.3)
pycparser (2.17)
pyparsing (2.2.0)
pytest (3.2.3)
readline (6.2.4.1)
setuptools (34.3.1)
six (1.10.0)
virtualenv (15.1.0)


(py36) $ python3.6 -m pip list
appdirs (1.4.3)
packaging (16.8)
pip (9.0.1)
py (1.4.32)
pyparsing (2.2.0)
pytest (3.2.3)
setuptools (34.3.2)
six (1.10.0)
wheel (0.29.0)

Packages installed outside of the virtualenv are not visible and usable by the interpreter in virtualenv.

But virtualenvs don't prevent us from running another interpreter and using the packages installed in it.

Let's run pytest in the virtualenv.

(py36) ~ $ pytest .
==== test session starts ====
platform darwin -- Python 3.6.3, pytest-3.2.3, py-1.4.32, pluggy-0.4.0
rootdir: /Users/codeghar, inifile:
^C
!!!! KeyboardInterrupt !!!!
to show a full traceback on KeyboardInterrupt use --fulltrace
/Users/codeghar/virtualenvs/py36/lib/python3.6/site-packages/py/_path/local.py:360: KeyboardInterrupt
==== no tests ran in 2.56 seconds ====

Now let's uninstall pytest from virtualenv and from home directory under Python3.6.

(py36) $ python3.6 -m pip uninstall -y pytest
Uninstalling pytest-3.2.3:
Successfully uninstalled pytest-3.2.3



(py36) $ /opt/local/bin/python3.6 -m pip uninstall -y pytest
Uninstalling pytest-3.2.3:
Successfully uninstalled pytest-3.2.3

Let's install pytest under Python 3.5.

(py36) $ /opt/local/bin/python3.5 -m pip install --user pytest
<SNIP>
Successfully installed pytest-3.2.3

Now the situation is that virtualenv doesn't have pytest installed, /opt/local/bin/python3.6 doesn't have it installed but /opt/local/bin/python3.5 does in the home directory. Let's run pytest.

(py36) $ pytest .
-bash: /Users/codeghar/virtualenvs/py36/bin/pytest: No such file or directory



(py36) $ python3.5 -m pytest .
==== test session starts ====
platform darwin -- Python 3.5.0, pytest-3.2.3, py-1.4.32, pluggy-0.4.0
rootdir: /Users/codeghar, inifile:
plugins: instafail-0.3.0, hidecaptured-0.1.2, cov-2.3.1
^C
!!!! KeyboardInterrupt !!!!
to show a full traceback on KeyboardInterrupt use --fulltrace
/Users/codeghar/Library/Python/3.5/lib/python/site-packages/py/_path/local.py:341: KeyboardInterrupt
==== no tests ran in 2.56 seconds ====

What if pytest was installed system-wide for Python3.5 not in the home directory?

(py36) $ python3.5 -m pip uninstall -y pytest
Uninstalling pytest-3.2.3:
Successfully uninstalled pytest-3.2.3



(py36) $ sudo python3.5 -m pip install pytest
<SNIP>
Successfully installed pytest-3.2.3

Now the situation is that virtualenv doesn't have pytest installed, /opt/local/bin/python3.6 doesn't have it installed but /opt/local/bin/python3.5 does system-wide. Let's run pytest.

(py36) $ pytest .
-bash: /Users/codeghar/virtualenvs/py36/bin/pytest: No such file or directory



(py36) $ python3.5 -m pytest .
==== test session starts ====
platform darwin -- Python 3.5.0, pytest-3.2.3, py-1.4.32, pluggy-0.4.0
rootdir: /Users/codeghar, inifile:
plugins: instafail-0.3.0, hidecaptured-0.1.2, cov-2.3.1
^C
!!!! KeyboardInterrupt !!!!
to show a full traceback on KeyboardInterrupt use --fulltrace
/Users/codeghar/Library/Python/3.5/lib/python/site-packages/py/_path/local.py:301: KeyboardInterrupt
==== no tests ran in 2.56 seconds ====

It's the same thing as before.

One Last Thing

The activate step modifies the PATH variable. This means that you can also use the full path to python or any other executable in the virtualenv's bin directory to execute it without activating the virtualenv.

For example, the following are equivalent:

$ /Users/codeghar/virtualenvs/py36/bin/pytest .


$ source /Users/codeghar/virtualenvs/py36/bin/activate
(py36) $ pytest .

This comes in handy when you are unable to split your steps between activating the virtualenv and executing some binary installed in it. For example, sometimes it's easier to use the full path in a Dockerfile. Use what works better for the situation.

Conclusion

I hope this tutorial has demonstrated the mechanics of Python's virtualenv. The final takeaways from this are:

• Always use virtualenvs
• Create as many virtualenvs as you need
• Do not install packages system-wide; instead, use virtualenvs
• Prefer to use virtualenvs even in Docker containers
• Invoke pip and virtualenv under the interpreter you mean to use (e.g. /opt/local/bin/python3.6 -m pip)