.. title: How to Use Python Virtualenv
.. slug: how-to-use-python-virtualenv
.. date: 2017-11-01 01:43:30 UTC
.. updated: 2017-11-01 01:43:30 UTC
.. tags: python
.. category:
.. link:
.. description:
.. type: text

A Python virtualenv is an isolated directory that serves as the root filesystem
(kind of) for an installation of Python. In simpler terms, Python is installed
in a non-privileged directory by a regular user. Anything installed for that
instance of Python does not affect any other Python instance installed on the
system.

This guide will show you how each instance of a Python interpreter differs from
others and how to use this to your advantage.

.. TEASER_END: Read more

I like to create a directory in my home directory to store all virtualenvs.

::

    $ mkdir ~/virtualenvs

Install Python
--------------

Install Python 2.7, 3.5 and 3.6.

If you're using MacPorts (recommended on macOS).

::

    $ sudo port install python27 py27-pip py27-virtualenv
    $ sudo port install python35 py35-pip py35-virtualenv
    $ sudo port install python36 py36-pip py36-virtualenv

Apologies to my Linux using friends. I don't have instructions on how to
install different versions of Python because each distro has its own process
and policy. Refer to the distro's docs for more information.

Verify all versions of Python have ``pip`` and ``virtualenv`` modules
installed.

::

    $ python2.7 -m pip --version
    $ python2.7 -m virtualenv --version
    $ python3.5 -m pip --version
    $ python3.5 -m virtualenv --version
    $ python3.6 -m pip --version
    $ python3.6 -m virtualenv --version

Create Virtualenv
-----------------

Create virtualenvs *py27*, *py35*, and *py36*. These names could be anything.
In this case they're more descriptive than something else I could have picked.

::

    $ python2.7 -m virtualenv ~/virtualenvs/py27
    $ python3.5 -m virtualenv ~/virtualenvs/py35
    $ python3.6 -m virtualenv ~/virtualenvs/py36

In my daily work I create separate virtualenvs for each project I work with.
For example, I have different virtualens for Ansible under Python 2.7
(called *ansible-py27*) and Ansible under Python 3.6 (called
*ansible-py36*).

You can (and should) create as many virtualenvs as needed. The only
considerations are remembering which ones to use for which purpose.

Basic Management
----------------

The easiest way to use a virtualenv is to *activate* it first. This step
modifies your PATH environment variable and other things to make it so when you
run ``python`` it runs the interpreter from the virtualenv.

For example, these are my system-wide installs of various Python interpreters.

::

    $ which -a python
    /usr/bin/python

    $ /usr/bin/python --version
    Python 2.7.10



    $ which -a python2.7
    /opt/local/bin/python2.7
    /usr/bin/python2.7

    $ /opt/local/bin/python2.7 --version
    Python 2.7.14

    $ /usr/bin/python2.7 --version
    Python 2.7.10



    $ which -a python3
    /opt/local/bin/python3

    $ /opt/local/bin/python3 --version
    Python 3.5.4



    $ which -a python3.5
    /opt/local/bin/python3.5

    $ /opt/local/bin/python3.5 --version
    Python 3.5.4



    $ which -a python3.6
    /opt/local/bin/python3.6

    $ /opt/local/bin/python3.6 --version
    Python 3.6.3

Let's activate Python 3.6 virtualenv called *py36* above.

::

    $ source ~/virtualenvs/py36/bin/activate

The above step prepends the name of the virtualenv to the prompt. This is
visual information on whether a virtualenv is currently active or not. For
example, in my case the prompt becomes.

::

    (py36) $

Now let's re-run the ``which`` commands above with the virtualenv *py36*
activated.

::

    (py36) $ which -a python
    /Users/codeghar/virtualenvs/py36/bin/python
    /usr/bin/python

    (py36) $ /Users/codeghar/virtualenvs/py36/bin/python --version
    Python 3.6.3

    (py36) $ /usr/bin/python --version
    Python 2.7.10



    (py36) $ which -a python2.7
    /opt/local/bin/python2.7
    /usr/bin/python2.7

    (py36) $ /opt/local/bin/python2.7 --version
    Python 2.7.14

    (py36) $ /usr/bin/python2.7 --version
    Python 2.7.10



    (py36) $ which -a python3
    /Users/codeghar/virtualenvs/py36/bin/python3
    /opt/local/bin/python3

    (py36) $ /Users/codeghar/virtualenvs/py36/bin/python3 --version
    Python 3.6.3

    (py36) $ /opt/local/bin/python3 --version
    Python 3.5.4



    (py36) $ which -a python3.5
    /opt/local/bin/python3.5

    (py36) $ /opt/local/bin/python3.5 --version
    Python 3.5.4



    (py36) $ which -a python3.6
    /Users/codeghar/virtualenvs/py36/bin/python3.6
    /opt/local/bin/python3.6

    (py36) $ /Users/codeghar/virtualenvs/py36/bin/python3.6 --version
    Python 3.6.3

    (py36) $ /opt/local/bin/python3.6 --version
    Python 3.6.3

Notice how ``which -a python``, ``which -a python3``, and
``which -a python3.6`` have additional paths. By activating the virtualenv,
PATH was modified to include the virtualenv's *bin* directory as the first path
to search.

To revert the changes made by activating the virtualenv run ``deactivate``.

::

    (py36) $ deactivate

The prompt also returns to your regular prompt to indicate the virtualenv has
been deactivated.

::

    $

Install Packages
----------------

You install packages using ``pip``. Although your system may have ``pip``,
``pip-2.7``, ``pip-3.5``, or ``pip-3.6`` or other variants installed, it's
better to invoke ``pip`` using the specific Python version you want to use.
For example,

::

    $ python3.6 -m pip

instead of

::

    $ pip-3.6

These should be equivalent invocations of ``pip`` but it's a good habit to
invoke it from the interpreter especially when you have lots of versions of
Python installed and have many virtualenvs available as well.

To install packages system-wide you need root permission. This is **highly
discouraged** since we want to use virtualenvs to manage our environments.
Nevertheless, the way to do this is.

::

    $ sudo /opt/local/bin/python3.6 -m pip install pytest

``pip`` has the ability to install packages in your home directory when the
``--user`` flag is used.. For this you don't need root permissions.

::

    $ /opt/local/bin/python3.6 -m pip install --user pytest

In my environment the system-wide packages installed for use with the Python
3.6 interpreter are.

::

    $ /opt/local/bin/python3.6 -m pip list
    appdirs (1.4.3)
    certifi (2017.1.23)
    packaging (16.8)
    pip (9.0.1)
    pyparsing (2.2.0)
    readline (6.2.4.1)
    setuptools (34.3.1)
    six (1.10.0)
    virtualenv (15.1.0)

For the same interpreter I have no packages installed in my home directory.

::

    $ /opt/local/bin/python3.6 -m pip list --user
    <empty line>

Let's install *pytest* in the home directory.

::

    $ /opt/local/bin/python3.6 -m pip install --user pytest
    <SNIP>
    Successfully installed py-1.4.34 pytest-3.2.3

Now two packages show up as installed in my home directory under the Python 3.6
interpreter.

::

    $ /opt/local/bin/python3.6 -m pip list --user
    py (1.4.34)
    pytest (3.2.3)

Let's import *pytest* in Python3.6.

::

    $ /opt/local/bin/python3.6
    Python 3.6.3 (default, Oct  5 2017, 23:34:28)
    [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import pytest
    >>>

Let's import *pytest* in Python 3.5. This fails because *pytest* was installed
under Python 3.6 and not under Python 3.5.

::

    $ /opt/local/bin/python3.5
    Python 3.5.4 (default, Sep 22 2017, 08:33:07)
    [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import pytest
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    ImportError: No module named 'pytest'
    >>>

The lesson learned here is each package is installed under a specific
interpreter. This is the functionality that powers virtualenvs. Since we
activate a virtualenv we use an independent interpreter which manages its own
packages and environment.

Let's install *pytest* in our *py36* virtualenv.

::

    $ source ~/virtualenvs/py36/bin/activate


    (py36) $ pip list
    appdirs (1.4.3)
    packaging (16.8)
    pip (9.0.1)
    pyparsing (2.2.0)
    setuptools (34.3.2)
    six (1.10.0)
    wheel (0.29.0)



    (py36) $ pip install pytest
    <SNIP>
    Successfully installed py-1.4.34 pytest-3.2.3



    (py36) $ python3.6
    Python 3.6.3 (default, Oct  5 2017, 23:34:28)
    [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import pytest
    >>>



    (py36) $ python
    Python 3.6.3 (default, Oct  5 2017, 23:34:28)
    [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import pytest
    >>>

Notice we didn't use ``--user`` flag or ``sudo`` here. It's because system-wide
in the context of a virtualenv is restricted to the virtualenv directory, which
in this case is ~/virtualenvs/py36/.

This point bears repeating: anything we install in a virtualenv does not
pollute anything outside of the virtualenv.

While a virtualenv is activated I can still use other Python interpreters or
packages they have installed.

Let's keep the *py36* virtualenv activated and install *fabric3* outside of
the virtualenv.

::

    (py36) $ /opt/local/bin/python3.6 -m pip install --user fabric3
    <SNIP>
    Successfully installed asn1crypto-0.22.0 cffi-1.9.1 cryptography-1.8.1 fabric3-1.13.1.post1 idna-2.5 paramiko-2.1.2 pyasn1-0.2.3 pycparser-2.17


    (py36) $ /opt/local/bin/python3.6 -m pip list
    appdirs (1.4.3)
    asn1crypto (0.22.0)
    certifi (2017.1.23)
    cffi (1.9.1)
    cryptography (1.8.1)
    Fabric3 (1.13.1.post1)
    idna (2.5)
    packaging (16.8)
    paramiko (2.1.2)
    pip (9.0.1)
    py (1.4.32)
    pyasn1 (0.2.3)
    pycparser (2.17)
    pyparsing (2.2.0)
    pytest (3.2.3)
    readline (6.2.4.1)
    setuptools (34.3.1)
    six (1.10.0)
    virtualenv (15.1.0)


    (py36) $ python3.6 -m pip list
    appdirs (1.4.3)
    packaging (16.8)
    pip (9.0.1)
    py (1.4.32)
    pyparsing (2.2.0)
    pytest (3.2.3)
    setuptools (34.3.2)
    six (1.10.0)
    wheel (0.29.0)

Packages installed outside of the virtualenv are not visible and usable by the
interpreter in virtualenv.

But virtualenvs don't prevent us from running another interpreter and using
the packages installed in it.

Let's run pytest in the virtualenv.

::

    (py36) ~ $ pytest .
    ==== test session starts ====
    platform darwin -- Python 3.6.3, pytest-3.2.3, py-1.4.32, pluggy-0.4.0
    rootdir: /Users/codeghar, inifile:
    ^C
    !!!! KeyboardInterrupt !!!!
    to show a full traceback on KeyboardInterrupt use --fulltrace
    /Users/codeghar/virtualenvs/py36/lib/python3.6/site-packages/py/_path/local.py:360: KeyboardInterrupt
    ==== no tests ran in 2.56 seconds ====

Now let's uninstall *pytest* from virtualenv and from home directory under
Python3.6.

::

    (py36) $ python3.6 -m pip uninstall -y pytest
    Uninstalling pytest-3.2.3:
    Successfully uninstalled pytest-3.2.3



    (py36) $ /opt/local/bin/python3.6 -m pip uninstall -y pytest
    Uninstalling pytest-3.2.3:
    Successfully uninstalled pytest-3.2.3

Let's install *pytest* under Python 3.5.

::

    (py36) $ /opt/local/bin/python3.5 -m pip install --user pytest
    <SNIP>
    Successfully installed pytest-3.2.3

Now the situation is that virtualenv doesn't have pytest installed,
/opt/local/bin/python3.6 doesn't have it installed but /opt/local/bin/python3.5
does in the home directory. Let's run ``pytest``.

::

    (py36) $ pytest .
    -bash: /Users/codeghar/virtualenvs/py36/bin/pytest: No such file or directory



    (py36) $ python3.5 -m pytest .
    ==== test session starts ====
    platform darwin -- Python 3.5.0, pytest-3.2.3, py-1.4.32, pluggy-0.4.0
    rootdir: /Users/codeghar, inifile:
    plugins: instafail-0.3.0, hidecaptured-0.1.2, cov-2.3.1
    ^C
    !!!! KeyboardInterrupt !!!!
    to show a full traceback on KeyboardInterrupt use --fulltrace
    /Users/codeghar/Library/Python/3.5/lib/python/site-packages/py/_path/local.py:341: KeyboardInterrupt
    ==== no tests ran in 2.56 seconds ====

What if *pytest* was installed system-wide for Python3.5 not in the home
directory?

::

    (py36) $ python3.5 -m pip uninstall -y pytest
    Uninstalling pytest-3.2.3:
    Successfully uninstalled pytest-3.2.3



    (py36) $ sudo python3.5 -m pip install pytest
    <SNIP>
    Successfully installed pytest-3.2.3

Now the situation is that virtualenv doesn't have pytest installed,
/opt/local/bin/python3.6 doesn't have it installed but /opt/local/bin/python3.5
does system-wide. Let's run ``pytest``.

::

    (py36) $ pytest .
    -bash: /Users/codeghar/virtualenvs/py36/bin/pytest: No such file or directory



    (py36) $ python3.5 -m pytest .
    ==== test session starts ====
    platform darwin -- Python 3.5.0, pytest-3.2.3, py-1.4.32, pluggy-0.4.0
    rootdir: /Users/codeghar, inifile:
    plugins: instafail-0.3.0, hidecaptured-0.1.2, cov-2.3.1
    ^C
    !!!! KeyboardInterrupt !!!!
    to show a full traceback on KeyboardInterrupt use --fulltrace
    /Users/codeghar/Library/Python/3.5/lib/python/site-packages/py/_path/local.py:301: KeyboardInterrupt
    ==== no tests ran in 2.56 seconds ====

It's the same thing as before.

One Last Thing
--------------

The *activate* step modifies the PATH variable. This means that you can also
use the full path to ``python`` or any other executable in the virtualenv's
*bin* directory to execute it without activating the virtualenv.

For example, the following are equivalent:

::

    $ /Users/codeghar/virtualenvs/py36/bin/pytest .


    $ source /Users/codeghar/virtualenvs/py36/bin/activate
    (py36) $ pytest .

This comes in handy when you are unable to split your steps between activating
the virtualenv and executing some binary installed in it. For example,
sometimes it's easier to use the full path in a *Dockerfile*. Use what works
better for the situation.

Conclusion
----------

I hope this tutorial has demonstrated the mechanics of Python's virtualenv. The
final takeaways from this are:

* Always use virtualenvs
* Create as many virtualenvs as you need
* Do not install packages system-wide; instead, use virtualenvs
* Prefer to use virtualenvs even in Docker containers
* Invoke ``pip`` and ``virtualenv`` under the interpreter you mean to use (e.g. ``/opt/local/bin/python3.6 -m pip``)