Run multiple python versions on your system

I'm a software development consultant.  I write python (as well as other languages) code for many clients and I don't get to dictate what their environment looks like.  I've got clients running python as old as 2.4 while others are on the bleeding edge.  Additionally, each client may have their own packages installed as well as differing lists of third party packages.

This post is a description of how I went about getting multiple versions of python installed in my ubuntu development machine and how I go about managing different package sets for different clients.

Get multiple pythons installed

Ubuntu typically only supports one python 2.x version and one 3.x version at a time.  There's a popular ppa (personal package archive) called deadsnakes that contains older versions of python.  You can find it:

https://launchpad.net/~fkrull/+archive/deadsnakes

To install it (per the instructions in the link above), you do the following:

steve@ubuntu64 ~ $ sudo add-apt-repository ppa:fkrull/deadsnakes

Then you need to update your cache:

steve@ubuntu64 ~ $ sudo apt-get update

Finally, simply install the other versions (I'm running on ubuntu 12.04 LTS, so I have python 2.7 already):

steve@ubuntu64 ~ $ sudo apt-get install python2.4 python2.5 python2.6

If you're following along, we now have python versions 2.4 through 2.7 installed the computer.  If you run 'python', you'll see the default version is still 2.7.


steve@ubuntu64 ~ $ python
Python 2.7.3 (default, Apr 20 2012, 22:39:59) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

Here's why:  Each python version is stored in /usr/bin as python2.X where X is the version.  There is a symbolic link named python that points to the version you want to be the default.  Instead of typing python from the bash prompt, you could just as easily type python2.5:


steve@ubuntu64 ~ $ ls -l /usr/bin/python*
lrwxrwxrwx 1 root root       9 Apr 17 13:20 /usr/bin/python -> python2.7
lrwxrwxrwx 1 root root       9 Apr 17 13:20 /usr/bin/python2 -> python2.7
-rwxr-xr-x 1 root root 1216520 May 21 12:13 /usr/bin/python2.4
-rwxr-xr-x 1 root root 1403624 May  3 00:17 /usr/bin/python2.5
-rwxr-xr-x 1 root root 2652056 May 12 08:43 /usr/bin/python2.6
-rwxr-xr-x 1 root root 2993560 Apr 20 19:37 /usr/bin/python2.7


steve@ubuntu64 ~ $ python2.5
Python 2.5.6 (r256:88840, May  3 2012, 04:16:14) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 

This is a point of interest.  I would not mess with the symbolic link.  Ubuntu runs python for many internal maintenance scripts and those scripts are expecting the python version that shipped with ubuntu.

Use virtualenv to manage your python installations and package sets

So now that you have multiple versions of python on your system, how to manage them?  How do you keep packages installed for one version separate from packages installed for another.  What if you want to run one version of django for client X and a different version for client Y?

That's where virtualenv comes in.  If your a ruby programmer, this is analogous to rvm.  Virtualenv lets you manage the python versions and package installations separately for different projects or clients.

Installing virtualenv is simple

As always, you're just a single apt-get command away from having virtualenv ready to go:

steve@ubuntu64 ~ $ sudo apt-get install python-virtualenv

That it.  Virtualenv is ready to go now.

Quick example

Say your starting a new project for a client.  They are running python2.5 and want to use the mocks, nose and coverage packages for testing.  Here a walkthrough of how to use virtualenv to manage the project.

First, let's create a directory for the project:

steve@ubuntu64 ~ $ mkdir -p ~/dev/project1
steve@ubuntu64 ~ $ cd ~/dev/project1

Next, run virtualenv to create the environment for the project:


steve@ubuntu64 ~/dev/project1 $ virtualenv -p /usr/bin/python2.5 .env
Running virtualenv with interpreter /usr/bin/python2.5
New python executable in .env/bin/python2.5
Also creating executable in .env/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
steve@ubuntu64 ~/dev/project1 $ 

This command tells virtualenv to create a .env directory and to place a copy of the 2.5 version of python in it.  This copy of the 2.5 python is brand-spankin' new.  It doesn't have any packages (beyond the standard library) installed.  You will need to install them yourself.  Any packages that you install in this instance of python will not be available to main python installation or other virtualenv instances.

Before your can use this new copy, you need to activate it:


steve@ubuntu64 ~/dev/project1 $ source .env/bin/activate
(.env)steve@ubuntu64 ~/dev/project1 $

The activate script manipulates your path environment variable, placing the new python instance first in your path.  This makes is so that when you run python, it will use the version from your instance:


(.env)steve@ubuntu64 ~/dev/project1 $ which python
/home/steve/dev/project1/.env/bin/python
(.env)steve@ubuntu64 ~/dev/project1 $ python
Python 2.5.6 (r256:88840, May  3 2012, 04:16:14) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

Also, notice that your prompt starts with (.env).  This tells you that you're running with the virtualenv instance activated.  To install packages in your instance, use the pip command:


(.env)steve@ubuntu64 ~/dev/project1 $ pip install mock nose coverage
Downloading/unpacking mock
  Downloading mock-0.8.0.tar.gz (749Kb): 749Kb downloaded
  Running setup.py egg_info for package mock
    warning: no files found matching '*.png' under directory 'docs'
    warning: no files found matching '*.css' under directory 'docs'
    warning: no files found matching '*.html' under directory 'docs'
    warning: no files found matching '*.js' under directory 'docs'
Downloading/unpacking nose
  Downloading nose-1.1.2.tar.gz (729Kb): 729Kb downloaded
  In the tar file /tmp/pip-dH_WYa-unpack/nose-1.1.2.tar.gz the member nose-1.1.2/doc/doc_tests/test_selector_plugin/support/tests/mymodule/my_function$py.class is invalid: 'filename None not found'
  In the tar file /tmp/pip-dH_WYa-unpack/nose-1.1.2.tar.gz the member nose-1.1.2/doc/doc_tests/test_restricted_plugin_options/restricted_plugin_options.rst.py3.patch is invalid: 'filename None not found'
  Running setup.py egg_info for package nose
Downloading/unpacking coverage  Downloading coverage-3.5.2.tar.gz (115Kb): 115Kb downloaded
  Running setup.py egg_info for package coverage
    no previously-included directories found matching 'test'Installing collected packages: mock, nose, coverage
  Running setup.py install for mock
    warning: no files found matching '*.png' under directory 'docs'
    warning: no files found matching '*.css' under directory 'docs'
    warning: no files found matching '*.html' under directory 'docs'
    warning: no files found matching '*.js' under directory 'docs'
  Running setup.py install for nose
    Installing nosetests script to /home/steve/dev/project1/.env/bin
    Installing nosetests-2.5 script to /home/steve/dev/project1/.env/bin
  Running setup.py install for coverage
    building 'coverage.tracer' extension
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c coverage/tracer.c -o build/temp.linux-x86_64-2.5/coverage/tracer.o
    coverage/tracer.c:3:20: fatal error: Python.h: No such file or directory
    compilation terminated.
    **
    ** Couldn't install with extension module, trying without it...
    ** SystemExit: error: command 'gcc' failed with exit status 1
    **
    no previously-included directories found matching 'test'
    Installing coverage script to /home/steve/dev/project1/.env/bin
Successfully installed mock nose coverage
Cleaning up...
(.env)steve@ubuntu64 ~/dev/project1 $

To see that the packages have been installed, simply use them:




(.env)steve@ubuntu64 ~/dev/project1 $ python
Python 2.5.6 (r256:88840, May  3 2012, 04:16:14) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mock
>>> import nose
>>> import coverage
>>>

When you're done working on the project, deactivate it.  You can always come back later and activate it again.


(.env)steve@ubuntu64 ~/dev/project1 $ deactivate
steve@ubuntu64 ~/dev/project1 $


Notice that when you deactivate the environment, your packages are no longer available:


steve@ubuntu64 ~/dev/project1 $ python
Python 2.7.3 (default, Apr 20 2012, 22:39:59) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mock
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named mock
>>> import nose
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named nose
>>> import coverage
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named coverage
>>>

You can create as many virtualenv environments as you like.  I create one for each project that I work on.

Bonus material

To make life even easier, here's a couple additional things I do that you might find helpful!

First, once you've started to use virtualenv with some frequency, you start to get tired of downloading and installing the same packages over and over.  Pip has the ability to cache your downloaded packages for reuse.  To do that, you'll need to create a directory to store the download packages in:

steve@ubuntu64 ~ $ mkdir ~/.pip_download_cache

Then you'll need to set a variable to inform pip of the new directory.  Add the following to your .bashrc file:

export PIP_DOWNLOAD_CACHE=/home/steve/.pip_download_cache

Now when you do a pip install, it will keep the downloaded files in the ~/.pip_download_cache directory.  The next time you do a pip install of the same package, it will just use the copy from the directory instead of downloading it again.


Second, it can be tedious to always have to type 'source .env/bin/activate' every time you want to activate an environment.  Since I always put my virtual environments in a .env directory I can count on the command to activate always being the same.  So I create an alias for it.  I added the the following to my ~/.bash_aliases file:

alias activate='source .env/bin/activate'

Now once I cd into the projects directory, I simply type activate to activate my virtual environment.

Fun with gnu screen

In my day job, I spend a fair amount of time working on numerous tasks on the command-line in Linux and OSX.  Way back in my college days, I used a program called gnu screen (aka 'screen') but recently I've "refound" it.  I have a new appreciation for screen and now use it every day.

What is screen?

Screen let's you have multiple shell sessions running simultaneously in the same terminal window.  Here's some of the features that I personally find appealing:

• With a little muscle memory, you can switch rapidly between the sessions.
• It has the ability to split up your terminal window into multiple "window-splits" so you can see two or more shells sessions at the same time.  The splits are even resizable.
• You can 'detach' and 'reattach' to a screen session, allowing you to sign-out, leave and come back later, all the while keeping your terminal sessions active and running.
• It's configurable and scriptable.
• You can copy and paste between sessions with 'vi' like keystrokes.

How do you get screen?

Well, if you're running ubuntu like me, it's pretty darn easy.  You're one command away from having it:

steve@ubuntu64 ~ $ sudo apt-get install screen

Pretty simple.

Getting started

Once you have screen installed, simply run it.

steve@ubuntu64 ~ $ screen

It will bring you to a welcome screen (I'll show you how to disable the welcome screen later).

At the bottom of the welcome screen, it will say

[Press Space or Return to end.]

Go ahead and do that.  You'll be returned to a bash prompt.  It's important to note that screen hasn't terminated, but rather it's spawned another shell prompt within itself and you're sitting at that new prompt. You can now begin using screen via keyboard command sequences.  To prove it, press Ctrl-a and then press " (double quote).  You should get a screen that looks like this:

This is the terminal session list.  If you have more than one terminal session running, you'd seem them all listed here.  Let's create another terminal session.  Press Ctrl-a and then press c.  This should take you back to a bash prompt.  Your now in a new bash prompt.  See see the updated list, press Ctrl-a and " again.

Now there's two bash sessions.  You can select the one you'd like to see by using your arrow keys (or vi cursor movement keys) to highlight the one you want and then press enter/return.

When you want to quit out, simply exit out of all your bash prompts.  Once the last bash session ends, screen will automatically terminate, leaving where you were when you initially started it.

My most used key sequences

As you've probably realized by now, screen uses Ctrl-a plus a key to drive it's functionality.  Here's a list of key sequences that I use most often.

Ctrl-a Ctrl-a - Toggles between two sessions.  It's kind of like the "last" button on your TV remote.

Ctrl-a c - Creates a new terminal session.

Ctrl-a A - Let's you set the title of a terminal session.  The title is what appears in the session list.

Ctrl-a " - Lists the terminal sessions.  You can then use arrow keys or vi keys to select which session you wan to go to.

Ctrl-a d - Detach screen so you can log off and come back later.  Later when you sign back on, just type screen -r to reattach to your running screen session.

Ctrl-a : - Bring up screen's colon prompt (similar to vi).  From the colon prompt you can issue commands directly to screen.  (You'll probably use it most resizing window splits).

Ctrl-a [ - Start copy mode (for copy/paste).

Ctrl-a ] - Paste what you copied from copy mode.

Ctrl-a S - Split the screen horizontally.

Ctrl-a | (pipe) - Split the screen vertically.

Ctrl-a {tab} - Move focus from window split to window split

Ctrl-a X - Close the current window split.

Split example

Here a quick example of doing a couple window splits.  First, start screen and a few terminal sessions:

steve@ubuntu64 ~ $ screen

{return}

Ctrl-a c

Ctrl-a c

Ctrl-a "

Your terminal window should look like this:

Now press Ctrl-a S to split the screen.  You should now have an upper split and a lower split.  Press {return} now to select terminal session 2 in the upper split.  In that terminal session run the top command.

Now press Ctrl-a {tab}.  This will move the cursor down to the bottom split.  Then press Ctrl-a " to bring up the session list.

Select session 0 and then issue the following command in the lower split: watch ls.

Next, let's split the lower window vertically.  Press Ctrl-a | (pipe), Ctrl-a {tab}, Ctrl-a ".  Select session 1.

As you can see, you can keep splitting, sub-splitting, etc.  Remember, Ctrl-a {tab} moves the cursor to the next window split.  That's how you move around.

When you're done, you can use Ctrl-a X to close the window splits.  This does not quit/kill the process running in that split.  The process stays in the session list until you exit it normally.

Scripting

Screen reads and processes the ~/.screenrc file at startup.  Here's mine:

vbell off

startup_message off

defscrollback 1000

bindkey -k k1 colon

bindkey -k k2 screen

bindkey -k F2 windowlist -b

hardstatus off

caption always '%{Yb}[%=%t%=]'

altscreen on

shell -${SHELL}

You can go through the screen man page for what each line does.  You can customize screen however you'd like.

There are two features that combined make startup processing especially interesting:

1. You can use the -c command line argument to give screen an alternate rc file to process on startup.
2. Using commands, you can do anything in the rc file that you could do interactively.

Using this notion, I create different screen rc files for different contexts.  For example, when I'm working at a certain client, I would like to have screen set up a certain way, with specific terminal sessions setup.  I create a bash aliases for each client.

In my ~/.bash_aliases file I have:

alias clientX='screen -c ~/screen_profiles/clientX'

alias clientY='screen -c ~/screen_profiles/clientY'

My ~/screen_profiles/clientX file looks like this:

altscreen on

vbell off

attrcolor b ".I"

termcapinfo xterm 'Co#256:AB=\E[48;5;%dm:AF=\E[38;5;%dm'

defbce "on"

defscrollback 1500

bindkey -k k1 colon

bindkey -k k2 screen

bindkey -k F2 windowlist -b

screen -t "Home" bash

screen -t "Build" ssh my_username@build_server

screen -t "Dev2" ssh my_username@dev_server

screen -t "WebLogic" my_username@weblogic_server

screen -t "Todo/Notes" vi ~/notes/todo.markdown

hardstatus off

caption always '%{Yb}[%=%t%=]'

select 0

split -v

resize +20

focus

select 5

focus

This rc file does the following:

• Does some normal setup that I do anyway (key bindings, colors, etc.)
• Creates a terminal session titled, "Home" and runs bash in it.
• Creates a terminal session titled, "Build" and ssh's me into the build server.
• Creates a terminal session titled, "Dev2" and ssh's me into the dev 2 server.
• Creates a terminal session titled, "WebLogic" and ssh's me into the weblogic server.
• Creates a terminal session titled, "Todo/Notes" and edits my ~/notes/todo.markdown file using vi in it.
• Splits the window vertically, resizing the left window to be 20 columns bigger.
• Selects the "Home" session for the left window.
• Selects the "Todo/Notes" session for the right window.
• Puts the cursor focus on the left window.

So when I come into clientX's office for the day, I simply type clientX at the bash prompt and within a second, I have screen all set up with my splits and automatically logged into all the servers I commonly use.  Pretty cool.