Automated python unit testing, code coverage and code quality analysis with Jenkins - part 3

This is the third and final posting in a series.  Check here for the first posting and here for the second posting.

In this posting I will be explaining how to automate the unit testing, code coverage and code quality analysis that we did manually in the first two posts.

As the title of this series suggests, I use Jenkins for this automation (also known as continuous integration - ci).  In keeping with the theme of using apt for everything, I will install Jenkins via apt.  That being said, in my real job, I have downloaded a jenkins.war file from the Jenkins web site and I start it up from the command line.

Install and start Jenkins

Jenkins appears in the default apt that ships with Ubuntu, but it's a really old version... so old, in fact, that it won't run the plugins that we'll need in order to do the code quality and code coverage analysis.  To get around this, refer to these instructions on the Jenkins web site to update your apt sources and get the latest version Jenkins via apt.

Here's a quick screen-shot of the instructions you'll need to follow:

Once you've got Jenkins installed, check to see if Jenkins is running.  If it isn't start it.

sudo /etc/init.d/jenkins status
sudo /etc/init.d/jenkins start

Once Jenkins is up and running, let's pull it up in a browser.  It will be listening on port 8080.

Install some needed Jenkins plugins

Jenkins has lots of plugins that will allow it to do lots of interesting things.  We will be needing a few of them.  To install plugins, click on the Manage Jenkins link from the list of links on the left-hand side of the page. Next click on the Manage Plugins link from the resulting page.  Finally click on the Available tab from the Plugin Manager page.

This will show you a list of all the available plugins that Jenkins can use. Find and select (check the checkbox) for the following plugins:

1. Jenkins Cobertura Plugin
2. Jenkins GIT plugin
3. Jenkins Violations

After you've check the checkboxes, click the Download now and install after restart button at the bottom of the page.

Jenkins will install the selected plugins and then take a moment to restart.  After Jenkins restarts we need to make a small configuration change in order for git to work properly.  When Jenkins is done restarting, click the Manage Jenkins link from the left-hand list of links.  The click Configure System from the resulting page.  This will bring up a global Jenkins configuration page.  Scroll down (past the Git section) to the Git Plugin section.  Enter values in the Global Config user.name Value and Global Config user.email Value fields.  Finally, click the Save button at the bottom of the page.

Our first job

When Jenkins comes back, click the New Job link from the left-hand list of links.  This will bring you to a page where you can enter a job name and select a job type.  Enter Project1 for the job name and select Build a free-style software project, then click the OK button.

This will bring you to the Configure page for your new job.  This is where you tell Jenkins WHAT, WHEN and HOW to do.  Let's get started!

1 - Where to get the source

Under the Source Code Management section, click the Git radio button.  This will expand more options.  For this project we'll be keeping things simple.  Simply enter the path to your git repository.  Mine's at /home/steve/dev/project1 (this presumes your git repo is on the same box that Jenkins is running on).

2 - When to run the job

Next go to the Build Triggers section and check the Poll SCM checkbox.  In the resulting Schedule field, enter 5 *'s.  (* * * * *)  This makes Jenkins check the git repository for changes once a minute, every minute of every day.  If it finds a change in the repository, it will "do a build".  We'll be configuring "the build" to run the unit tests and code coverage and quality tests for us.

BTW, for larger Jenkins implementations (more than 10 or so Jenkins jobs), the Poll SCM option is a terrible idea.  I can show you another way using git hooks if there is interest.  For now, though, let's just use the Poll SCM option since this is just an example project.

3 - What to run

Next, let's configure the job to do what we want it to when it detects a change in the git repository.  In the Build section, click the Add build step button and select Execute shell from the resulting popup.  In the resulting Command field, enter the following text:

PYTHONPATH='' nosetests --with-xunit --all-modules --traverse-namespace --with-coverage --cover-package=project1 --cover-inclusive python -m coverage xml --include=project1* pylint -f parseable -d I0011,R0801 project1 | tee pylint.out

Let's review what's going on here.  When the Jenkins job detects that a source code change has occurred in the git repository, it will clone a copy of the repository and then run the above script against the clone.

You'll notice that the nosetests command has become a little more complex.  I've added a fair number of command-line arguments to it. This is to make sure that the code coverage runs over all the code, not just the code that's involved in the unit tests.  It also ensures that nosetests write's it's output to a file that Jenkins can interpret.

When nosetests runs the code coverage, it generates a .coverage file.  Jenkins can't read that.  The third line of the script (python -m coverage xml...) converts the .coverage file to an xml format that Jenkins Cobertura plugin can read.

The last line (pylint...) runs pylint on the project and outputs it in a format that the Violations Jenkins plugin can read.  I also have it disabling a couple warnings that I don't care to know about.  (You can customize this all you want BTW).

4 - Interpret the results

Not only can Jenkins detect changes and run our tests.  Given the correct plugins are installed (which we did at the beginning), it can interpret the results and display them in charts/trees/etc.  For me this is the best part.  All the other stuff, I could have done with some clever scripting and cron.

First stop is coverage.  In the Post-build Actions section click the Public Cobertura Coverage Report checkbox.  Then in the Cobertura xml report pattern field, enter coverage.xml.

This tells Jenkins to read a file named coverage.xml that will contain testing code coverage information in it.  The line listed below (from the build script in step 3) is what creates this file:

python -m coverage xml --include=project1*

Next, click the Publish JUnit test result report checkbox and enter nosetests.xml in the Test report XMLs field.

This instructs Jenkins to interpret the a file named nosetests.xml that the nose command from creates:

nosetests --with-xunit --all-modules --traverse-namespace --with-coverage --cover-package=project1 --cover-inclusive

Finally, check the Report Violations checkbox and enter **/pylint.out in the pylint field.

This instructs Jenkins to interpret the pylint.out file that is generated by the pylint command from the build script:

pylint -f parseable -d I0011,R0801 project1 | tee pylint.out

At this point, you're done configuring the job.  Click the Save button.  Now you're ready to give it a go!

Run the job

Click the Build Now link on the left-hand list of links.  This will manually kick off a run of the job.  If no errors.  You should see a screen that looks like the one below.  If not refresh your page.

Not very interesting, is it?  That's because we currently have 100% coverage and no test failures!  Let's add some code (without tests) and see what happens.

Update the code

Add the following code to your ~/dev/project1/project1/authentication.py file.  Then save and commit it.


def logout():
    print 'You are now logged out.'

No go back to your Jenkins window and refresh the Project1 page.  You may need to wait a minute.  Remember, Jenkins only checks the git repository for changes once a minute.  You should see some new info.  The top graph is your code coverage.  It indicates that you've dropped from 100% line coverage to 92%.  This makes sense because we added function, but no test for it.

The second graph is your unit test pass/fail trend.  It's all blue because all 3 of our tests are still passing.  We'll break that next by adding a test for logout that fails.  :-)

The third graph is your code quality report.  It's indicating that we had 0 issues in the first build, but now have 1 issue in the second build.

You can get more info by clicking on the graphs.  For example, let's drill into the code coverage graph.

You can browse even deeper.  Click on the project1 link at the bottom.

Then click on the authentication.py link at the bottom.

As you can see by the red line at the bottom, the print statement at the bottom is never exercised by a unit test.  Hmmmm.  We'll need to fix that.

Add a failing test

Add this code to the ~/dev/project1/tests/authentication_tests.py file and then commit it to git:

    def test_logout(self):

        """Test the logout function...badly."""

        self.assertEqual(0, 1)

Wait a minute and then go back to the Jenkins Project1 page.  It should now have a third build reporting the test failure.

That's it.  Hope you found this helpful.  Happy coding.

Automated python unit testing, code coverage and code quality analysis with Jenkins - part 2

This is the second posting in a series.  Check here for the first posting.

Where we gonna do this?

First let's create a directory and repository to house the source code.   Enter the following commands:

mkdir -p ~/dev/project1
git init ~/dev/project1
cd ~/dev/project1

It should go something like this:

Next, I always add a .gitignore file to my repository so unrelated files aren't tracked in source control.  Mine contains the following:

*.swp
*.pyc
tags
.coverage

Once I've created this file, I add it to the git repository with the following commands:

git add .gitignore
git commit -m"Initial commit.  Added a .gitignore file."

It should look like this:

Let's get some source code in place

Before we get too ahead of ourselves, let's pause for some quick explanation.  I don't like to mix my application code with my unit test code.  I keep them separate.  My application code will eventually need to be deployed somewhere, but the unit test code doesn't.  To that end, I put the application code in one python package named after the project and the unit test code in another package called "tests".  So let's create the two packages with the following commands:

mkdir project1
touch project1/__init__.py
mkdir tests
touch tests/__init__.py

This might not be a bad place to do a quick commit to source control:

git add .
git commit -m"Added two blank packages: project1 and tests"

Again, should look like this:

You could run the test runner, coverage and code analysis tools now, but the output wouldn't be very interesting.  Instead, let's move on and add a failing test.

The first test

If you're a TDD type developer, this shouldn't come as a surprise.  We'll write a test first.  Then write some code in order to get the test to pass.  In the tests directory, create a file named "authentication_tests.py" and place the following code in it:


#!/usr/bin/env python
"""Unit tests for the project1.authentcation module."""


from unittest import TestCase
from mock import patch
import project1.authentication as auth


class StandAloneTests(TestCase):
    """Test the stand-alone module functions."""


    @patch('__builtin__.open')
    def test_login(self, mock_open):
        """Test the login function."""
        mock_open.return_value.read.return_value = \
            "george|bosco\n"
        self.assertTrue(auth.login('george', 'bosco'))

This is a test the will test a function named login that will reside project1.authentication module.  If you run it now, it should fail because we haven't created an authentication module yet, much less a login function.  To run the tests, simply type "nosetests" from the project root directory.  Nose will scan your whole directory structure looking for unit tests to run.

Let's get the test to pass

Before we can move on to the topics of coverage and quality, we need to get the test to pass.  To do this,  create a file in the project1 directory named "authentication.py" and place the following code in it:

#!/usr/bin/env python

"""This module provides functions for authenticating users."""

def login(username, password):

    try:

        user_file = open('/etc/users.txt')    

        user_buf = user_file.read()

        users = [line.split("|") for line in user_buf.split("\n")]

        if [username, password] in users:

            return True

        else:

            return False

    except Exception, exc:

        print "I can't authenticate you."

        return False

Now when you run the test, everything should be hunky-dory.

Code Coverage

You're probably thinking things look pretty good right about now.  Let's run the the code coverage tool to see where we stand.  To run a coverage test, use this command: "nosetests --with-coverage --cover-package=project1".  You'll find that we only have 67% coverage and that our test doesn't exercise lines 12-15 of our login method. Yikes.

You see...  In our single test, we're only covering the happy path(tm).  That is, we only check to see what the function does when a user enters a valid username/password.  We don't check what happens when they enter an invalid username/password.  Additionally, we don't check to see what the function does when it can't read the user file at all.  We would need to add more tests for that.  Edit the tests/authentication_tests.py file and make it look like this:

#!/usr/bin/env python

"""Unit tests for the project1.authentication module."""

from unittest import TestCase

from mock import patch

import project1.authentication as auth

class StandAloneTests(TestCase):

    """Test the stand-alone module functions."""

    @patch('__builtin__.open')

    def test_login_success(self, mock_open):

        """Test the login function when things go right."""

        mock_open.return_value.read.return_value = \

            "george|bosco"

        self.assertTrue(auth.login('george', 'bosco'))

    @patch('__builtin__.open')

    def test_login_bad_creds(self, mock_open):

        """Test the login function when bad creds are passed."""

        mock_open.return_value.read.return_value = \

            "george|bosco"

        self.assertFalse(auth.login('george', 'marbleRye'))

    @patch('__builtin__.open')

    def test_login_error(self, mock_open):

        """Test the login function when an error happens."""

        mock_open.side_effect = IOError()

        self.assertFalse(auth.login('george', 'bosco'))

Notice I've added two more tests:  one to test invalid creds and one to test an exception occurring.  Now we have 100% coverage.

Code Quality

So we now have tests that pass and 100% coverage.  That's awesome!  Everything's under control now.  What did you say?  "How about the quality of that code?"  Let's check it out and see.  To take a look at code quality use the following command: "pylint -r n project1".  Ah. Well.  We have a few things to clean up.  Pylint is notoriously whiny.  That said, the things it's complaining about in this case are legitimate if you're concerned about being a good python developer.

So let's get ourselves into compliance.  First let's add a docstring to the project1 package.  Edit project1/__init__.py and make it look like this:

#!/usr/bin/env python

"""

Project1 is Steve's example project for the blog.

It contains sample code here and there.

"""

That takes care of the first pylint complaint.  Next let's fix up the project1/authentication.py file.  Edit it and make it look like this:

#!/usr/bin/env python

"""This module provides functions for authenticating users."""

def login(username, password):

    """Log the user in."""

    try:

        user_file = open('/etc/users.txt')    

        user_buf = user_file.read()

        users = [line.split("|") for line in user_buf.split("\n")]

        return [username, password] in users

    except IOError:

        print "I can't authentication you."

        return False

Now when you run pylint, you should have no complaints:

If you want a more detailed report from pylint, try using the command, "pylint -r y project1" instead.  Used that way, it's very much more verbose.

Let's clean up

So here's what we've accomplished so far.  We created a project and source code repository.  We've created a sample piece of code and some tests for it.  We've ensured that we have 100% code coverage and we've analysed our code quality and found it to be in good shape.  Let's pause and make sure that all our work is committed to source control.

That's it for now.  In the next post, I will walk through installing Jenkins and configuring it to monitor your source code repository.  It will notice when you commit new code and automatically run your unit tests, code coverage and code quality analysis.  It can be configured to notify you when things don't look so good.  It can also display nice graphs over time of how your code is progressing.  Stay tuned.

Automated python unit testing, code coverage and code quality analysis with Jenkins - part 1

As a software development consultant, I do a fair amount of python development and even though python is a scripting language, I feel that it's still worthy of good coding practices, namely: unit testing, code coverage analysis and code quality analysis.  In this series of blog postings I will describe my python development environment and how I engage the tools.

In this specific blog post, I will talk about the tools I use and how to get them installed/configured.  In future postings I'll set up a sample code base with sample unit tests and run the tests manually.  Finally in the last posting, I'll talk about using Jenkins to automate the testing.

Let's get started
I tend to do most of my python development on the Linux platform.  I love python's cross platform features.  While most of my client implementations actually run in Microsoft Windows, I still do my development in Linux.

For the purposes of this blog series, I will be using Ubuntu 11.10, the 64 bit version.  This distribution/version isn't necessary, but that what I'm using.  Python 2.7 comes with the default installation.  If for whatever reason python isn't installed on your system, just type "sudo apt-get install python" from the bash prompt:

Source control

When we move on to automating the testing/analysis, you'll need to have some source control in place.  I use git.  Even if you aren't planning on doing the automation part, source control is still a good idea.  Again, you can get it with a single command: "sudo apt-get install git"

Tools of the trade
There are a number of testing, quality and coverage tools out there for python.  These are the ones I use:

Unit testing:  unittest - Comes with python.  It's part of the standard library.
Mocking:  mock by Michael Foord - It's available in apt.
Test runner: nose - Also available in apt.
Code coverage: coverage - Also, also available in apt.
Code quality: pylint -  Also, also, also available in apt.

So to get all these great tools you simply need to type one command: "sudo apt-get install python-mock python-nose python-coverage pylint"

That's it.  If you're following along, you should have a fully ready python development environment.  Tune in to the next posting where I'll create a repository, some sample code and sample unit tests.  Then I'll manually run the testing, coverage and quality analysis tools against it.

Copy and paste from vim

I've added the following to my .vimrc:

map <Leader>c :w !xclip -sel clip<cr><cr> map <Leader>p :r !xclip -o -sel clip<cr>

It allows me to highlight text in a file and then hit '\c' to copy the highlighted text to my X11 clipboard in Ubuntu.

It also allows me to just hit '\p' to paste from the X11 clipboard to the vim buffer.

My .vimrc in OSX has the following to provide the same functionality when I'm on the Mac:

map <Leader>c :w !pbcopy<cr><cr> map <Leader>p :r !pbpaste<cr>

Indent/unindent tip in vim

In vim, I do a fair amount of code refactoring.  As a result, I often find myself indenting and unindenting blocks of code (sometimes I'm cleaning up my own code, but often I'm cleaning code that has passed through many hands).

As you may already know, you can use shift-v  and then j and k to highlight lines of code in vim.  You may also already know that you can use > and < to indent and unindent a highlighted chunk of text by one tab.

The thing that always bothered me is that after I use >, the code is automatically un-highlighted.  So if I want to indent a chunk of code multiple times I have three options:

Solution 1:  Highlight the code with shift-v.  Then guess how many times I want to indent and then prefix my > with that number:  shift-v, j, j, j, 3, >
Problem 1:  I never guess right.  I'm always off by 1.

Solution 2:  Highlight the code with shift-v.  Indent once with >.  Then use gv to re-highlight the previous chunk. Then use > again.  Repeat until I have the code where I want it: shift-v, j, j, j, >, gv, >, gv, >
Problem 2: I never remember to use gv.  It's just not in my muscle memory.  I start re-highlighting, which clears the gv register... THEN I remember about gv.  Arrrg.

Solution 3:  Just give up and manually re-highlight with shift-v after doing a single indent: shift-v, j, j, j,  >, k, k, k, shift-v, j, j, j,  >, k, k, k, shift-v, j, j, j,  >
Problem 3:  This is really inefficient adding tons of keystrokes.

Finally, I came up with this solution.  I added the following lines to my .vimrc:

map > >gv map < <gv

This remaps > to do a > and then do a gv right away (and the same for <).  This gives me the benefit of solution 2 from above without me having to remember to use gv.  Doing this, I could accomplish the same task as above with a simple:

shift-v, j, j, j, >, >, >

The only catch is that you have to manually turn off highlighting when you're done with either another shift-v or an escape press.

Simple environment toggle plugin for vim

I use vim a lot for work.  I use it for a number of different things.  I've found that in certain circumstances, I need some settings in place and in others situations, I need different settings.  For example, when I'm editing a python script, I need some specific settings in order to keep pylint happy.

Below is a file that I've dropped in my ~/.vim/plugins directory.  I named it pyvi.vim.  It let's me toggle between "normal" settings and "python" settings:

function! PyVi()
if exists('g:pyvi_on')
unlet g:pyvi_on
set noexpandtab
set textwidth=78
set tabstop=8
set softtabstop=0
set shiftwidth=8
set noautoindent
unmap <Leader>r
echo "Pyvi off."
else
let g:pyvi_on = 1
set expandtab
set textwidth=79
set tabstop=8
set softtabstop=4
set shiftwidth=4
set autoindent
map <Leader>r :!nosetests<cr>
echo "Pyvi on."
endif
endfunction
command Pyvi call PyVi()

Now, when I'm in vim, if I issue the command "Pyvi" once, it set's a bunch of settings for me that are specific to making pylint happy.  If i issue the same command again, it will undo my settings changes.  I also added the following to my .vimrc to create a mapping for \-d.

map <Leader>d :Pyvi<cr>

Pretty simple and yet effective.

An Alfred extension to run Linux apps in OSX

I run a Linux Virtual on my Mac via Parallels Desktop.  In a prior post, I showed how to run X11 apps on the Linux virtual, but directing the output to OSX via the ssh's X11 tunneling.  Keep in mind, the post isn't Parallels specific.  Parallels is just what I use for running virtuals.

In this post, I'll show how to create an Alfred extension to run Linux X11 apps directly from an Alfred prompt.

Things you'll need:

• You will need to have gone through my prior post and ensure you have ssh X11 tunneling working correctly.
• You will need to have key based authentication working with your Linux guest so that you are NOT prompted for any password when ssh'ing into it.
• You will need to have Alfred installed (with the Powerpack).

Now let's get started.

First, pull up your Alfred preferences and go to the Extensions tab.  Click the "+" in the bottom left-hand corner and select "Shell Script".  Fill out the resulting sheet naming the extension whatever you'd like and click "Create".

This will bring you into the extension editor.  You want to keep the Keyword checkbox checked.  In the textbox next to that, enter a keyword.  I use 'lr' for 'Linux Run'.  Also, keep the Silent checkbox checked and the Action checkbox unchecked.

In the command text area, enter the following:

ssh -Y -C linux "{query}"

Be sure to replace "linux" with whatever you've named your Linux guest in the Mac's /etc/hosts file.  Also, if you're logging into the Linux guest with a user that is different than your OSX username you will want to change the ssh command format to:

ssh -Y -C johndoe@linux "{query}"

The -Y turns on ssh X11 tunneling.  The -C turns on compression.  It's not needed, but might speed things up depending on what apps you're running.  The "{query}" tells Alfred to send whatever you enter after the "lr" to ssh for processing on the Linux virtual.  Here's what mine looks like:

Be sure to click the "Save" button then close the window.

Now you're ready to give it a try.  Bring up the Alfred prompt (alt/option->space) and type "lr xlogo" and press enter/return.  You should get the XWindow logo app up on your screen.  You will need to be sure that your Linux guest is up and running when you do this.

If nothing comes up, go back into the extension editor and uncheck the Silent checkbox.  (Be sure to save).  Then try it again.  This time it will bring up a terminal window and enter your ssh command in it.  You'll be able to see any error message that results.  This should work with any X11 app that is in the Linux path.  I use it to bring up Eclipse quickly when I need to run Eclipse from Linux and not on my Mac;

lr eclipse

Also, sadly, for whatever reason, Firefox runs faster on my Linux virtual than it does natively in OSX.  So sometimes, I run firefox this way as well:

lr firefox

Coherence-like Linux Integration with Parallels Desktop (or any other VM system)

Here's a nifty trick you can use to integrate your Linux virtual machine with OSX's windowing system:

First ensure that you've installed ssh on your Linux virtual and that you have sshd running.  On Ubuntu, it's as easy as typing this command from a terminal window:

sudo apt-get install ssh

Once you have ssh installed, you will want to figure out the IP address of your Linux guest virtual.  I've configured my Linux instance's network to use NAT.  If you're running Ubuntu, you can simply type this command from a terminal window:

ifconfig eth0 | grep "inet addr" | awk '{print $2}'

I've found that in Parallels, even though my virtual guest is configured to get it's IP address from a DHCP server, it always get's the same IP address when I run it in NAT mode.  You may want to consider changing your Linux guest configuration from DHCP to static if you find that your IP address is always changing.

Once you have your guest computer's IP address in hand, you will want to add it to the /etc/hosts file on your Mac.  You will need to have admin privileges when editing it.  I edit mine using vim from a OSX terminal window.

sudo vi /etc/hosts

You will want to add an entry at the bottom of your /etc/hosts file for your Linux guest.  Name it whatever you'd like.

 Once you've done that, you should be able to ssh into your Linux guest from an OSX terminal window using the hostname you provided in the hosts file:

ssh steve@linux

If this is working, you're in good shape.  You may want to consider using key-based authentication so that you're not constantly being prompted for a password.

Now we're getting to the interesting part.  You can use ssh's X11 tunneling capability to run X11 apps on your Linux guest, but have the output directed to your Mac using either X11.app or XQuartz.app.  X11.app comes with OSX Lion and below.  If you're considering Mountain Lion, you'll need to download XQuartz.app and install it yourself.

To use the X11 tunneling, you simply need to ssh into your Linux guest with the "-Y" option:

ssh -Y steve@linux

Once you're logged into the Linux guest, ensure that your DISPLAY environment variable is set to:  localhost:10.0.

echo $DISPLAY

If it is, you're all set.  Just try to run an X11 app from your Linux bash prompt:

xeyes

libreoffice --calc

firefox

sudo synaptic 

Your Mac should pop-up a new window with the app you specified:

Pretty cool huh?