Friday, April 27, 2012

Automated python unit testing, code coverage and code quality analysis with Jenkins - part 3

This is the third and final posting in a series.  Check here for the first posting and here for the second posting.

In this posting I will be explaining how to automate the unit testing, code coverage and code quality analysis that we did manually in the first two posts.

As the title of this series suggests, I use Jenkins for this automation (also known as continuous integration - ci).  In keeping with the theme of using apt for everything, I will install Jenkins via apt.  That being said, in my real job, I have downloaded a jenkins.war file from the Jenkins web site and I start it up from the command line.

Install and start Jenkins

Jenkins appears in the default apt that ships with Ubuntu, but it's a really old version... so old, in fact, that it won't run the plugins that we'll need in order to do the code quality and code coverage analysis.  To get around this, refer to these instructions on the Jenkins web site to update your apt sources and get the latest version Jenkins via apt.

Here's a quick screen-shot of the instructions you'll need to follow:

Once you've got Jenkins installed, check to see if Jenkins is running.  If it isn't start it.

sudo /etc/init.d/jenkins status
sudo /etc/init.d/jenkins start

Once Jenkins is up and running, let's pull it up in a browser.  It will be listening on port 8080.

Install some needed Jenkins plugins

Jenkins has lots of plugins that will allow it to do lots of interesting things.  We will be needing a few of them.  To install plugins, click on the Manage Jenkins link from the list of links on the left-hand side of the page. Next click on the Manage Plugins link from the resulting page.  Finally click on the Available tab from the Plugin Manager page.

This will show you a list of all the available plugins that Jenkins can use. Find and select (check the checkbox) for the following plugins:
  1. Jenkins Cobertura Plugin
  2. Jenkins GIT plugin
  3. Jenkins Violations
After you've check the checkboxes, click the Download now and install after restart button at the bottom of the page.

Jenkins will install the selected plugins and then take a moment to restart.  After Jenkins restarts we need to make a small configuration change in order for git to work properly.  When Jenkins is done restarting, click the Manage Jenkins link from the left-hand list of links.  The click Configure System from the resulting page.  This will bring up a global Jenkins configuration page.  Scroll down (past the Git section) to the Git Plugin section.  Enter values in the Global Config Value and Global Config Value fields.  Finally, click the Save button at the bottom of the page.

Our first job

When Jenkins comes back, click the New Job link from the left-hand list of links.  This will bring you to a page where you can enter a job name and select a job type.  Enter Project1 for the job name and select Build a free-style software project, then click the OK button.

This will bring you to the Configure page for your new job.  This is where you tell Jenkins WHAT, WHEN and HOW to do.  Let's get started!

1 - Where to get the source

Under the Source Code Management section, click the Git radio button.  This will expand more options.  For this project we'll be keeping things simple.  Simply enter the path to your git repository.  Mine's at /home/steve/dev/project1 (this presumes your git repo is on the same box that Jenkins is running on).

2 - When to run the job

Next go to the Build Triggers section and check the Poll SCM checkbox.  In the resulting Schedule field, enter 5 *'s.  (* * * * *)  This makes Jenkins check the git repository for changes once a minute, every minute of every day.  If it finds a change in the repository, it will "do a build".  We'll be configuring "the build" to run the unit tests and code coverage and quality tests for us.

BTW, for larger Jenkins implementations (more than 10 or so Jenkins jobs), the Poll SCM option is a terrible idea.  I can show you another way using git hooks if there is interest.  For now, though, let's just use the Poll SCM option since this is just an example project.

3 - What to run

Next, let's configure the job to do what we want it to when it detects a change in the git repository.  In the Build section, click the Add build step button and select Execute shell from the resulting popup.  In the resulting Command field, enter the following text:

Let's review what's going on here.  When the Jenkins job detects that a source code change has occurred in the git repository, it will clone a copy of the repository and then run the above script against the clone.

You'll notice that the nosetests command has become a little more complex.  I've added a fair number of command-line arguments to it. This is to make sure that the code coverage runs over all the code, not just the code that's involved in the unit tests.  It also ensures that nosetests write's it's output to a file that Jenkins can interpret.

When nosetests runs the code coverage, it generates a .coverage file.  Jenkins can't read that.  The third line of the script (python -m coverage xml...) converts the .coverage file to an xml format that Jenkins Cobertura plugin can read.

The last line (pylint...) runs pylint on the project and outputs it in a format that the Violations Jenkins plugin can read.  I also have it disabling a couple warnings that I don't care to know about.  (You can customize this all you want BTW).

4 - Interpret the results

Not only can Jenkins detect changes and run our tests.  Given the correct plugins are installed (which we did at the beginning), it can interpret the results and display them in charts/trees/etc.  For me this is the best part.  All the other stuff, I could have done with some clever scripting and cron.

First stop is coverage.  In the Post-build Actions section click the Public Cobertura Coverage Report checkbox.  Then in the Cobertura xml report pattern field, enter coverage.xml.

This tells Jenkins to read a file named coverage.xml that will contain testing code coverage information in it.  The line listed below (from the build script in step 3) is what creates this file:

python -m coverage xml --include=project1*

Next, click the Publish JUnit test result report checkbox and enter nosetests.xml in the Test report XMLs field.

This instructs Jenkins to interpret the a file named nosetests.xml that the nose command from creates:

nosetests --with-xunit --all-modules --traverse-namespace --with-coverage --cover-package=project1 --cover-inclusive

Finally, check the Report Violations checkbox and enter **/pylint.out in the pylint field.

This instructs Jenkins to interpret the pylint.out file that is generated by the pylint command from the build script:

pylint -f parseable -d I0011,R0801 project1 | tee pylint.out

At this point, you're done configuring the job.  Click the Save button.  Now you're ready to give it a go!

Run the job

Click the Build Now link on the left-hand list of links.  This will manually kick off a run of the job.  If no errors.  You should see a screen that looks like the one below.  If not refresh your page.
Not very interesting, is it?  That's because we currently have 100% coverage and no test failures!  Let's add some code (without tests) and see what happens.

Update the code

Add the following code to your ~/dev/project1/project1/ file.  Then save and commit it.

def logout():
    print 'You are now logged out.'

No go back to your Jenkins window and refresh the Project1 page.  You may need to wait a minute.  Remember, Jenkins only checks the git repository for changes once a minute.  You should see some new info.  The top graph is your code coverage.  It indicates that you've dropped from 100% line coverage to 92%.  This makes sense because we added function, but no test for it.

The second graph is your unit test pass/fail trend.  It's all blue because all 3 of our tests are still passing.  We'll break that next by adding a test for logout that fails.  :-)

The third graph is your code quality report.  It's indicating that we had 0 issues in the first build, but now have 1 issue in the second build.
You can get more info by clicking on the graphs.  For example, let's drill into the code coverage graph.
You can browse even deeper.  Click on the project1 link at the bottom.
Then click on the link at the bottom.
As you can see by the red line at the bottom, the print statement at the bottom is never exercised by a unit test.  Hmmmm.  We'll need to fix that.

Add a failing test

Add this code to the ~/dev/project1/tests/ file and then commit it to git:

    def test_logout(self):
        """Test the logout function...badly."""
        self.assertEqual(0, 1)

Wait a minute and then go back to the Jenkins Project1 page.  It should now have a third build reporting the test failure.
That's it.  Hope you found this helpful.  Happy coding.


  1. I'm getting started with automated testing. Found this very useful. Thanks!

  2. This is very useful article. Good Job.
    Thanks a lot.

  3. Thanks a lot! Unfortunately it seem that the cobertura plugin cannot parse my coverage.xml:
    [Cobertura] Publishing Cobertura coverage report...
    ERROR: Publisher hudson.plugins.cobertura.CoberturaPublisher aborted due to exception
    java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Boolean
    at hudson.plugins.cobertura.CoberturaPublisher$ParseReportCallable.invoke(
    at hudson.plugins.cobertura.CoberturaPublisher$ParseReportCallable.invoke(
    at hudson.FilePath.act(
    at hudson.FilePath.act(
    at hudson.plugins.cobertura.CoberturaPublisher.perform(
    at hudson.tasks.BuildStepMonitor$3.perform(
    at hudson.model.AbstractBuild$AbstractRunner.perform(
    at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(
    at hudson.model.Build$RunnerImpl.post2(
    at hudson.model.AbstractBuild$
    at hudson.model.ResourceController.execute(

    Any ideas?

    1. Honestly, I'd have to see your coverage.xml file to know.

  4. Now, I can see what is bad part of my coding and explore to improve it.
    Thank you for the great tutorial! Good Job! :-)

  5. This comment has been removed by the author.

  6. I Jenkins installed on a remote server(not sure what OS it is)
    and python test file which have many test cases( i.e def() )

    When i run the Jenkins job .It gives output on console where certain test pass and other fails but at the end of the console output it shows:

    Test complete. Result code = 0
    STS: smoke_1
    Unlock Code: 2014021009141392041655
    Archiving artifacts
    Email was triggered for: Success
    Sending email for trigger: Success
    Sending email to: xxx
    Notifying upstream projects of job completion
    Finished: SUCCESS

    Why is the Result Code = 0 above when some test pass and others fail in the python file,
    how can i correct the result code to non zero value?

    Is it some error with Jenkins or Python code?

    1. This is my code:
      print 'result code before subprocess: '
      print result_code
      result_code =
      print 'result code after subporcess: '
      print result_code
      Before result_code is -1(default)
      after result_code is 0

      where command = 'jenkins/'
      where calls file from inside it !

      Will ) return zero if all def() inside are executed successfully
      or will it return zero if all def() inside return passed ?

      In my case some def() passed and some failed but still returns zero !
      Is is right ?

      What if i want return non zero if some def() failed inside ?

  7. Hi bhfsteve!
    I found this article very helpful although I have a question.
    I dont know how it worked in April 2012 but now if you run nosetests and one on tests fails then whole nosetests returns 1. That terminates "Execute shell" in jenkins and set build as a FAILURE.
    We can do something like:
    nosetests .... || echo "Failures".

    Do you have any better way to continue "Execute shell" after return code 1?

  8. Really useful article. Thanks!

  9. Very helpful and easy to follow article. Thanks for posting it!

  10. Hello,
    first of all i ould like to thank you for this useful tutorial.
    And i have a question please,
    I did all the steps but i can not display the cobertura report !
    The error message is "No report data"!

    Tahnk you fo answer

  11. Good tutorial:-). Does cobertura works effectively for measuring the python code coverage effectively in all cases?

  12. You saved my life several times with that article. Thank you very much :)

  13. Steve you are the man
    Thanks so much!!

  14. Very useful, thanks for the knowledge sharing

  15. Very nice sets of tutorials. Thank you.

  16. Great job!
    Thanks for the tutorial!

  17. Why did you choose nosetests over py.test & unitest? Should it work with them as well?

  18. This comment has been removed by the author.

  19. автоизсшатоп-4-лучших-авто-аукционов-сша/

    Многиеотечественныеавтолюбителивыбираютдляпокупкиновоймашиныавтоаукционы США. Bот Топ-4 лучшихавто-аукционов США

  20. Enjoyed reading the article above, really explains everything in detail, the article is very interesting and effective. Thank you and good luck for the upcoming articles learn python training in Bangalore