Monday, June 25, 2012

Bash script for pulling/fetching multiple git clones


In my current assignment, I'm acting as the main build guy for a number of projects that use git for source control.  As such, I find it very useful to keep all my git clones up to date whether I'm actively developing in them or not.  Additionally, I need to review the changes other developers are committing, so I'd like to get a summary of recent git activities.

Over time, I've put this little bash script together to help me with that.  I've included the script in this posting so I can remember later what/why I did this.  Disclaimer, I wrote this script and run this script in bash on Linux (not via git-bash in Windows).  Also, I'm using Zenity for some nice UI look/feel.

  1 #!/bin/bash
  2
  3 pushd ~/dev/repos > /dev/null
  4
  5 # The log file
  6 PULL_LOG="$(mktemp)"
  7
  8 # Get a list of all the clones in this directory.
  9 CLONES=$(find -maxdepth 2 -mindepth 2 -type d -name ".git" | sed -e 's|\./||' -e 's|/\.git||')
 10
 11 # Get a list of all the branches in clone/branch format
 12 ALL_BRANCHES=$(for clone in $CLONES; do cd $clone; for branch in $(git branch -l | sed 's/\s\|\*//g'); do echo $clone/$branch; done; cd ..; done)
 13
 14 # Count the branches
 15 BRANCH_COUNT=$(echo $ALL_BRANCHES | sed 's/ /\n/g' | wc -l)
 16
 17 # Start the log file
 18 echo "Pull log for $(date)" >> $PULL_LOG
 19 echo "--------------------------------------------------------------------------------" >> $PULL_LOG
 20
 21 # Function for pipping output to zenity progress dialog
 22 function pull_clones() {
 23     clone_counter=0
 24     for clone in $CLONES; do
 25         echo "Pulling branches for clone $clone" >> $PULL_LOG
 26         echo "--------------------------------------------------------------------------------" >> $PULL_LOG
 27         cd $clone
 28         echo "# Fetching changes for clone $clone"
 29         git fetch origin 2>> $PULL_LOG
 30         for branch in $(git branch -l | sed 's/\s\|\*//g'); do
 31             echo "# Merging branch $clone/$branch"
 32             echo "Merging branch $branch" >> $PULL_LOG
 33             git checkout $branch 2> /dev/null
 34             git merge origin/$branch >> $PULL_LOG
 35             echo | awk '{print count / total * 100}' count=$clone_counter total=$BRANCH_COUNT
 36             let clone_counter=clone_counter+1
 37         done
 38         cd ..
 39         echo >> $PULL_LOG
 40     done
 41 }
 42
 43 # Do it
 44 pull_clones | zenity --progress --title='Pulling development clones' --width=512
 45 zenity --text-info --filename=$PULL_LOG --title="Pull log" --width=500 --height=450
 46
 47 #Clean up
 48 rm $PULL_LOG
 49
 50 popd > /dev/null



Wednesday, June 6, 2012

Patching tip using mocks in python unit tests

I use the mock library by Michael Foord in my python unit tests and one problem always plagued me.  Here's the problem and the solution.

Sometimes when I import a package/module in my code I use this pattern (let's call it pattern A):

"""file_module_pattern_a.py"""
import os

def get_files(path):
    """Return list of files"""
    return os.listdir(path)


Other times, I use this pattern (let's call it pattern B):

"""file_module_pattern_b.py"""
from os import listdir

def get_files(path):
    """Return list of files"""
    return listdir(path_variable)

Note the differente.  In pattern A, I import the whole os package, while in pattern B, I only import the listdir function.  Now in my unit tests, here's what I use for pattern A:

"""Unit tests for module file_module_pattern_a"""

from file_module_pattern_a import get_files
from unittest import TestCase
from mock import patch, sentinel

class StandloneTests(TestCase):
    """Test the standalone functions"""
    
    @patch('os.listdir')
    def test_get_files(self, mock_listdir):
        """Test the get_files function"""
        test_result = get_files(sentinel.PATH)
        mock_listdir.assert_called_once_with(sentinel.PATH)
        self.assertEqual(test_result, mock_listdir.return_value)

This works great.  The only problem is... if I use pattern B with this unit test, the mock_listdir never gets called.  The unit test tries to use the REAL os.listdir function.

Here's the issue at hand.  When I use pattern B, I'm actually adding the function to my module, not the global scope.  As a result, the patch directive needs to reference my module, not os.  Here's the correct unit test patch syntax:

"""Unit tests for module file_module_pattern_b"""

from file_module_pattern_b import get_files
from unittest import TestCase
from mock import patch, sentinel

class StandloneTests(TestCase):
    """Test the standalone functions"""
    
    @patch('file_module_pattern_b.listdir')
    def test_get_files(self, mock_listdir):
        """Test the get_files function"""
        test_result = get_files(sentinel.PATH)
        mock_listdir.assert_called_once_with(sentinel.PATH)
        self.assertEqual(test_result, mock_listdir.return_value)

Monday, June 4, 2012

Using SSHFS from OSX

SSHFS is a FUSE (Filesystem in Userspace) plugin that allows you to mount a drive/filesystem on your systems via SSH.  This is a great way to transfer files securely over the public Internet.  Best of all, it's free!

OSX supports FUSE through a program called OSXFUSE (or Fuse for OSX).  You can download it at:


Note, you'll have to download the plugins for OSXFUSE separately.  The two most popular are:  SSHFS and NTFS-3G (a plugin that allows you to mount NFTS volumes in read/write mode... which I will not be covering in this post).

Computers hosting sshfs directories don't need to have any special software installed on them.  They simply need to be running sshd (the secure shell daemon) and not have file transfers disabled.  Usually, sshd is configured by default to have file transfers enabled.

One thing that often confuses people coming from the Mac world is that SSHFS has no GUI.  So how do you mount remote drives?  Well, you'd normally do it from a terminal prompt.

In this blog entry, I will go over three ways to mount sshfs drives/filesystems using SSHFS:
  1. Using terminal.app and some bash commands
  2. Using Automator
  3. By creating an Alfred.app extension

Get the software

Before we begin, first make sure you have the OSXFUSE software installed.  Download it from the link listed above.  You'll know you have it properly installed when you see it in your System Preferences:

You'll also want to ensure that you've installed the SSHFS plugin from the same site.  To verify that you've installed it correctly, you'll need to run sshfs -h from a terminal window:
If you've got these two items installed, you're ready to go!

First way: From the bash prompt

This method is the most traditional way to mount a sshfs filesystem... and actually the other two methods will be invoking the same commands...we'll just be hiding them with the graphical interface.

First start a terminal window and create a directory to house the filesystem.  I always put my filesystems in the /Volumes directory.  You aren't required to put your directory there.  You can put it anywhere you'd like.  I just do so out of convention.  Here's the command to do so:

steve@l00-1nsv01 $ mkdir /Volumes/ssh_fs_mount

I named my mount ssh_fs_mount.  You can name yours anything you'd like.
Then you need to run the sshfs command to mount the remote computer's directory to your hosting directory. The command takes the following format:

sshfs username@hostname:/path/to/directory /local/directory

...where username is your username on the remote computer, hostname is the name of the remote computer contains the directory that you'd like to mount, /path/to/directory is the full path to the directory on the remote computer that you'd like to mount and /local/directory is the directory on your local computer that you'd like to house the mount on.  Maybe an example will make it more clear.

I have a Linux computer that I do a lot of development on.  It's called ubuntu64.local.  My username on that computer it steve and my development folder on that computer is at /home/steve/dev.  Here's the sshfs command that I would issue to mount my development folder from the Linux computer into the /Volumes/ssh_fs_mount folder on my Mac

sshfs steve@ubuntu64.local:/home/steve/dev /Volumes/ssh_fs_mount

Let's try it real quick:
A couple of items to note...  First, I'm prompted for a password.  This is prompting me for my password on the remote computer (ubuntu64.local in this instance).  It's not prompting me for my Mac password.  This is the remote computer verifying that I'm indeed steve@ubuntu64.local and not some other impostor.    If I have key-based authentication working between my Mac account and my Linux account, I would not be prompted for the password.  I normally do have key-based authentication between these two accounts, but I turned it off for this demonstration.  If you're not using key-based authentication when ssh'ing between computers... you should.  In fact, the other two methods in this blog entry (using Automator and the Alfred extension) assume you are using key-based authentication and aren't prompted for a password. You can find out more about setting-up key-based authentication at any of these links:


Second, there's no other output from the command.  No output means that sshfs was able to successfully complete the mount.  If there had been a problem, sshfs would have complained with error messages.  Once the mount is completed, the drive should appear on the desktop:


It should behave like a normal Mac drive.  You can get info on it.  You can browse it in Finder.  You can even add/edit/remote files (assuming your account on the remote computer has the appropriate privileges).
When you're done and want to unmount the drive, simply right-click on it and select Eject.


Note, when the SSHSFS volume has been ejected, OSX automatically deletes the /Volumes/ssh_fs_mount directory.  If you put your mounting directories in /Volumes, you'll always have to recreate them  after ejecting.

Second Way: Create an Automator task

Using Automator, you can create a workflow to automate the steps you did in the first method.  After all, that's what Automator is for.. automating repetitive things.

First, start Automator and create a new Application workflow:
Next, in the Text Library, drag the Ask For Text action to the workflow.
Check the Ignore this action's input and Require an answer checkboxes.  Enter Enter a remote sshfs url and click OK in the question textbox.
Now, from the Utilities library, drag the Run Shell Script action to the workflow and drop it below the Ask for Text action.
Change the Pass input pulldown from to stdin to as arguments.  Then past the following text into the textarea:

volume_name=/Volumes/sshfs_volume_$$
mkdir $volume_name
/usr/local/bin/sshfs $1 $volume_name
if [ "$?" == "0" ]; then
open $volume_name
else
echo "Unable to mount sshfs volume."
rmdir $volume_name
fi

Save it to your desktop as SSHFS Workflow.  It should now look like this:
Now, if you run it from your desktop, it will prompt you for a sshfs url.  The format is the same from the bash method.  I'll reuse my example:

steve@ubuntu64.local:/home/steve/dev

Again, like I mentioned before, this method assumes that you've already set up key-based authentication between your Mac account and your remote account.
If all goes well, the automator will automatically open the mounted volume.  Just like the first method, when you're done, you simply right-click the volume and eject it via Finder.

Third Way: Create an Alfred Extension

Now if you're running Alfred, you're probably thinking, "I can just run the automator task from Alfred and be done with it."  Yes.  You could.  But you can also create an extension.  That way you can enter the sshfs url directly into the Alfred window and save yourself one last data-entry step.

To create the extension go to the Extensions tab of the Alfred preferences window.  Click the + and select Shell Script.
In the Extension Name field, enter SSHFS and click Create.
In the Title field, enter SSHFS Mounter.  In the Description field, enter Mount a sshfs volume.  Make sure the Keyword checkbox is checked and the enter sshfs in the textbox next to it.  Make sure the Silent checkbox is checked.  Finally in the Command field, enter the following text:

volume_name=/Volumes/sshfs_volume_$$
mkdir $volume_name
/usr/local/bin/sshfs {query} $volume_name
if [ "$?" == "0" ]; then
open $volume_name
else
echo "Unable to mount sshfs volume."
rmdir $volume_name
fi

If should look like this:
If you have Growl installed on your Mac, click the Advanced button and check the Display script output in Growl checkbox.
Save the extension and close the preferences window.  Now from the the Alfred prompt, you can enter sshfs your_sshfs_url and hit enter, where your_sshfs_url is an actual sshfs url.  Here's me mounting my ubuntu64.local system's tmp directory:


Just like the Automator method, the Alfred method presumes you are using key-based authentication between your Mac account and the remote computer's account.  It should automatically open the folder you mounted.  When you're done, simply right-click on the volume and eject it.