tag:blogger.com,1999:blog-43124814433495221502024-03-14T01:32:41.100-04:00Steve's BlogTechnical tipsbhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comBlogger29125tag:blogger.com,1999:blog-4312481443349522150.post-27632141584332714032017-02-23T10:49:00.000-05:002017-02-24T15:23:30.487-05:00Python streamsI've become really comfortable with Java 8's streams over the past couple years. As a result, when I go back and do Python work, I get stream envy. I looked around and found a couple options, but none of them were really what I'm after... something quick and dirty that will just make my syntax a bit more readable.<br />
<br />
So I threw this together and find it quite handy. I hope you find it helpful as well. If you save this to a module and name it 'streams.py', you can use it like the following:<br />
<br />
<span style="color: blue; font-family: "courier new" , "courier" , monospace;"> from lazy_streams import stream</span><br />
<span style="color: blue; font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="color: blue; font-family: "courier new" , "courier" , monospace;"> S = stream(range(250)) \</span><br />
<span style="color: blue; font-family: "courier new" , "courier" , monospace;"> .filter(lambda x: (x%2 == 0))</span><br />
<span style="color: blue; font-family: "courier new" , "courier" , monospace;"> print S.size()</span><br />
<span style="color: blue; font-family: "courier new" , "courier" , monospace;"> print S.take(10).to_string()</span><br />
<span style="color: blue; font-family: "courier new" , "courier" , monospace;"> print S.reverse().take(10).to_list()</span><br />
<span style="color: blue; font-family: "courier new" , "courier" , monospace;"> S1 = S.map(lambda x: "Item %d" % x)</span><br />
<span style="color: blue; font-family: "courier new" , "courier" , monospace;"> print S1.first_or_else('Nothing to see')</span><br />
<span style="color: blue; font-family: "courier new" , "courier" , monospace;"> print stream(['Patty Cake', 'Jim Shoe', 'Justin Case']) \</span><br />
<span style="color: blue; font-family: "courier new" , "courier" , monospace;"> .sort(lambda x: x.split(' ')[1]) \</span><br />
<span style="color: blue; font-family: "courier new" , "courier" , monospace;"> .to_list()</span><br />
<span style="color: blue; font-family: "courier new" , "courier" , monospace;"> print stream([[1, 2], 3, [4, 5, 6], 'seven']) \</span><br />
<span style="color: blue; font-family: "courier new" , "courier" , monospace;"> .flatten() \</span><br />
<span style="color: blue; font-family: "courier new" , "courier" , monospace;"> .to_list()</span><br />
<br />
I've tested it on lists as big as 2.5 million items and the lazy evaluation seems to work really great!<br />
<br />
Here's the gist:<br />
<br />
<script src="https://gist.github.com/brettschneider/4b91533e3a43f6b724fd8348eaf5d2ae.js"></script>bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-2691211057012805902017-02-17T16:06:00.001-05:002017-02-17T16:07:56.942-05:00Sharing a Python generator across multiple multiprocessing processes.Sometimes I need to do some work on a seemingly endless set of data. I often use generators to create the endless data. Say, for example I were trying to brute-force crack a zip file's password. I'd create a generator that methodically and continually creates new passwords for the cracking to to use in attempting to unzip the file.<br />
<br />
Once I have my generator in place, I have a problem. What if I want to spread this brute-force attack across all the cores in my system to speed up this slow endless effort. One would think that multiprocessing will solve the problem.<br />
<br />
Well, no. Generators aren't natively shared across processes in Python. They're copied to each process. This means that if you try to use the generator with multiprocessing, each process will get it's own copy of the generator and each process would get the same values from each copy.<br />
<br />
To solve that, I devised a quick and dirty solution: Place the generator in it's own process and then have each worker process request the next value from it via inter-process communication using the multi-processing Pipe class.<br />
<br />
To that end, here's an example. I post this here mainly to jog my memory next time I need to do this, but if you find it useful, great!<br />
<br />
<br />
<script src="https://gist.github.com/brettschneider/ff173aad40f5eddca8fc1e56fdd66938.js"></script>bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-13709249098829215992017-02-08T13:50:00.001-05:002017-02-08T13:50:59.149-05:00Non-freeze, single file distribution of your python projectIn my job, I often have to share/distribute Python code to others on my development team or to others in sibling development teams. These folks are technical and I can easily rely on them to have a reasonably contemporary version of Python already installed. Thus I don't really want to completely freeze my Python project (via cxFreeze, py2app, py2exe, etc.). Nor do I want to send my coworkers through the 'create a virtualenv, pip install -r, etc.).<div>
<br /></div>
<div>
I want a middle-ground. I want to just send them a file, knowing that it contains all the required dependencies and will just run, assuming they have Python in their path.</div>
<div>
<br /></div>
<div>
I've known for a while that if you zip up a directory full of Python files and so long as the root of the zip file contains a __main__.py file and you can directly run the zip file via <span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>python {filename.zip}</b></span>.</div>
<div>
<br /></div>
<div>
What I didn't know is that you can concatenate a text file containing the Python shebang ('#!/usr/bin/env python') and a python zip file and the resulting file is runnable.</div>
<div>
<br /></div>
<div>
<div style="background-color: black; color: #5330e1; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">#!/usr/bin/env python</span></div>
<div style="background-color: black; color: #5330e1; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">PK</span><span style="font-variant-ligatures: no-common-ligatures;">^C^D^T^@^B^@^H^@</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">nLHJb±gL</span><span style="font-variant-ligatures: no-common-ligatures;"><94>^A^@^@</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">Ò</span><span style="font-variant-ligatures: no-common-ligatures;">^B^@^@^K^@^\^@</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">__main__.pyUT </span><span style="font-variant-ligatures: no-common-ligatures;">^@^C</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">¯,</span><span style="font-variant-ligatures: no-common-ligatures;"><9b></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">Xö,</span><span style="font-variant-ligatures: no-common-ligatures;"><9b></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">Xux</span><span style="font-variant-ligatures: no-common-ligatures;">^K^@^A^D</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">«</span><span style="font-variant-ligatures: no-common-ligatures;"><82><90></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">~</span><span style="font-variant-ligatures: no-common-ligatures;">^D^T^@^@^@</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">u</span><span style="font-variant-ligatures: no-common-ligatures;"><92></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">ÑkÛ0</span><span style="font-variant-ligatures: no-common-ligatures;">^P</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">ÆßýWܲ</span><span style="font-variant-ligatures: no-common-ligatures;">^G</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">ÙÐ$},</span><span style="font-variant-ligatures: no-common-ligatures;">^E</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">ÃÂ(</span><span style="font-variant-ligatures: no-common-ligatures;"><85></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">2Ú</span><span style="font-variant-ligatures: no-common-ligatures;"><90></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">ìq </span><span style="font-variant-ligatures: no-common-ligatures;">^T</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">ûâ</span><span style="font-variant-ligatures: no-common-ligatures;"><88></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">Z</span><span style="font-variant-ligatures: no-common-ligatures;"><92></span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">9</span><span style="font-variant-ligatures: no-common-ligatures;"><9d></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">º</span><span style="font-variant-ligatures: no-common-ligatures;"><95></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">±ÿ}</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">';ÎJ`Â</span><span style="font-variant-ligatures: no-common-ligatures;">^O<96></span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">ôéû~ºÓ</span></div>
<div style="background-color: black; color: #5330e1; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">ÈY©1</span><span style="font-variant-ligatures: no-common-ligatures;">^F<93></span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">9£4@</span><span style="font-variant-ligatures: no-common-ligatures;">^Q</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">&</span><span style="font-variant-ligatures: no-common-ligatures;"><81></span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">+ü</span><span style="font-variant-ligatures: no-common-ligatures;"><8e></span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">O</span><span style="font-variant-ligatures: no-common-ligatures;">^X<88></span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">2#tCrÌ</span><span style="font-variant-ligatures: no-common-ligatures;">^N</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">0©¸</span><span style="font-variant-ligatures: no-common-ligatures;"><83></span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">ÛÑw°ÌÕÂ#®Ë¾</span><span style="font-variant-ligatures: no-common-ligatures;">^A</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">aÀ</span><span style="font-variant-ligatures: no-common-ligatures;">^P</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">Å >«¼h@</span><span style="font-variant-ligatures: no-common-ligatures;">^P</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">;c6</span><span style="font-variant-ligatures: no-common-ligatures;"><8d></span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">"</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">^Âý=</span><span style="font-variant-ligatures: no-common-ligatures;">^\</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">AB!ej</span><span style="font-variant-ligatures: no-common-ligatures;"><91></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="font-variant-ligatures: no-common-ligatures;"><9e></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">g¦ñ</span><span style="font-variant-ligatures: no-common-ligatures;"><95></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">Vgé</span><span style="font-variant-ligatures: no-common-ligatures;"><87></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">ÌqÏ</span><span style="font-variant-ligatures: no-common-ligatures;">^R</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">C</span><span style="font-variant-ligatures: no-common-ligatures;"><8a></span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">9</span><span style="font-variant-ligatures: no-common-ligatures;"><99></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">aq</span><span style="font-variant-ligatures: no-common-ligatures;">^O^O<9f></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">Cv</span><span style="font-variant-ligatures: no-common-ligatures;">^U^A</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">þXQi</span><span style="font-variant-ligatures: no-common-ligatures;"><88></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">ÛÌ®°Üx</span><span style="font-variant-ligatures: no-common-ligatures;"><90></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">µó</span><span style="font-variant-ligatures: no-common-ligatures;"><98>^G</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">l ´Z</span><span style="font-variant-ligatures: no-common-ligatures;">^B<81>^E</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">·¾&</span><span style="font-variant-ligatures: no-common-ligatures;">^B^C<83></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">ÕÚ§</span></div>
<div style="background-color: black; color: #c33720; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">,g</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;">^L</span><span style="font-variant-ligatures: no-common-ligatures;">Ã=kW</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;">^Y</span><span style="font-variant-ligatures: no-common-ligatures;">$râ Æ]HÉ</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;">^U</span><span style="font-variant-ligatures: no-common-ligatures;">Ý5AÂ[V</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;">^F</span><span style="font-variant-ligatures: no-common-ligatures;">C$y</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;"><80></span><span style="font-variant-ligatures: no-common-ligatures;">¢a</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;"><82></span><span style="font-variant-ligatures: no-common-ligatures;">y</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;">^V^U</span><span style="font-variant-ligatures: no-common-ligatures;">{°(Ô</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;"><9b></span><span style="font-variant-ligatures: no-common-ligatures;">|Ü</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;">^X</span><span style="font-variant-ligatures: no-common-ligatures;">W}¤h</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;"><8f></span><span style="font-variant-ligatures: no-common-ligatures;">aØ}¦</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;">^G</span><span style="font-variant-ligatures: no-common-ligatures;">>o</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;"><96></span><span style="font-variant-ligatures: no-common-ligatures;">êu</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;"><9c></span><span style="font-variant-ligatures: no-common-ligatures;">e?k</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;"><8e></span><span style="font-variant-ligatures: no-common-ligatures;">,àbºÔ/ýëh´¶</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;">^L</span><span style="font-variant-ligatures: no-common-ligatures;">Çp</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;">^A^Y</span><span style="font-variant-ligatures: no-common-ligatures;">e</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;"><8b></span><span style="font-variant-ligatures: no-common-ligatures;">LÕBt</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;"><85></span><span style="font-variant-ligatures: no-common-ligatures;">w£hi¿&(</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;">^H</span><span style="font-variant-ligatures: no-common-ligatures;">Ûîä</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;"><96></span><span style="font-variant-ligatures: no-common-ligatures;">tÿ.</span><span style="color: #5330e1; font-variant-ligatures: no-common-ligatures;">^@</span><span style="font-variant-ligatures: no-common-ligatures;">A></span></div>
<div style="background-color: black; color: #c33720; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">dJuëX</span></div>
</div>
<div>
<br /></div>
<div>
I stumbled upon a tool called <a href="https://github.com/pantsbuild/pex" target="_blank">pex</a> which will automate generating these shebang/zip files. While I found pex useful, it seems too much for my needs. So I started cobbling together my own shebang/zip files by hand as I needed them.</div>
<div>
<br /></div>
<div>
Ultimately, wanted to automate this some, so I ended up creating a simple bash script (which I named 'compile.sh') to speed things up. Here's that script. Hope you find this useful.</div>
<div>
<br /></div>
<div>
<div style="background-color: black; color: #5330e1; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">#!/bin/bash</span></div>
<div style="background-color: black; color: #5330e1; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">#</span></div>
<div style="background-color: black; color: #5330e1; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;"># compile.sh</span></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal; min-height: 15px;">
<span style="font-variant-ligatures: no-common-ligatures;"></span><br /></div>
<div style="background-color: black; color: #5330e1; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;"># Creates a self-runnable, single-file deployable for your python project.</span></div>
<div style="background-color: black; color: #5330e1; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;"># Your python project's main entry point must be in a file named __main__.py.</span></div>
<div style="background-color: black; color: #5330e1; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;"># It will embed in the deployable every dependency you've pip-installed in</span></div>
<div style="background-color: black; color: #5330e1; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;"># your virtualenv .env directory.</span></div>
<div style="background-color: black; color: #5330e1; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">#</span></div>
<div style="background-color: black; color: #5330e1; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;"># The deployable still requires that python is installed on the target system.</span></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal; min-height: 15px;">
<span style="font-variant-ligatures: no-common-ligatures;"></span><br /></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">if </span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">[[</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">!</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">-d</span><span style="font-variant-ligatures: no-common-ligatures;"> .env/lib/python2.</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">7</span><span style="font-variant-ligatures: no-common-ligatures;">/site-packages </span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">]]</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">;</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">then</span></div>
<div style="background-color: black; color: #c33720; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">echo</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="font-variant-ligatures: no-common-ligatures;">This script must be run from a virtualenv root.</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">exit</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">1</span></div>
<div style="background-color: black; color: #ce7924; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">fi</span></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal; min-height: 15px;">
<span style="font-variant-ligatures: no-common-ligatures;"></span><br /></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">if </span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">[[</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">!</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">-f</span><span style="font-variant-ligatures: no-common-ligatures;"> __main__.py </span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">]]</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">;</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">then</span></div>
<div style="background-color: black; color: #c33720; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">echo</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="font-variant-ligatures: no-common-ligatures;">This will only work if you have a __main__.py file.</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">exit</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">1</span></div>
<div style="background-color: black; color: #ce7924; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">fi</span></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal; min-height: 15px;">
<span style="font-variant-ligatures: no-common-ligatures;"></span><br /></div>
<div style="background-color: black; color: #ce7924; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">if </span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">[[</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="font-variant-ligatures: no-common-ligatures;">-z</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="font-variant-ligatures: no-common-ligatures;">"</span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">$1</span><span style="font-variant-ligatures: no-common-ligatures;">"</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">]]</span><span style="font-variant-ligatures: no-common-ligatures;">;</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="font-variant-ligatures: no-common-ligatures;">then</span></div>
<div style="background-color: black; color: #c33720; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">echo</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="font-variant-ligatures: no-common-ligatures;">Usage: </span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">$(basename $0)</span><span style="font-variant-ligatures: no-common-ligatures;"> {output_filename}</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">exit</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">1</span></div>
<div style="background-color: black; color: #ce7924; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">fi</span></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal; min-height: 15px;">
<span style="font-variant-ligatures: no-common-ligatures;"></span><br /></div>
<div style="background-color: black; color: #d53bd3; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: #34bbc7; font-variant-ligatures: no-common-ligatures;">TMPDIR</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">=</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="font-variant-ligatures: no-common-ligatures;">$(mktemp -u /tmp/python.compile.XXXXXXX)</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span></div>
<div style="background-color: black; color: #34bbc7; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">CURDIR</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">=</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">$(</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">pwd</span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">)</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span></div>
<div style="background-color: black; color: #34bbc7; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">TARGET</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">=</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">$1</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal; min-height: 15px;">
<span style="font-variant-ligatures: no-common-ligatures;"></span><br /></div>
<div style="background-color: black; color: #5330e1; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;"># Create payload zip with our code (remove extra junk)</span></div>
<div style="background-color: black; color: #d53bd3; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">mkdir</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="font-variant-ligatures: no-common-ligatures;">-p</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="font-variant-ligatures: no-common-ligatures;">$TMPDIR</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">/deps</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span></div>
<div style="background-color: black; color: #c33720; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">zip </span><span style="font-variant-ligatures: no-common-ligatures;">-9</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">$TMPDIR</span><span style="font-variant-ligatures: no-common-ligatures;">/payload.zip</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> *</span></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">zip </span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">-d</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">$TMPDIR</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">/payload.zip</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="font-variant-ligatures: no-common-ligatures;"> compile.sh</span></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">zip </span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">-d</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">$TMPDIR</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">/payload.zip</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="font-variant-ligatures: no-common-ligatures;"> requirements.txt</span></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">zip </span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">-d</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">$TMPDIR</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">/payload.zip</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">$TARGET</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">></span><span style="font-variant-ligatures: no-common-ligatures;"> /dev/null</span></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal; min-height: 15px;">
<span style="font-variant-ligatures: no-common-ligatures;"></span><br /></div>
<div style="background-color: black; color: #5330e1; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;"># Gather the virtualenv packages and clean them up</span></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">cp </span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">-R</span><span style="font-variant-ligatures: no-common-ligatures;"> .env/lib/python2.</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">7</span><span style="font-variant-ligatures: no-common-ligatures;">/site-packages/* </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">$TMPDIR</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">/deps</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span></div>
<div style="background-color: black; color: #d53bd3; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">find</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="font-variant-ligatures: no-common-ligatures;">$TMPDIR</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">/deps</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="font-variant-ligatures: no-common-ligatures;">-iname</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">'</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">*.pyc</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">'</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="font-variant-ligatures: no-common-ligatures;">-delete</span></div>
<div style="background-color: black; color: #d53bd3; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">find</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="font-variant-ligatures: no-common-ligatures;">$TMPDIR</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">/deps</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="font-variant-ligatures: no-common-ligatures;">-ipath</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">'</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">*pip*</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">'</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="font-variant-ligatures: no-common-ligatures;">-delete</span></div>
<div style="background-color: black; color: #d53bd3; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">find</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="font-variant-ligatures: no-common-ligatures;">$TMPDIR</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">/deps</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="font-variant-ligatures: no-common-ligatures;">-ipath</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">'</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">*easy_install*</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">'</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="font-variant-ligatures: no-common-ligatures;">-delete</span></div>
<div style="background-color: black; color: #d53bd3; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">find</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="font-variant-ligatures: no-common-ligatures;">$TMPDIR</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">/deps</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="font-variant-ligatures: no-common-ligatures;">-ipath</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">'</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">*dist-info*</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">'</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="font-variant-ligatures: no-common-ligatures;">-delete</span></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal; min-height: 15px;">
<span style="font-variant-ligatures: no-common-ligatures;"></span><br /></div>
<div style="background-color: black; color: #5330e1; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;"># Add the virtualenv packages to the payload</span></div>
<div style="background-color: black; color: #d53bd3; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">cd</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="font-variant-ligatures: no-common-ligatures;">$TMPDIR</span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">/deps</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;">zip </span><span style="color: #c33720; font-variant-ligatures: no-common-ligatures;">-9</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">-r</span><span style="font-variant-ligatures: no-common-ligatures;"> ../payload.zip *</span></div>
<div style="background-color: black; color: #d53bd3; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">cd</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="font-variant-ligatures: no-common-ligatures;">$CURDIR</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal; min-height: 15px;">
<span style="font-variant-ligatures: no-common-ligatures;"></span><br /></div>
<div style="background-color: black; color: #5330e1; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;"># Assemble the payload into a runable</span></div>
<div style="background-color: black; color: #c33720; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">echo</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">'</span><span style="font-variant-ligatures: no-common-ligatures;">#!/usr/bin/env python</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">'</span><span style="font-variant-ligatures: no-common-ligatures;"> </span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;">| cat - </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">$TMPDIR</span><span style="font-variant-ligatures: no-common-ligatures;">/payload.zip</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">></span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="color: #d53bd3; font-variant-ligatures: no-common-ligatures;">$TARGET</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span></div>
<div style="background-color: black; color: #d53bd3; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">chmod</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="font-variant-ligatures: no-common-ligatures;">+x</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="font-variant-ligatures: no-common-ligatures;">$TARGET</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span></div>
<div style="background-color: black; color: whitesmoke; font-family: Monaco; font-size: 11px; line-height: normal; min-height: 15px;">
<span style="font-variant-ligatures: no-common-ligatures;"></span><br /></div>
<div style="background-color: black; color: #5330e1; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="font-variant-ligatures: no-common-ligatures;"># Cleanup</span></div>
<div style="background-color: black; color: #d53bd3; font-family: Monaco; font-size: 11px; line-height: normal;">
<span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">rm</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="font-variant-ligatures: no-common-ligatures;">-r</span><span style="color: whitesmoke; font-variant-ligatures: no-common-ligatures;"> </span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span><span style="font-variant-ligatures: no-common-ligatures;">$TMPDIR</span><span style="color: #ce7924; font-variant-ligatures: no-common-ligatures;">"</span></div>
</div>
bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-54916521079073407622015-10-07T11:33:00.001-04:002015-10-07T11:34:06.442-04:00Python Decorators ExamplesI often go long stretches between times when I aggressively use python at clients. As a result, I get rusty on what I could consider somewhat advanced topics and have to spend time googling around to jog my memory. Reading my own code usually gets me back up and running quicker than reading others' code. Leaving this here as a breadcrumb to help me remember how to use python decorators.<br />
<br />
If you find this helpful, awesome!<br />
<br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>#!/usr/bin/env python</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>"""Decorator examples"""</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>def decorator_without_arguments(func):</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> def wrapped_function(*args, **kwargs):</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> """Reverses the arguments before calling"""</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> new_args = reversed(args)</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> return func(*new_args, **kwargs)</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> wrapped_function.__name__ = "(%s -> %s)" % \</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> (decorator_without_arguments.__name__, func.__name__)</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> return wrapped_function</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>def decorator_with_arguments(arg1, arg2=None):</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> def actual_decorator(func):</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> def wrapped_function(*args, **kwargs):</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> """Wraps results in explanation string"""</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> result = "(%s: %s) " % (arg1, ', '.join([str(x) for x in args]))</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> result += str(func(*args, **kwargs))</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> if arg2 is not None:</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> result += arg2</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> return result</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> wrapped_function.__name__ = func.__name__</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> return wrapped_function</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> return actual_decorator</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>def undecorated_subtract(number_1, number_2):</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> return number_1 - number_2</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>@decorator_without_arguments</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>def decorated_subtract(number_1, number_2):</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> return number_1 - number_2</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>def undecorated_add(number_1, number_2):</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> return number_1 + number_2</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>@decorator_with_arguments('ADD')</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>def decorated_add(number_1, number_2):</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> return number_1 + number_2</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>if __name__ == '__main__':</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> print undecorated_subtract(7, 3)</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> print decorated_subtract(7, 3)</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> print decorated_subtract</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> print undecorated_add(3, 7)</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> print decorated_add(3, 7)</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> print decorated_add</b></span><br />
<div>
<br /></div>
bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-31570200051530659912014-07-23T13:17:00.003-04:002014-07-23T13:17:43.040-04:00Embedding python in bash scriptsAs a software development consultant, I do a lot of bash scripting. I can do a lot of really creative things using nothing but bash and the binutils at my disposal, but sometimes I'll come across something that's just easier to do in a higher level scripting language. Enter python.<div>
<br /></div>
<div>
Conversely, there are a lot of things that are just easier and more straight forward to do in bash, so writing everything in straight python may be more work than it's worth.</div>
<div>
<br /></div>
<div>
Here's a quick posting to describe how you can embed some python code into your bash scripts and get the best of both worlds. <i>Note: just as a heads, up, the examples in this posting are quite contrived.</i></div>
<div>
<br /></div>
<div>
Calling python from bash is easy. You simply use python's '-' argument and pipe in your python code. I typically wrap my python code in a bash function.</div>
<div>
<br /></div>
<div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>#!/bin/bash</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>function current_datetime {</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>python - <<END</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>import datetime</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>print datetime.datetime.now()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>END</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>}</b></span></div>
</div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b># Call it</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>current_datetime</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b># Call it and capture the output</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>DT=$(current_datetime)</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>echo Current date and time: $DT</b></span></div>
<div>
<br /></div>
<div>
You can also pass data into your embedded python script. I do that using environment variables:</div>
<div>
<br /></div>
<div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>#!/bin/bash</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>function line {</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>PYTHON_ARG="$1" python - <<END</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>import os</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>line_len = int(os.environ['PYTHON_ARG'])</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>print '-' * line_len</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>END</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>}</b></span></div>
</div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b># Do it one way</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>line 80</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>echo 'Handy'</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b># Do it another way</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>echo $(line 80)</b></span></div>
<div>
<br /></div>
<div>
My usual use-case for doing this is if I'm extending someone else's bash scripts and have to 'go off the reservation' a bit. Sometimes I'm updating an existing 'legacy' script and need to look up some data... maybe do a REST call or something. Here's an example bash script that uses curl to call a REST service to get some weather data. Then is passes the raw JSON response to an embedded python script to interpret and format the results:</div>
<div>
<br /></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>#!/bin/bash</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>function format_weather_data() {</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>PYTHON_ARG="$1" python - <<END</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>import os</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>import json</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>json_data = os.environ['PYTHON_ARG']</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>data =json.loads(json_data)</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>lookup = {</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '200': 'thunderstorm with light rain',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '201': 'thunderstorm with rain',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '202': 'thunderstorm with heavy rain',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '210': 'light thunderstorm',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '211': 'thunderstorm',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '212': 'heavy thunderstorm',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '221': 'ragged thunderstorm',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '230': 'thunderstorm with light drizzle',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '231': 'thunderstorm with drizzle',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '232': 'thunderstorm with heavy drizzle',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '300': 'light intensity drizzle',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '301': 'drizzle',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '302': 'heavy intensity drizzle',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '310': 'light intensity drizzle rain',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '311': 'drizzle rain',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '312': 'heavy intensity drizzle rain',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '313': 'shower rain and drizzle',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '314': 'heavy shower rain and drizzle',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '321': 'shower drizzle',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '500': 'light rain',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '501': 'moderate rain',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '502': 'heavy intensity rain',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '503': 'very heavy rain',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '504': 'extreme rain',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '511': 'freezing rain',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '520': 'light intensity shower rain',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '521': 'shower rain',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '522': 'heavy intensity shower rain',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '531': 'ragged shower rain',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '600': 'light snow',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '601': 'snow',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '602': 'heavy snow',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '611': 'sleet',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '612': 'shower sleet',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '615': 'light rain and snow',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '616': 'rain and snow',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '620': 'light shower snow',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '621': 'shower snow',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '622': 'heavy shower snow',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '701': 'mist',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '711': 'smoke',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '721': 'haze',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '731': 'sand, dust whirls',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '741': 'fog',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '751': 'sand',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '761': 'dust',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '762': 'volcanic ash',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '771': 'squalls',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '781': 'tornado',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '800': 'clear sky',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '801': 'few clouds',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '802': 'scattered clouds',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '803': 'broken clouds',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '804': 'overcast clouds',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '900': 'tornado',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '901': 'tropical storm',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '902': 'hurricane', </b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '903': 'cold',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '904': 'hot',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '905': 'windy',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '906': 'hail',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '950': 'setting',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '951': 'calm',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '952': 'light breeze',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '953': 'gentle breeze',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '954': 'moderate breeze',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '955': 'fresh breeze',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '956': 'strong breeze',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '957': 'high wind, near gale',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '958': 'gale',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '959': 'severe gale',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '960': 'storm',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '961': 'violent storm',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> '962': 'hurricane',</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>}</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>print "Current temperature: %g F" % data['main']['temp']</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>print "Today's high: %g F" % data['main']['temp_max']</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>print "Today's low: %g F" % data['main']['temp_min']</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>print "Wind speed: %g mi/hr" % data['wind']['speed']</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>weather_descs = [lookup.get(str(i['id']), '*error*') for i in data['weather']]</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>print "Weather: %s" % ', '.join(weather_descs)</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>END</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>}</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>WEATHER_URL="http://api.openweathermap.org/data/2.5/weather?q=Cincinnati,OH&units=imperial"</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>format_weather_data "$(curl -s $WEATHER_URL)"</b></span></div>
<div>
<br /></div>
<div>
Hope you find this information useful. </div>
<div>
<br /></div>
bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-81028735519233423482014-01-22T16:21:00.002-05:002014-01-22T16:21:19.080-05:00Formatting GUIDs with sedAt my current client, I'm often given GUIDs for one reason or another. Often they are not in the format that I need them to be. Sometimes they have the dashes and I need them to be dash-free. Other times, the dashes have been removed and I need them back.<br />
<br />
Here's some quick, down and dirty <i>sed </i>commands to swap them back and forth:<br />
<br />
To remove the dashes:<br />
<br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>$ echo 8EC60070-685F-41DB-C881-EACF9E74E4BD | sed 's/-//g'</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>8EC60070685F41DBC881EACF9E74E4BD</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>$</b></span><br />
<br />
To put them back:<br />
<br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>$ echo 8EC60070685F41DBC881EACF9E74E4BD | sed -rn 's/([0-9A-F]{8})([0-9A-F]{4})([0-9A-F]{4})([0-9A-F]{4})([0-9A-F]{12})/\1-\2-\3-\4-\5/p')</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>8EC60070-685F-41DB-C881-EACF9E74E4BD</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>$</b></span><br />
<br />
It goes without saying you can embed this very <i>sed</i> command in a bash script and run it over many rows in a file.bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-2500036347878942622013-03-14T14:26:00.000-04:002013-03-14T14:26:28.739-04:00Monitoring a web page for changes using bashThere's this conference that I'd like to attend and I've heard that it's a hard-to-get-into type conference. When I go to their site it doesn't have any new info.<br />
<div>
<br /></div>
<div>
Rather than checking the site every day, I'd like to have it monitored and be alerted when something new DOES appear on it.</div>
<div>
<br /></div>
<div>
Now I know there are services like <a href="http://changedetection.com/">ChangeDetection.com</a> that can monitor it for me, but I was wanting to cobble something together with the tools I already have. I'd also like to have the ability to customize what it consider "a change" at my disposal when/if I need it.</div>
<div>
<br /></div>
<div>
To that end, I threw together the following bash script. It monitors a URL and if it detects a change, it sends an email to my gmail account letting me know.</div>
<div>
<br /></div>
<div>
Hope you find it useful. BTW, I'm using a program called sendEmail to send the email notification. It's in apt if you're using a debian/ubuntu-like distribution.</div>
<div>
<br /></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>#!/bin/bash</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b><br /></b></span>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b># monitor.sh - Monitors a web page for changes</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b># sends an email notification if the file change</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>USERNAME="me@gmail.com"</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>PASSWORD="itzasecret"</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>URL="http://thepage.com/that/I/want/to/monitor"</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>for (( ; ; )); do</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> mv new.html old.html 2> /dev/null</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> curl $URL -L --compressed -s > new.html</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> DIFF_OUTPUT="$(diff new.html old.html)"</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> if [ "0" != "${#DIFF_OUTPUT}" ]; then</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> sendEmail -f $USERNAME -s smtp.gmail.com:587 \</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> -xu $USERNAME -xp $PASSWORD -t $USERNAME \</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> -o tls=yes -u "Web page changed" \</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> -m "Visit it at $URL"</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> sleep 10</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b> fi</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>done</b></span></div>
<div>
<br /></div>
<div>
Then from a bash prompt I run it with the following command:</div>
<div>
<br /></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>nohup ./monitor.sh &</b></span></div>
<div>
<br /></div>
<div>
Using nohup and throwing it in the background allows me to log out and have the script continue to run.</div>
bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-4318600755524270152013-02-04T22:38:00.000-05:002013-02-04T22:41:09.595-05:00Fun with Python's multiprocessing moduleAt my current client, they asked me to write a Python script to aggregate data from a RESTful web service. Essentially, they have a web service end-point that takes a single customer ID as an argument and it returns some customer profile records as JSON.<br />
<br />
They wanted my script to call the web service for each and every customer listed in a text file. The script will be run from a scheduled job (cron, windows scheduled task, etc.). Once productionalized, the text file will contain a large number of customer ID's (like a million or so). All the results need to be stored in a single text file for the run.<br />
<br />
My first thought was to use Python's threading features, but I came with 2 problems with this:<br />
<br />
<ul>
<li>Python's threading isn't "real" threading. A single Python process will run all threads on a single physical core regardless of how many cores the server has. The other cores will not be utilized.</li>
<li>Python's urllib and urllib2 packages aren't even thread safe.</li>
</ul>
<div>
Now I know there's a third-party thread-safe urllib3 package, but that doesn't solve the multi-core issue and I wasn't looking to install anything beyond Python 2.7's standard library on the server. After doing a little research, I came up with the idea of using Python's multiprocessing module.</div>
<div>
<br /></div>
<div>
The rest of this blog posting is a walkthrough of what I ended up building.</div>
<h3>
First things first - configuration</h3>
<div>
Since this is a script that will be run often and certain configuration settings will need to be updated from time to time by support staff, I decided to extract the configuration out to a separate configuration file. Initially, I had the settings stored in a .ini file, but I later thought why not use Python for settings. To that end, I have a <b><span style="color: blue;">settings.py</span></b> file that holds a class that contains the settings:</div>
<div>
<br /></div>
<div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>#!/usr/bin/env python</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>"""</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>Settings file. See job.py for more info.</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>"""</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b># -----------------------------------------------------------</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b># Edit the values in this Settings class to control settings.</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b># -----------------------------------------------------------</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>class Settings:</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> """Settings as a class. Why? Because I'm lazy."""</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> source_file = 'TestData.csv'</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> profile_data_file = 'results.txt'</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> log_file = 'log.txt'</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> failed_customer_file = 'failed.txt'</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> process_count = 10</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> url = </b></span><b style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: small;">'http://division-1.internal-server.com/rest-service/endpoint/{0}'</b></div>
</div>
<div>
<br /></div>
<div>
There is one input file (<b>source_file</b>). This is the file that contains the list of customer ID's. Then there are 3 output files: <b>profile_data_file</b>, <b>log_file</b> and <b>failed_customer_file</b>. The <b>profile_data_file</b> file is where the script will put the customer detail records that it receives from the web service. The <b>log_file</b> is where the script will write logging messages for debugging problems after the fact. Finally, <b>failed_customer_file</b> is where the script will write customer ID's of customers that the script fails to retrieve from the web services.</div>
<div>
<br /></div>
<div>
<b>process_count</b> is a setting that will allow you to specify how many parallel processes the script should spawn.</div>
<div>
<br /></div>
<div>
<b>url</b> is the url of the web service end-point.</div>
<div>
<br /></div>
<div>
The main job script is called <b><span style="color: blue;">job.py</span></b>. It imports the settings.py module and then creates a new class that inherits from the Settings class.</div>
<div>
<br /></div>
<div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>class Config(settings.Settings):</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> """Configuration class"""</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> _runtime = datetime.datetime.now().strftime('%Y%m%d-%H%M%S')</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> </b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> @staticmethod</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> def timestamp(filename):</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> """Stamps a filename with the timestamp for this run"""</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> parts = os.path.splitext(filename)</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> return "%s-%s%s" % (parts[0], Config._runtime, parts[1])</b></span></div>
</div>
<div>
<br /></div>
<div>
This class has a static method called timestamp. In the main job, I will call the timestamp method to add the date/time of the run to the output filenames. So you might see something like:</div>
<div>
<br /></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;">outfile = open(Config.timestamp(Config.log_file), 'w')</span></b></div>
<div>
<br /></div>
<div>
This will open a file named 'log-20120204-112530.txt' for writing.</div>
<h3>
The Job class</h3>
<div>
The Job class is the main class that executes the job and spawns the worker processes. It has the following methods:</div>
<div>
<ul>
<li><b>__init__</b> - constructor, set's up a multiprocessing Pool and 3 Queue's</li>
<li><b>run</b> - Kicks off the job</li>
<li><b>get_customers</b> - Reads the customer file</li>
<li><b>get_requests</b> - Transforms the customers list into a list of processes requests</li>
<li><b>process_customer_queue</b> - Deals with responses from the web service</li>
<li><b>process_log_queue</b> - Deals with log messages from the worker processes</li>
<li><b>process_exceptions_queue</b> - Deals with exceptions that occur in the worker processes</li>
<li><b>log</b> - Writes log messages to the log file</li>
</ul>
</div>
<div>
Here's a copy of the Job class:</div>
<div>
<br /></div>
<div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b>class Job:</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> """Job that performs the run."""</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> def __init__(self):</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> self._manager = multiprocessing.Manager()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> self._pool = multiprocessing.Pool(Config.process_count)</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> self._customer_queue = self._manager.Queue()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> self._log_queue = self._manager.Queue()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> self._exception_queue = self._manager.Queue()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> def run(self):</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> """Do it."""</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> start_time = datetime.datetime.now()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> self.log('[Run started]')</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> customers = self.get_customers()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> requests = self.get_requests(customers)</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> result = self._pool.map_async(get_customer, requests)</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> while not result.ready():</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> self.process_customer_queue()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> self.process_log_queue()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> self.process_exceptions_queue()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> self._pool.close()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> self._pool.join()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> self.process_customer_queue()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> self.process_log_queue()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> self.process_exceptions_queue()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> self.log('[Run finished]')</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> self.log('[Total runtime: %s]' % (datetime.datetime.now() - start_time))</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> def get_customers(self):</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> """Read the source file."""</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> buf = [line.strip() for line in open(Config.source_file).readlines()[1:]]</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> return buf</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> def get_requests(self, customers):</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> """Generate requests from customers."""</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> requests = [{ \</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> 'customer': customer, \</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> 'data': self._customer_queue, \</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> 'log': self._log_queue, \</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> 'exceptions': self._exception_queue, \</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> 'url': Config.url, \</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> } for customer in customers]</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> return requests</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> def process_customer_queue(self):</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> """Pull messages off the data queue."""</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> try:</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> message = self._customer_queue.get_nowait()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> except Queue.Empty:</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> return</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> customer = message['customer']</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> details = message['server_response']['details']</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> outfile = open(Config.timestamp(Config.profile_data_file), 'a')</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> for record in details:</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> buf = "%s, %s, %s, %s\n" % (customer, record['id'], \</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> record['relevanceScore'], record['relevanceRank'])</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> outfile.write(buf)</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> outfile.close()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> </b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> def process_log_queue(self):</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> """Pull messages off the log queue."""</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> try:</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> message = self._log_queue.get_nowait()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> except Queue.Empty:</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> return</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> self.log(message)</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> def process_exceptions_queue(self):</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> """Pull messages off the exceptions queue."""</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> try:</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> message = self._exception_queue.get_nowait()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> except Queue.Empty:</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> return</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> customer = message['customer']</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> exception = message['exception']</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> self.log("EXCEPTION GETTING %s! - %s" % \</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> (customer, str(exception)))</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> failed_file = open(Config.timestamp(Config.failed_customer_file), 'a')</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> buf = "%s\n" % customer</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> failed_file.write(buf)</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> failed_file.close()</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> def log(self, message):</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> """Write message to the log file."""</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> logfile = open(Config.timestamp(Config.log_file), 'a')</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> timestamp = datetime.datetime.now().strftime('%Y/%m/%d %H:%M:%S')</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> logfile.write("%s - %s\n" % (timestamp, message))</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"><b> logfile.close()</b></span></div>
</div>
<div>
<br /></div>
<div>
Some important things to highlight. The constructor creates three queues: customer_queue, log_queue, and exceptions_queue. These queues are used for interprocess communication. The constructor creates a Pool of worker processes. Each process will communicate back to the parent process via these queues. The customer_queue is responsible for holding the results of the web services calls. The log_queue is responsible for holding log messages. The exceptions_queue is responsible for holding exceptions that occur when worker processes have problems talking to the web service.</div>
<div>
<br /></div>
<div>
In the run method, the Job class will call the Pool class' map_async method. The map_async method splits the requests list into multiple lists and passes each list to a separate worker process for processing. Then the run method goes into a loop, waiting for the worker processes to all complete. While it's waiting, it will continually check the queues to see if any messages have been received.</div>
<h3>
The stand-alone function</h3>
<div>
The map_async method's first argument is the name of the function that each worker process should run the request list items with. In this case, the name of that function is get_customer. get_customer looks like this:</div>
<div>
<br /></div>
<div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;">def get_customer(request):</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> """Retrieve the customer details from the given request."""</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> customer = request['customer']</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> url = request['url']</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> data = request['data']</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> log = request['log']</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> exceptions = request['exceptions']</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> try:</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> log.put("Requesting details for customer %s" % customer)</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> req_url = url.replace('{0}', customer)</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> request = urllib2.Request(req_url)</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> response = urllib2.urlopen(request)</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> buf = response.read()</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> data_records = json.loads(buf)</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> coupons.put({'customer: customer, 'server_response': data_records})</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> log.put("Successfully retrieved %d details(s) for %s" % \</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> (len(data_records), customer))</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> except Exception, exc:</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> exceptions.put({'customer': customer, 'exception': exc})</span></b></div>
<div>
<br /></div>
</div>
<div>
This function is called for each request in the worker processes. It extracts queues from the request. It uses the put method on each queue to pass information back to the parent process.</div>
<h3>
Pulling it all together</h3>
<div>
The last thing in my script file is this:</div>
<div>
<br /></div>
<div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;">if __name__ == '__main__':</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> # Let's do this!</span></b></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;"> Job().run()</span></b></div>
</div>
<div>
<br /></div>
<div>
It's important to embed the Job().run() call inside the 'if __main__' branching. This is because of how the multiprocessing internals work. When the Pool.map_async method is executed, Python will create a number of sub processes. When each process starts up, it's actually spawning a copy of the Python executable. When that process comes up, it imports the job module and then starts calling the get_customer function over and over with the requests. If the Job().run() were not embedded in a 'if __main__' branch, every worker process would spawn more Job classes. Nothing good would come of that!</div>
<div>
<br /></div>
<div>
Hope you find this helpful.</div>
bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-64374455331214857642013-01-29T11:41:00.002-05:002013-01-31T09:18:55.099-05:00Create and track svn branches using git-svnMy most recent client is utilizing svn for the source control needs. While I used to be a big svn guy, I've more recently been using git and have found the transition back to svn a little awkward.<br />
<div>
<br /></div>
<div>
Enter: git-svn. I love git-svn because I can exist in a svn environment without giving up all the things I love about git. One thing that continually escapes my memory is how to create a svn branch from the git CLI as well as how to associate a git branch with a svn branch.</div>
<div>
<br /></div>
<div>
This blog entry is my own cheat-sheet.</div>
<div>
<br /></div>
<div>
Say, I want to create a svn branch off of the trunk. Here's what I would do:</div>
<div>
<br /></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>$ git checkout master</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>$ git svn rebase</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>$ git svn branch -m "Creating a feature branch" feature_1001</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>$ git svn dcommit</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>$ git checkout -b my_feature_1001 feature_1001</b></span></div>
<div>
<br /></div>
<div>
Another thing that isn't so intuitive is merging your changes back to trunk when you're done. Here's what I would do:<br />
<br />
<br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>$ git checkout feature_1001</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>$ git rebase master</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>$ git checkout master</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>$ git merge feature_1001</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>$ git svn dcommit</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>$ git branch -D feature_1001</b></span><br />
<br />
Then I have to use a svn client to actually delete the svn branch.<br />
<br />
Also, if a svn branch is removed, you need to remove it by hand from git with<br />
<br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>$ git branch -rd feature_1001</b></span><br />
<br />
<br />
<br /></div>
bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-11600048216647134772012-10-19T10:24:00.003-04:002012-10-19T10:24:35.844-04:00Reformatting for 80 characters in vimAfter reading <a href="http://www.amazon.com/Pragmatic-Programmer-Journeyman-Master/dp/020161622X" target="_blank">The Pragmatic Programmer</a> and <a href="http://www.amazon.com/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882" target="_blank">Clean Code</a>, I've pretty much settled on using text files (specifically in <a href="http://daringfireball.net/projects/markdown/" target="_blank">markdown</a> format) for all my personal notes and documentation and editing them with <a href="http://www.vim.org/" target="_blank">vim</a>. That being said, I like to keep my files limited to 80 columns wide. That way I can look at and review multiple documents simultaneously in splits and vertical split easily.<div>
<br /></div>
<div>
In my .vimrc I have the following setting to get the auto wrapping working for me:</div>
<div>
<br /></div>
<div>
<b><span style="color: blue; font-family: Courier New, Courier, monospace;">set textwidth=80</span></b></div>
<div>
<br /></div>
<div>
That way as I type along, vim is keeping track of the file's column width and when I get to the 80th column it automatically moves me to the next line. Pretty handy.</div>
<div>
<br /></div>
<div>
Here's my problem though, if I change some text that I've already entered and reword it, my paragraph loses it's nicely formatted set column width. Here's an example but with the textwidth setting set at 40:</div>
<div>
<br /></div>
<div>
Before my edit change:</div>
<div>
<br /></div>
<div>
<div>
<div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;">Lorem ipsum dolor sit amet, consectetur</span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;">adipiscing elit. Proin neque sapien,</span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;">facilisis eget tincidunt ut, porttitor</span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;">laoreet lacus. Class aptent taciti</span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;">sociosqu ad litora torquent per conubia</span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;">nostra, per inceptos himenaeos. Etiam</span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;">semper elementum congue.</span></div>
</div>
</div>
</div>
<div>
<br /></div>
<div>
After my edit change:</div>
<div>
<br /></div>
<div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;">Lorem ipsum dolor sit amet, consectetur</span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;">adipiscing elit. Proin neque sapien,</span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;">facilisis eget tincidunt ut, porttitor</span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;">laoreet lacus. Class aptent taciti</span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;">sociosqu ad litora torquent per conubia</span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;">nostra, per inceptos himenaeos. <b>I</b></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;"><b>forgot some text.</b> Etiam</span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace;">semper elementum congue.</span></div>
</div>
<div>
<br /></div>
<div>
Notice the problem: After editing the paragraph, the second to last line doesn't go to the 40th column before wrapping. This is a little annoying to me because I feel the need to reformat the paragraph to get it back to looking like the "before". It's not too big of a deal when I only have to reformat a line or two but if I've change something high up in a fairly long paragraph, reformatting dozens of lines can get tedious.</div>
<div>
<br /></div>
<div>
Here's a little tip that I stumbled across that can make that reformatting go a bit quicker:</div>
<div>
<br /></div>
<div>
1. Select the whole paragraph with <b><span style="color: blue; font-family: Courier New, Courier, monospace;">V</span></b>. and then <b><span style="color: blue; font-family: Courier New, Courier, monospace;">h</span></b>,<b><span style="color: blue; font-family: Courier New, Courier, monospace;">j</span></b>,<b><span style="color: blue; font-family: Courier New, Courier, monospace;">k</span></b> and/or <b><span style="color: blue; font-family: Courier New, Courier, monospace;">l</span></b>.</div>
<div>
2. Then strip the new-line characters with <b><span style="color: blue; font-family: Courier New, Courier, monospace;">J</span></b>.</div>
<div>
3. Finally reformat it with <b><span style="color: blue; font-family: Courier New, Courier, monospace;">gq</span></b>.</div>
<div>
<br /></div>
<div>
Hope you find this helpful!</div>
bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-57628583747086860382012-07-06T12:32:00.002-04:002012-07-06T12:47:42.488-04:00CGI Script to display clone urls<br />
After some discussion, I was able to convince my client to convert from gitorious to gitolite. One nice feature that people like about gitorious is that the web interface provides an easy way to look up urls for cloning repositories.<br />
<br />
In my mind, that's a pretty legitimate need. To that end, I threw together this bash script, which acts as a cgi and dropped it in the <b><span style="color: blue;">cgi-bin</span></b> directory of the server that's running gitolite. Hope you find this helpful.<br />
<br />
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">#!/bin/bash</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">REPOSITORY_DIR="/home/git/repositories/"</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">URL_PREFIX="git clone git@internal-build-server:"</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">cat <<DONE</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Content-type: text/html</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><html></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><head></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span><title>Repository URLs</title></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span><link rel="stylesheet" type="text/css" href="/index.css" /></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"></head></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><body></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><div id="page_container"></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span><h1></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="background-color: white; white-space: pre;"> </span><span style="background-color: white;">Repository URLs</span></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span></h1></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span><p></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Repository URLs on this server follow a specific pattern. The pattern is</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>as follows:</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span></p></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span><center></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>git@</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">internal-build-server</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">:<i><font color="darkblue">{category}</font></i>/<i><font color="darkblue">{project}</font></i></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span></center></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span><p></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>These URLs are both pull and push URLs. You do not need separate URLs</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>for pulling and pushing. Access control will be handled by a</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>server git update hook that is provided by gitolite.</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span></p></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span><p></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>In an effort to make life a little easier in locating your URLs, this script</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>enumerates URLs for the repositories located on this machine below.</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span></p></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">DONE</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">CATEGORIES=$(find $REPOSITORY_DIR -type d -maxdepth 1 -mindepth 1 -not \</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">-iname '*.git' | sed -e "s|$REPOSITORY_DIR||g")</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">for CATEGORY in $CATEGORIES; do</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>echo "<h2>Category: $CATEGORY</h2>"</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>CAT_REPOSITORIES=$(find $REPOSITORY_DIR$CATEGORY -type d -iname \</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">'*.git' | sed -e "s|$REPOSITORY_DIR||g" -e 's/.git$//g')</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>for REPOSITORY in $CAT_REPOSITORIES; do</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>echo "$URL_PREFIX$REPOSITORY<br />"</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>done</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">done</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">ROOT_REPOSITORIES=$(find $REPOSITORY_DIR -type d -maxdepth 1 -mindepth 1 \</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">-iname '*.git' | sed -e "s|$REPOSITORY_DIR||g" -e 's/.git$//g')</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">echo "<h2>Uncategorized Repositories</h2>"</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">for REPOSITORY in $ROOT_REPOSITORIES; do</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>echo "$URL_PREFIX$REPOSITORY<br />"</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">done</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">cat <<DONE</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"></div></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"></body></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"></html></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">DONE</span></div>
<div>
<div style="text-align: left;">
<br /></div>
</div>bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-40736343638849485272012-06-25T12:11:00.001-04:002012-06-25T12:16:58.376-04:00Bash script for pulling/fetching multiple git clones<br />
In my current assignment, I'm acting as the main build guy for a number of projects that use git for source control. As such, I find it very useful to keep all my git clones up to date whether I'm actively developing in them or not. Additionally, I need to review the changes other developers are committing, so I'd like to get a summary of recent git activities.<br />
<br />
Over time, I've put this little bash script together to help me with that. I've included the script in this posting so I can remember later what/why I did this. Disclaimer, I wrote this script and run this script in bash on Linux (not via git-bash in Windows). Also, I'm using Zenity for some nice UI look/feel.<br />
<br />
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 1</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> #!/bin/bash</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 2</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 3</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> pushd ~/dev/repos > /dev/null</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 4</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 5</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> # The log file</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 6</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> PULL_LOG="$(mktemp)"</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 7</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 8</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> # Get a list of all the clones in this directory.</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 9</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> CLONES=$(find -maxdepth 2 -mindepth 2 -type d -name ".git" | sed -e 's|\./||' -e 's|/\.git||')</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 10</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 11</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> # Get a list of all the branches in clone/branch format</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 12</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> ALL_BRANCHES=$(for clone in $CLONES; do cd $clone; for branch in $(git branch -l | sed 's/\s\|\*//g'); do echo $clone/$branch; done; cd ..; done)</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 13</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 14</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> # Count the branches</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 15</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> BRANCH_COUNT=$(echo $ALL_BRANCHES | sed 's/ /\n/g' | wc -l)</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 16</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 17</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> # Start the log file</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 18</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> echo "Pull log for $(date)" >> $PULL_LOG</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 19</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> echo "--------------------------------------------------------------------------------" >> $PULL_LOG</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 20</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 21</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> # Function for pipping output to zenity progress dialog</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 22</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> function pull_clones() {</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 23</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> clone_counter=0</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 24</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> for clone in $CLONES; do</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 25</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> echo "Pulling branches for clone $clone" >> $PULL_LOG</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 26</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> echo "--------------------------------------------------------------------------------" >> $PULL_LOG</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 27</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> cd $clone</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 28</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> echo "# Fetching changes for clone $clone"</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 29</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> git fetch origin 2>> $PULL_LOG</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 30</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> for branch in $(git branch -l | sed 's/\s\|\*//g'); do</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 31</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> echo "# Merging branch $clone/$branch"</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 32</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> echo "Merging branch $branch" >> $PULL_LOG</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 33</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> git checkout $branch 2> /dev/null</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 34</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> git merge origin/$branch >> $PULL_LOG</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 35</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> echo | awk '{print count / total * 100}' count=$clone_counter total=$BRANCH_COUNT</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 36</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> let clone_counter=clone_counter+1</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 37</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> done</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 38</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> cd ..</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 39</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> echo >> $PULL_LOG</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 40</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> done</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 41</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> }</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 42</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 43</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> # Do it</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 44</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> pull_clones | zenity --progress --title='Pulling development clones' --width=512</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 45</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> zenity --text-info --filename=$PULL_LOG --title="Pull log" --width=500 --height=450</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 46</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 47</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> #Clean up</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 48</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> rm $PULL_LOG</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 49</span></div>
<div style="text-align: left;">
<span style="color: red; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> 50</span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> popd > /dev/null</span></div>
<div style="text-align: left;">
<br /></div>
<br />
<br />bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-90071900100591542282012-06-06T10:47:00.003-04:002012-06-06T10:47:55.013-04:00Patching tip using mocks in python unit testsI use the <a href="http://www.voidspace.org.uk/python/mock/" target="_blank">mock</a> library by Michael Foord in my python unit tests and one problem always plagued me. Here's the problem and the solution.<div>
<br /></div>
<div>
Sometimes when I import a package/module in my code I use this pattern (let's call it pattern <b><span style="color: red;">A</span></b>):</div>
<div>
<br /></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">"""<b>file_module_pattern_a</b>.py"""</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">import <b>os</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><br /></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">def get_files(path):</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> """Return list of files"""</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> return <b>os.listdir</b>(path)</span></div>
<div>
<br /></div>
<div>
<br />Other times, I use this pattern (let's call it pattern <b><span style="color: red;">B</span></b>):</div>
<div>
<br /></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">"""<b>file_module_pattern_b</b>.py"""</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">from <b>os</b> import <b>listdir</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">def get_files(path):</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> """Return list of files"""</span></div>
<div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> return <b>listdir</b>(path_variable)</span></div>
</div>
<div>
<br /></div>
<div>
Note the differente. In pattern <b><span style="color: red;">A</span></b>, I import the whole <span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>os</b></span> package, while in pattern <b><span style="color: red;">B</span></b>, I only import the <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">listdir</span></b> function. Now in my unit tests, here's what I use for pattern <b><span style="color: red;">A</span></b>:</div>
<div>
<br /></div>
<div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">"""Unit tests for module file_module_pattern_a"""</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><br /></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><div>
from file_module_pattern_a import get_files</div>
<div>
from unittest import TestCase</div>
<div>
from mock import patch, sentinel</div>
<div>
<br /></div>
</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">class StandloneTests(TestCase):</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> """Test the standalone functions"""</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> </span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> @patch('<b>os.listdir</b>')</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> def test_get_files(self, mock_listdir):</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> """Test the get_files function"""</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> test_result = get_files(sentinel.PATH)</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> mock_listdir.assert_called_once_with(sentinel.PATH)</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> self.assertEqual(test_result, mock_listdir.return_value)</span></div>
</div>
<div>
<br /></div>
<div>
This works great. The only problem is... if I use pattern <b><span style="color: red;">B</span></b> with this unit test, the <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">mock_listdir</span></b> never gets called. The unit test tries to use the REAL <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">os.listdir</span></b> function.</div>
<div>
<br /></div>
<div>
Here's the issue at hand. When I use pattern <b><span style="color: red;">B</span></b>, I'm actually adding the function to my module, not the global scope. As a result, the patch directive needs to reference my module, not <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">os</span></b>. Here's the correct unit test patch syntax:</div>
<div>
<br /></div>
<div>
<div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">"""Unit tests for module file_module_pattern_b"""</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><br /></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><div>
from file_module_pattern_b import get_files</div>
<div>
from unittest import TestCase</div>
<div>
from mock import patch, sentinel</div>
<div>
<br /></div>
</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">class StandloneTests(TestCase):</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> """Test the standalone functions"""</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> </span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> @patch('<b>file_module_pattern_b.listdir</b>')</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> def test_get_files(self, mock_listdir):</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> """Test the get_files function"""</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> test_result = get_files(sentinel.PATH)</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> mock_listdir.assert_called_once_with(sentinel.PATH)</span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> self.assertEqual(test_result, mock_listdir.return_value)</span></div>
</div>
</div>bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-33381365466739827672012-06-04T11:43:00.000-04:002012-06-04T11:47:15.719-04:00Using SSHFS from OSXSSHFS is a FUSE (Filesystem in Userspace) plugin that allows you to mount a drive/filesystem on your systems via SSH. This is a great way to transfer files securely over the public Internet. Best of all, it's free! <br />
<div>
<br /></div>
<div>
OSX supports FUSE through a program called OSXFUSE (or Fuse for OSX). You can download it at:</div>
<div>
<br /></div>
<div style="text-align: center;">
<a href="http://osxfuse.github.com/">http://osxfuse.github.com/</a></div>
<div>
<br /></div>
<div>
Note, you'll have to download the plugins for OSXFUSE separately. The two most popular are: SSHFS and NTFS-3G (a plugin that allows you to mount NFTS volumes in read/write mode... which I will not be covering in this post).</div>
<div>
<br /></div>
<div>
Computers hosting sshfs directories don't need to have any special software installed on them. They simply need to be running sshd (the secure shell daemon) and not have file transfers disabled. Usually, sshd is configured by default to have file transfers enabled.</div>
<div>
<br /></div>
<div>
One thing that often confuses people coming from the Mac world is that SSHFS has no GUI. So how do you mount remote drives? Well, you'd normally do it from a terminal prompt.</div>
<div>
<br /></div>
<div>
In this blog entry, I will go over three ways to mount sshfs drives/filesystems using SSHFS:</div>
<div>
<ol>
<li>Using terminal.app and some bash commands</li>
<li>Using Automator</li>
<li>By creating an Alfred.app extension</li>
</ol>
<div>
<br /></div>
<h3>
Get the software</h3>
</div>
<div>
Before we begin, first make sure you have the OSXFUSE software installed. Download it from the link listed above. You'll know you have it properly installed when you see it in your System Preferences:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-SaeWulVffJg/T8y88uoIeJI/AAAAAAAAAMM/Nfnt3NMdHZ4/s1600/System+Preferences.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="276" src="http://2.bp.blogspot.com/-SaeWulVffJg/T8y88uoIeJI/AAAAAAAAAMM/Nfnt3NMdHZ4/s320/System+Preferences.png" width="320" /></a></div>
<div>
You'll also want to ensure that you've installed the SSHFS plugin from the same site. To verify that you've installed it correctly, you'll need to run <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">sshfs -h</span></b> from a terminal window:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-KfQL_6n8Gd4/T8y9uoIwiKI/AAAAAAAAAMU/hKTTntZZp8w/s1600/sshfs+-h.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="http://2.bp.blogspot.com/-KfQL_6n8Gd4/T8y9uoIwiKI/AAAAAAAAAMU/hKTTntZZp8w/s320/sshfs+-h.png" width="217" /></a></div>
<div>
If you've got these two items installed, you're ready to go!</div>
<div>
<br /></div>
<h3>
First way: From the bash prompt</h3>
<div>
This method is the most traditional way to mount a sshfs filesystem... and actually the other two methods will be invoking the same commands...we'll just be hiding them with the graphical interface.</div>
<div>
<br /></div>
<div>
First start a terminal window and create a directory to house the filesystem. I always put my filesystems in the <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">/Volumes</span></b> directory. You aren't required to put your directory there. You can put it anywhere you'd like. I just do so out of convention. Here's the command to do so:</div>
<div>
<br /></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@l00-1nsv01 $ <b>mkdir /Volumes/ssh_fs_mount</b></span></div>
<div>
<br /></div>
<div>
I named my mount <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">ssh_fs_mount</span></b>. You can name yours anything you'd like.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-zCLXEhBV9fE/T8y_aVk_ZdI/AAAAAAAAAMc/c7lDD-qEZSo/s1600/mkdir.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="http://2.bp.blogspot.com/-zCLXEhBV9fE/T8y_aVk_ZdI/AAAAAAAAAMc/c7lDD-qEZSo/s320/mkdir.png" width="320" /></a></div>
<div>
Then you need to run the <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">sshfs</span></b> command to mount the remote computer's directory to your hosting directory. The command takes the following format:</div>
<div>
<br /></div>
<div style="text-align: center;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>sshfs username@hostname:/path/to/directory /local/directory</b></span></div>
<div>
<br /></div>
<div>
...where <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">username</span></b> is your username on the remote computer, <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">hostname</span></b> is the name of the remote computer contains the directory that you'd like to mount, <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">/path/to/directory</span></b> is the full path to the directory on the remote computer that you'd like to mount and <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">/local/directory</span></b> is the directory on your local computer that you'd like to house the mount on. Maybe an example will make it more clear.</div>
<div>
<br /></div>
<div>
I have a Linux computer that I do a lot of development on. It's called <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">ubuntu64.local.</span></b> My username on that computer it <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve</span></b> and my development folder on that computer is at <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">/home/steve/dev</span></b>. Here's the sshfs command that I would issue to mount my development folder from the Linux computer into the <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">/Volumes/ssh_fs_mount</span></b> folder on my Mac</div>
<div>
<br /></div>
<div style="text-align: center;">
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">sshfs steve@ubuntu64.local:/home/steve/dev /Volumes/ssh_fs_mount</span></b></div>
<div>
<br /></div>
<div>
Let's try it real quick:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-g1AyQ3WL6uY/T8zCELdYd3I/AAAAAAAAAMo/GFeILkUo4Js/s1600/terminal1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="http://2.bp.blogspot.com/-g1AyQ3WL6uY/T8zCELdYd3I/AAAAAAAAAMo/GFeILkUo4Js/s320/terminal1.png" width="320" /></a></div>
<div>
A couple of items to note... First, I'm prompted for a password. This is prompting me for my password on the <b>remote computer</b> (ubuntu64.local in this instance). It's <b>not</b> prompting me for my Mac password. This is the remote computer verifying that I'm indeed steve@ubuntu64.local and not some other impostor. If I have key-based authentication working between my Mac account and my Linux account, I would not be prompted for the password. I normally do have key-based authentication between these two accounts, but I turned it off for this demonstration. If you're not using key-based authentication when ssh'ing between computers... you should. In fact, the other two methods in this blog entry (using Automator and the Alfred extension) assume you are using key-based authentication and aren't prompted for a password. You can find out more about setting-up key-based authentication at any of these links:</div>
<div>
<br /></div>
<div style="text-align: center;">
<a href="https://www.google.com/#sclient=psy-ab&hl=en&q=ssh+key-based+authentication&oq=ssh+key-based+authentication&aq=f&aqi=g1g-K3&aql=&gs_l=hp.3..0j0i30l3.1071.5243.0.5877.30.29.1.0.0.0.222.3587.10j17j2.29.0...0.0.GCevakOUaRQ&pbx=1&bav=on.2,or.r_gc.r_pw.,cf.osb&fp=89c2cc531d5730ec&biw=1277&bih=1039" target="_blank">Google Search</a></div>
<div>
<br /></div>
<div>
Second, there's no other output from the command. No output means that sshfs was able to successfully complete the mount. If there had been a problem, sshfs would have complained with error messages. Once the mount is completed, the drive should appear on the desktop:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-tvubAdDeU4o/T8zDLdbb-QI/AAAAAAAAAMw/yVHGsosw0Yc/s1600/drive+on+desktop.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-tvubAdDeU4o/T8zDLdbb-QI/AAAAAAAAAMw/yVHGsosw0Yc/s1600/drive+on+desktop.png" /></a></div>
<div>
<br /></div>
<div>
It should behave like a normal Mac drive. You can get info on it. You can browse it in Finder. You can even add/edit/remote files (assuming your account on the remote computer has the appropriate privileges).</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-VW4PfFySz9w/T8zD1OWAFXI/AAAAAAAAAM4/UkmzfF5F-M8/s1600/browse+in+finder.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="199" src="http://2.bp.blogspot.com/-VW4PfFySz9w/T8zD1OWAFXI/AAAAAAAAAM4/UkmzfF5F-M8/s320/browse+in+finder.png" width="320" /></a></div>
<div>
When you're done and want to unmount the drive, simply right-click on it and select <b><span style="color: blue;">Eject</span></b>.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-g7W4-h-nBnQ/T8zEaIagGYI/AAAAAAAAANA/xA5nzTWMOnQ/s1600/eject.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="http://2.bp.blogspot.com/-g7W4-h-nBnQ/T8zEaIagGYI/AAAAAAAAANA/xA5nzTWMOnQ/s320/eject.png" width="317" /></a></div>
<div>
<br /></div>
<div>
Note, when the SSHSFS volume has been ejected, OSX automatically deletes the <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">/Volumes/ssh_fs_mount</span></b> directory. If you put your mounting directories in <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">/Volumes</span></b>, you'll always have to recreate them after ejecting.</div>
<div>
<br /></div>
<h3>
Second Way: Create an Automator task</h3>
<div>
Using Automator, you can create a workflow to automate the steps you did in the first method. After all, that's what Automator is for.. automating repetitive things.</div>
<div>
<br /></div>
<div>
First, start Automator and create a new <b><span style="color: blue;">Application</span></b> workflow:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-LuobVmNo60E/T8zLBMnmCHI/AAAAAAAAANM/xXiUeemjKr0/s1600/Automator+New+Workflow.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="274" src="http://1.bp.blogspot.com/-LuobVmNo60E/T8zLBMnmCHI/AAAAAAAAANM/xXiUeemjKr0/s320/Automator+New+Workflow.png" width="320" /></a></div>
<div>
Next, in the <b><span style="color: blue;">Text</span></b> Library, drag the <b><span style="color: blue;">Ask For Text</span></b> action to the workflow.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-EeOr4ZnJoos/T8zLnhPRoyI/AAAAAAAAANU/f503NRm8ClM/s1600/Ask+For+Text.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="274" src="http://3.bp.blogspot.com/-EeOr4ZnJoos/T8zLnhPRoyI/AAAAAAAAANU/f503NRm8ClM/s320/Ask+For+Text.png" width="320" /></a></div>
<div>
Check the <b><span style="color: blue;">Ignore this action's input</span></b> and <b><span style="color: blue;">Require an answer</span></b> checkboxes. Enter <b><span style="color: blue;">Enter a remote sshfs url and click OK</span></b> in the question textbox.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-UZOXnJIg6So/T8zMl0IwT4I/AAAAAAAAANc/OGnwrhBPs1I/s1600/Ask+for+Text+2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="274" src="http://2.bp.blogspot.com/-UZOXnJIg6So/T8zMl0IwT4I/AAAAAAAAANc/OGnwrhBPs1I/s320/Ask+for+Text+2.png" width="320" /></a></div>
<div>
Now, from the <b><span style="color: blue;">Utilities</span></b> library, drag the <b><span style="color: blue;">Run Shell Script</span></b> action to the workflow and drop it below the <b><span style="color: blue;">Ask for Text</span></b> action.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-FSLSSX7rUW0/T8zNPSIntAI/AAAAAAAAANk/fIK-RdrJ7qM/s1600/Run+Shell+Script.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="274" src="http://3.bp.blogspot.com/-FSLSSX7rUW0/T8zNPSIntAI/AAAAAAAAANk/fIK-RdrJ7qM/s320/Run+Shell+Script.png" width="320" /></a></div>
<div>
Change the <b><span style="color: blue;">Pass input</span></b> pulldown from <b><span style="color: blue;">to stdin</span></b> to <b><span style="color: blue;">as arguments</span></b>. Then past the following text into the textarea:</div>
<div>
<br /></div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>volume_name=/Volumes/sshfs_volume_$$</b></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>mkdir $volume_name</b></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>/usr/local/bin/sshfs $1 $volume_name</b></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>if [ "$?" == "0" ]; then</b></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>open $volume_name</b></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>else</b></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>echo "Unable to mount sshfs volume."</b></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>rmdir $volume_name</b></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>fi</b></span></div>
</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Save it to your desktop as <b><span style="color: blue;">SSHFS Workflow</span></b>. It should now look like this:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-8AipcY6GbDI/T8zO7rpxO-I/AAAAAAAAAN0/xuq4jPZNjWU/s1600/Final+Workflow.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="274" src="http://4.bp.blogspot.com/-8AipcY6GbDI/T8zO7rpxO-I/AAAAAAAAAN0/xuq4jPZNjWU/s320/Final+Workflow.png" width="320" /></a></div>
<div>
Now, if you run it from your desktop, it will prompt you for a sshfs url. The format is the same from the bash method. I'll reuse my example:</div>
<div>
<br /></div>
<div style="text-align: center;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>steve@ubuntu64.local:/home/steve/dev</b></span></div>
<div>
<br /></div>
<div>
Again, like I mentioned before, this method assumes that you've already set up key-based authentication between your Mac account and your remote account.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-WqKEekvAmHM/T8zP0YaASYI/AAAAAAAAAN8/stSbQ-YH2D8/s1600/workflow+running.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="158" src="http://2.bp.blogspot.com/-WqKEekvAmHM/T8zP0YaASYI/AAAAAAAAAN8/stSbQ-YH2D8/s320/workflow+running.png" width="320" /></a></div>
<div>
If all goes well, the automator will automatically open the mounted volume. Just like the first method, when you're done, you simply right-click the volume and eject it via Finder.</div>
<div>
<br /></div>
<h3>
Third Way: Create an Alfred Extension</h3>
<div>
Now if you're running Alfred, you're probably thinking, "I can just run the automator task from Alfred and be done with it." Yes. You could. But you can also create an extension. That way you can enter the sshfs url directly into the Alfred window and save yourself one last data-entry step.</div>
<div>
<br /></div>
<div>
To create the extension go to the <b><span style="color: blue;">Extensions</span></b> tab of the Alfred preferences window. Click the <b><span style="color: blue;">+</span></b> and select <b><span style="color: blue;">Shell Script</span></b>.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-fMsCAzFMimY/T8zTLvWH6HI/AAAAAAAAAOI/9t54w5NuD-Y/s1600/Alfred+Prefs.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="251" src="http://1.bp.blogspot.com/-fMsCAzFMimY/T8zTLvWH6HI/AAAAAAAAAOI/9t54w5NuD-Y/s320/Alfred+Prefs.png" width="320" /></a></div>
<div>
In the <b>Extension Name</b> field, enter <b><span style="color: blue;">SSHFS</span></b> and click <b><span style="color: blue;">Create</span></b>.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-yi8VD8mLStA/T8zTnVjYctI/AAAAAAAAAOQ/_HfSXy9F_6I/s1600/Extension+name.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="251" src="http://2.bp.blogspot.com/-yi8VD8mLStA/T8zTnVjYctI/AAAAAAAAAOQ/_HfSXy9F_6I/s320/Extension+name.png" width="320" /></a></div>
<div>
In the <b>Title</b> field, enter <b><span style="color: blue;">SSHFS Mounter</span></b>. In the <b>Description</b> field, enter <b><span style="color: blue;">Mount a sshfs volume</span></b>. Make sure the <b>Keyword</b> checkbox is checked and the enter <b><span style="color: blue;">sshfs</span></b> in the textbox next to it. Make sure the <b>Silent</b> checkbox is checked. Finally in the <b>Command</b> field, enter the following text:</div>
<div>
<br /></div>
<div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">volume_name=/Volumes/sshfs_volume_$$</span></b></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">mkdir $volume_name</span></b></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">/usr/local/bin/sshfs {query} $volume_name</span></b></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">if [ "$?" == "0" ]; then</span></b></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>open $volume_name</span></b></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">else</span></b></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>echo "Unable to mount sshfs volume."</span></b></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>rmdir $volume_name</span></b></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">fi</span></b></div>
</div>
<div>
<br /></div>
<div>
If should look like this:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-ZTdShBuqkVo/T8zVUfK2mWI/AAAAAAAAAOY/AK9JYPs93eI/s1600/extension+settings.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="251" src="http://1.bp.blogspot.com/-ZTdShBuqkVo/T8zVUfK2mWI/AAAAAAAAAOY/AK9JYPs93eI/s320/extension+settings.png" width="320" /></a></div>
<div>
If you have Growl installed on your Mac, click the <b>Advanced</b> button and check the <b>Display script output in Growl</b> checkbox.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-vTHm_6VORVE/T8zVufxPANI/AAAAAAAAAOg/BaJpKINgsmo/s1600/turn+on+growl.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="251" src="http://1.bp.blogspot.com/-vTHm_6VORVE/T8zVufxPANI/AAAAAAAAAOg/BaJpKINgsmo/s320/turn+on+growl.png" width="320" /></a></div>
<div>
Save the extension and close the preferences window. Now from the the Alfred prompt, you can enter <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">sshfs your_sshfs_url</span></b> and hit enter, where <b>your_sshfs_url</b> is an actual sshfs url. Here's me mounting my ubuntu64.local system's tmp directory:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-4IkHO1Tfr1M/T8zWkmPSwWI/AAAAAAAAAOo/nw7LNPY61yk/s1600/alfred+sshfs.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="133" src="http://3.bp.blogspot.com/-4IkHO1Tfr1M/T8zWkmPSwWI/AAAAAAAAAOo/nw7LNPY61yk/s320/alfred+sshfs.png" width="320" /></a></div>
<div>
<br /></div>
<div>
Just like the Automator method, the Alfred method presumes you are using key-based authentication between your Mac account and the remote computer's account. It should automatically open the folder you mounted. When you're done, simply right-click on the volume and eject it.</div>bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-13092546080550437752012-05-30T10:53:00.000-04:002012-05-30T10:53:06.474-04:00Run multiple python versions on your systemI'm a software development consultant. I write python (as well as other languages) code for many clients and I don't get to dictate what their environment looks like. I've got clients running python as old as 2.4 while others are on the bleeding edge. Additionally, each client may have their own packages installed as well as differing lists of third party packages.<br />
<br />
This post is a description of how I went about getting multiple versions of python installed in my ubuntu development machine and how I go about managing different package sets for different clients.<br />
<br />
<h3>
Get multiple pythons installed</h3>
Ubuntu typically only supports one python 2.x version and one 3.x version at a time. There's a popular ppa (personal package archive) called deadsnakes that contains older versions of python. You can find it:<br />
<br />
<div style="text-align: center;">
<a href="https://launchpad.net/~fkrull/+archive/deadsnakes">https://launchpad.net/~fkrull/+archive/deadsnakes</a></div>
<br />
To install it (per the instructions in the link above), you do the following:<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">steve@ubuntu64 ~ $ <b>sudo add-apt-repository ppa:fkrull/deadsnakes</b></span><br />
<br />
Then you need to update your cache:<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">steve@ubuntu64 ~ $ <b>sudo apt-get update</b></span><br />
<br />
Finally, simply install the other versions (I'm running on ubuntu 12.04 LTS, so I have python 2.7 already):<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">steve@ubuntu64 ~ $ <b>sudo apt-get install python2.4 python2.5 python2.6</b></span><br />
<br />
If you're following along, we now have python versions 2.4 through 2.7 installed the computer. If you run 'python', you'll see the default version is still 2.7.<br />
<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">steve@ubuntu64 ~ $ <b>python</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Python <b>2.7.3</b> (default, Apr 20 2012, 22:39:59) </span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">[GCC 4.6.3] on linux2</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Type "help", "copyright", "credits" or "license" for more information.</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">>>></span><br />
<br />
Here's why: Each python version is stored in <b>/usr/bin</b> as <b>python2.X</b> where X is the version. There is a symbolic link named <b>python</b> that points to the version you want to be the default. Instead of typing <b>python</b> from the bash prompt, you could just as easily type <b>python2.5</b>:<br />
<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">steve@ubuntu64 ~ $ <b>ls -l /usr/bin/python*</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">lrwxrwxrwx 1 root root 9 Apr 17 13:20 /usr/bin/python -> python2.7</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">lrwxrwxrwx 1 root root 9 Apr 17 13:20 /usr/bin/python2 -> python2.7</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">-rwxr-xr-x 1 root root 1216520 May 21 12:13 /usr/bin/python2.4</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">-rwxr-xr-x 1 root root 1403624 May 3 00:17 /usr/bin/python2.5</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">-rwxr-xr-x 1 root root 2652056 May 12 08:43 /usr/bin/python2.6</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">-rwxr-xr-x 1 root root 2993560 Apr 20 19:37 /usr/bin/python2.7</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">steve@ubuntu64 ~ $ <b>python2.5</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Python <b>2.5.6</b> (r256:88840, May 3 2012, 04:16:14) </span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">[GCC 4.6.3] on linux2</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Type "help", "copyright", "credits" or "license" for more information.</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">>>> </span><br />
<div>
<br /></div>
<br />
This is a point of interest. I would not mess with the symbolic link. Ubuntu runs python for many internal maintenance scripts and those scripts are expecting the python version that shipped with ubuntu.<br />
<br />
<h3>
Use virtualenv to manage your python installations and package sets</h3>
So now that you have multiple versions of python on your system, how to manage them? How do you keep packages installed for one version separate from packages installed for another. What if you want to run one version of django for client X and a different version for client Y?<br />
<br />
That's where virtualenv comes in. If your a ruby programmer, this is analogous to rvm. Virtualenv lets you manage the python versions and package installations separately for different projects or clients.<br />
<br />
<h3>
Installing virtualenv is simple</h3>
As always, you're just a single apt-get command away from having virtualenv ready to go:<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">steve@ubuntu64 ~ $ <b>sudo apt-get install python-virtualenv</b></span><br />
<br />
That it. Virtualenv is ready to go now.<br />
<br />
<h3>
Quick example</h3>
Say your starting a new project for a client. They are running python2.5 and want to use the mocks, nose and coverage packages for testing. Here a walkthrough of how to use virtualenv to manage the project.<br />
<br />
First, let's create a directory for the project:<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">steve@ubuntu64 ~ $ <b>mkdir -p ~/dev/project1</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">steve@ubuntu64 ~ $ <b>cd ~/dev/project1</b></span><br />
<br />
Next, run virtualenv to create the environment for the project:<br />
<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">steve@ubuntu64 ~/dev/project1 $ <b>virtualenv -p /usr/bin/python2.5 .env</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Running virtualenv with interpreter /usr/bin/python2.5</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">New python executable in .env/bin/python2.5</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Also creating executable in .env/bin/python</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Installing distribute.............................................................................................................................................................................................done.</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Installing pip...............done.</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">steve@ubuntu64 ~/dev/project1 $ </span><br />
<br />
This command tells virtualenv to create a <b>.env</b> directory and to place a copy of the <b>2.5</b> version of python in it. This copy of the 2.5 python is brand-spankin' new. It doesn't have any packages (beyond the standard library) installed. You will need to install them yourself. Any packages that you install in this instance of python will not be available to main python installation or other virtualenv instances.<br />
<br />
Before your can use this new copy, you need to activate it:<br />
<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">steve@ubuntu64 ~/dev/project1 $ <b>source .env/bin/activate</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">(.env)steve@ubuntu64 ~/dev/project1 $</span><br />
<br />
The activate script manipulates your path environment variable, placing the new python instance first in your path. This makes is so that when you run python, it will use the version from your instance:<br />
<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">(.env)steve@ubuntu64 ~/dev/project1 $ <b>which python</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">/home/steve/dev/project1/.env/bin/python</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">(.env)steve@ubuntu64 ~/dev/project1 $ <b>python</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Python <b>2.5.6</b> (r256:88840, May 3 2012, 04:16:14) </span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">[GCC 4.6.3] on linux2</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Type "help", "copyright", "credits" or "license" for more information.</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">>>></span><br />
<br />
Also, notice that your prompt starts with <b>(.env)</b>. This tells you that you're running with the virtualenv instance activated. To install packages in your instance, use the <b>pip</b> command:<br />
<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">(.env)steve@ubuntu64 ~/dev/project1 $ <b>pip install mock nose coverage</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Downloading/unpacking mock</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> Downloading mock-0.8.0.tar.gz (749Kb): 749Kb downloaded</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> Running setup.py egg_info for package mock</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> warning: no files found matching '*.png' under directory 'docs'</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> warning: no files found matching '*.css' under directory 'docs'</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> warning: no files found matching '*.html' under directory 'docs'</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> warning: no files found matching '*.js' under directory 'docs'</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Downloading/unpacking nose</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> Downloading nose-1.1.2.tar.gz (729Kb): 729Kb downloaded</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> In the tar file /tmp/pip-dH_WYa-unpack/nose-1.1.2.tar.gz the member nose-1.1.2/doc/doc_tests/test_selector_plugin/support/tests/mymodule/my_function$py.class is invalid: 'filename None not found'</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> In the tar file /tmp/pip-dH_WYa-unpack/nose-1.1.2.tar.gz the member nose-1.1.2/doc/doc_tests/test_restricted_plugin_options/restricted_plugin_options.rst.py3.patch is invalid: 'filename None not found'</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> Running setup.py egg_info for package nose</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Downloading/unpacking coverage Downloading coverage-3.5.2.tar.gz (115Kb): 115Kb downloaded</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> Running setup.py egg_info for package coverage</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> no previously-included directories found matching 'test'Installing collected packages: mock, nose, coverage</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> Running setup.py install for mock</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> warning: no files found matching '*.png' under directory 'docs'</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> warning: no files found matching '*.css' under directory 'docs'</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> warning: no files found matching '*.html' under directory 'docs'</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> warning: no files found matching '*.js' under directory 'docs'</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> Running setup.py install for nose</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> Installing nosetests script to /home/steve/dev/project1/.env/bin</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> Installing nosetests-2.5 script to /home/steve/dev/project1/.env/bin</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> Running setup.py install for coverage</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> building 'coverage.tracer' extension</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c coverage/tracer.c -o build/temp.linux-x86_64-2.5/coverage/tracer.o</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> coverage/tracer.c:3:20: fatal error: Python.h: No such file or directory</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> compilation terminated.</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> **</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> ** Couldn't install with extension module, trying without it...</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> ** SystemExit: error: command 'gcc' failed with exit status 1</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> **</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> no previously-included directories found matching 'test'</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> Installing coverage script to /home/steve/dev/project1/.env/bin</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Successfully installed mock nose coverage</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Cleaning up...</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">(.env)steve@ubuntu64 ~/dev/project1 $</span><br />
<br />
To see that the packages have been installed, simply use them:<br />
<br />
<br />
<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">(.env)steve@ubuntu64 ~/dev/project1 $ <b>python</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Python <b>2.5.6</b> (r256:88840, May 3 2012, 04:16:14) </span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">[GCC 4.6.3] on linux2</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Type "help", "copyright", "credits" or "license" for more information.</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">>>> <b>import mock</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">>>> <b>import nose</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">>>> <b>import coverage</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">>>></span><br />
<br />
When you're done working on the project, deactivate it. You can always come back later and activate it again.<br />
<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">(.env)steve@ubuntu64 ~/dev/project1 $ <b>deactivate</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">steve@ubuntu64 ~/dev/project1 $</span><br />
<br />
<br />
Notice that when you deactivate the environment, your packages are no longer available:<br />
<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">steve@ubuntu64 ~/dev/project1 $ <b>python</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Python <b>2.7.3</b> (default, Apr 20 2012, 22:39:59) </span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">[GCC 4.6.3] on linux2</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Type "help", "copyright", "credits" or "license" for more information.</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">>>> <b>import mock</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Traceback (most recent call last):</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> File "<stdin>", line 1, in <module></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">ImportError: No module named mock</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">>>> <b>import nose</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Traceback (most recent call last):</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> File "<stdin>", line 1, in <module></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">ImportError: No module named nose</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">>>> <b>import coverage</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Traceback (most recent call last):</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"> File "<stdin>", line 1, in <module></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">ImportError: No module named coverage</span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">>>></span><br />
<br />
You can create as many virtualenv environments as you like. I create one for each project that I work on.<br />
<br />
<br />
<h3>
Bonus material</h3>
<br />
To make life even easier, here's a couple additional things I do that you might find helpful!<br />
<br />
First, once you've started to use virtualenv with some frequency, you start to get tired of downloading and installing the same packages over and over. Pip has the ability to cache your downloaded packages for reuse. To do that, you'll need to create a directory to store the download packages in:<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">steve@ubuntu64 ~ $ <b>mkdir ~/.pip_download_cache</b></span><br />
<br />
Then you'll need to set a variable to inform pip of the new directory. Add the following to your .bashrc file:<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b>export PIP_DOWNLOAD_CACHE=/home/steve/.pip_download_cache</b></span><br />
<br />
Now when you do a pip install, it will keep the downloaded files in the ~/.pip_download_cache directory. The next time you do a pip install of the same package, it will just use the copy from the directory instead of downloading it again.<br />
<br />
<br />
Second, it can be tedious to always have to type 'source .env/bin/activate' every time you want to activate an environment. Since I always put my virtual environments in a .env directory I can count on the command to activate always being the same. So I create an alias for it. I added the the following to my ~/.bash_aliases file:<br />
<br />
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">alias activate='source .env/bin/activate'</span></b><br />
<br />
Now once I <b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">cd</span></b> into the projects directory, I simply type <b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">activate</span></b> to activate my virtual environment.<br />
<br />
<br />
<br />
<br />
<br />
<br />bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-76290675425566675442012-05-25T10:05:00.002-04:002012-05-25T20:41:25.568-04:00File encryption with vimYou may not know it, but you can encrypt your files using vim... and it's pretty easy to do.<br />
<br />
<h3>
Turning encryption on for a file</h3>
If you're editing a file, to encrypt it, simply enter <b><span style="color: blue;">X</span></b> at the <b><span style="color: blue;">:</span></b> prompt.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-n6dJTRGyrjI/T7-LaVLWgyI/AAAAAAAAALI/JHA9eGLOQ3g/s1600/x_prompt.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="http://2.bp.blogspot.com/-n6dJTRGyrjI/T7-LaVLWgyI/AAAAAAAAALI/JHA9eGLOQ3g/s320/x_prompt.png" width="320" /></a></div>
Vim will then prompt you for a key. A key is the password for decrypting the file in the future. You'll have to enter it twice to ensure you typed it correctly.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-jYhdeoSVlpY/T7-MKOj6drI/AAAAAAAAALQ/eKOgNrc-Yak/s1600/key_prompt.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="http://4.bp.blogspot.com/-jYhdeoSVlpY/T7-MKOj6drI/AAAAAAAAALQ/eKOgNrc-Yak/s320/key_prompt.png" width="320" /></a></div>
Once you've entered your key, the next time you save/write, the file should be encrypted, see the status text at the bottom to be sure.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-jEbThBBMR3Y/T7-MhYgb8eI/AAAAAAAAALY/pZJ35wn6lnY/s1600/crypted.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="236" src="http://4.bp.blogspot.com/-jEbThBBMR3Y/T7-MhYgb8eI/AAAAAAAAALY/pZJ35wn6lnY/s320/crypted.png" width="320" /></a></div>
Once the file is encrypted, it will be unreadable to others.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-vd8FhiS86yY/T7-NBmmnMyI/AAAAAAAAALg/S74u-6poBu4/s1600/encrypted_file.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="http://3.bp.blogspot.com/-vd8FhiS86yY/T7-NBmmnMyI/AAAAAAAAALg/S74u-6poBu4/s320/encrypted_file.png" width="320" /></a></div>
Vim auto-detects whether files are encrypted. If you try to open an encrypted file, vim will prompt you for the key.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-vkoMW0VNSV0/T7-NaSrLqzI/AAAAAAAAALo/FwuOxRBegDk/s1600/open_key_prompt.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="http://4.bp.blogspot.com/-vkoMW0VNSV0/T7-NaSrLqzI/AAAAAAAAALo/FwuOxRBegDk/s320/open_key_prompt.png" width="320" /></a></div>
Vim uses whatever you enter for the key to decrypt the file. If you enter the correct things will look great. If you enter an invalid key, vim will present you with garbage to edit.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-DS2ths_lvT0/T7-N6OIyciI/AAAAAAAAALw/xeZFgqEdcC8/s1600/garbage.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="http://4.bp.blogspot.com/-DS2ths_lvT0/T7-N6OIyciI/AAAAAAAAALw/xeZFgqEdcC8/s320/garbage.png" width="320" /></a></div>
The encryption will stay on for the file until you tell vim you'd like to remove it.<br />
<br />
Be mindful of your key. If you forget it, you will not be able to retrieve the contents of your file.<br />
<br />
<h3>
Turning off encryption for a file</h3>
Turning encryption off is equally as easy. Simply set the <b><span style="color: blue;">key</span></b> setting to nothing. With the command:<br />
<br />
<div style="text-align: center;">
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">:set key=</span></b></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-s5YtQOmzLWY/T7-O5TVzgoI/AAAAAAAAAL4/-uuAYdKc4wQ/s1600/set_key_blank.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="http://2.bp.blogspot.com/-s5YtQOmzLWY/T7-O5TVzgoI/AAAAAAAAAL4/-uuAYdKc4wQ/s320/set_key_blank.png" width="320" /></a></div>
Be sure to write the file. Note that it is no longer encrypted (status text)<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-BEbS25Ua9VY/T7-PMLrc4jI/AAAAAAAAAMA/S2vdLoDzps0/s1600/write_decrypted.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="http://2.bp.blogspot.com/-BEbS25Ua9VY/T7-PMLrc4jI/AAAAAAAAAMA/S2vdLoDzps0/s320/write_decrypted.png" width="320" /></a></div>
<h3>
Wait! Before you go</h3>
As of vim 7.3, vim uses one of two encryption methods: zip and blowfish. Zip is the same encryption that PkZip uses and is somewhat weak (can be cracked). Conversely, blowfish is more contemporary and is much harder to crack. Zip, unfortunately, is the default encryption method in vim. I would strongly suggest that you use blowfish when encrypting your files. To do this is simple enough. Simply enter the command:<br />
<br />
<div style="text-align: center;">
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">:set cm=blowfish</span></b></div>
<br />
I put it in my .vimrc file so it's always set.bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-35105991364261235092012-05-24T11:00:00.001-04:002012-05-25T20:42:26.633-04:00Some light string manipulation in bashThis post is more for me than you. I had to do a little bash work this morning and thought I'd keep a record of some of the syntax and concepts that I used in case I have to do more in the future.<br />
<br />
There are two files here: a function library and a script that consumes it. The function library is used to do different kinds of string justification (left justify, right justify and centering). Kind of like 'echo' on steroids. This could have easily been done in python, perl or ruby, but I wanted it to stay in bash this time to keep the calling scripts uniform.<br />
<br />
Here's the sample file that consumes the library:<br />
<br />
<div style="text-align: left;">
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">#!/bin/bash</span></b></div>
<div style="text-align: left;">
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></b></div>
<div style="text-align: left;">
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">. echo_helper</span></b></div>
<div style="text-align: left;">
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></b></div>
<div style="text-align: left;">
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">x='Steve is here!'</span></b></div>
<div style="text-align: left;">
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">left_justify "$x" 40 .</span></b></div>
<div style="text-align: left;">
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">right_justify "$x" 40 .</span></b></div>
<div style="text-align: left;">
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">center_justify "$x" 40 .</span></b></div>
<div style="text-align: left;">
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></b></div>
<div style="text-align: left;">
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">echo $(pad_string 40 -)</span></b></div>
<div style="text-align: left;">
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></b></div>
<div style="text-align: left;">
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">x='Steve was here.'</span></b></div>
<div style="text-align: left;">
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">left_justify "$x" 40 .</span></b></div>
<div style="text-align: left;">
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">right_justify "$x" 40 .</span></b></div>
<div style="text-align: left;">
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">echo $(center_justify "$x" 40 .)</span></b></div>
<div style="text-align: left;">
<br /></div>
<div>
And here's what the output should look like:</div>
<div>
<br /></div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Steve is here!..........................</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">..........................Steve is here!</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">.............Steve is here!.............</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">----------------------------------------</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">Steve was here..........................</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">.........................Steve was here.</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">............Steve was here..............</span></div>
</div>
<div style="text-align: left;">
<br /></div>
<div>
Finally, here's the <b><span style="color: blue;">echo_helper</span></b> library code:<br />
<span style="font-size: x-small;"><br /></span><br />
<span style="font-size: x-small;"><b style="color: blue; font-family: 'Courier New', Courier, monospace; text-align: left;"></b></span><br />
<div style="display: inline !important;">
<b style="color: blue; font-family: 'Courier New', Courier, monospace; text-align: left;"><span style="font-size: x-small;">#!/bin/bash</span></b></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b></b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><br /></b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b># Syntaxes to remember:</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b>#</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b># String length of a variable: ${#varname}</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b># String offsetting of a variable: ${varname:offset} or ${varname:offset:length}</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><br /></b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b># Usage: pad_string LENGTH {PADDING_CHARACTER=' '}</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b>function pad_string {</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>local line=''</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>local length=$1</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>local padding_character=' '</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if [[ "$2" != "" ]]; then</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>padding_character=$2</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>fi</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>while((${#line} < $length)); do</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>line="$line$padding_character"</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>done</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>echo -n "$line"</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b>}</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><br /></b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b># Usage: left_justify MESSAGE LENGTH {PADDING_CHARACTER=' '}</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b>function left_justify {</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>local message=$1</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>local length=$2</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>local padding_character=' '</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if [[ "$3" != "" ]]; then</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>padding_character=$3</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>fi</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>echo "$message$(pad_string $length-${#message} $padding_character)"</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b>}</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><br /></b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b># Usage: right_justify MESSAGE LENGTH {PADDING_CHARACTER=' '}</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b>function right_justify {</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>local message=$1</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>local length=$2</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>local padding_character=' '</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if [[ "$3" != "" ]]; then</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>padding_character=$3</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>fi</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>echo "$(pad_string $length-${#message} $padding_character)$message"</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b>}</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><br /></b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b># Usage: center_justify MESSAGE LENGTH {PADDING_CHARACTER=' '}</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b>function center_justify {</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>local message=$1</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>local length=$2</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>local padding_character=' '</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if [[ "$3" != "" ]]; then</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>padding_character=$3</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>fi</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>local half_length=$(expr $length / 2)</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>local half_message_length=$(expr ${#message} / 2)</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>local padding_length=$(expr $half_length - $half_message_length)</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>padding="$(pad_string $padding_length $padding_character)"</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>message="$padding$message$padding"</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if [[ ${#message} > $length ]]; then # handle even/odd length issue</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>echo "${message:1}"</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>else</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>echo "$message"</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>fi</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b>}</b></span><br />
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b><br /></b></span></div>
</div>
</div>bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-84591111619047789092012-05-23T14:25:00.002-04:002012-05-23T14:25:10.698-04:00QuickLook from vimHere's another nifty trick I came up with. I'm probably not the first to think of it but I'll put it out there anyway.<br />
<br />
Quicklook is a neat feature of OSX (versions since Leopard) that allows you to preview the contents of a file without having to open it. The way I've typically used it is to select a file in finder and hit the spacebar. OSX then opens a nicely formated window with the contents of my selected file displayed.<br />
<br />
I keep a lot of my documentation, notes, etc in <a href="http://daringfireball.net/projects/markdown/" target="_blank">markdown</a> format. Back in January, I attended <a href="http://codemash.org/" target="_blank">CodeMash</a> in Sandusky, Ohio and someone turned me onto a markdown rendering Quicklook plugin. It's a great little tool. I can keep writing my docs and notes in markdown, but when I Quicklook them, they are rendered for display nicely. If you have interest in this free plugin, you can download it <a href="https://github.com/toland/qlmarkdown" target="_blank">here</a>.<br />
<br />
Anyway, I settled into this habit of editing a file in vim, saving it, then pressing spacebar on it from finder to preview it. It got me thinking, wouldn't it be great if I could just open Quicklook directly from vim?<br />
<br />
After some quick googling around, I found a command-line tool in OSX that does just that. It's called qlmanage. From there it was easy enough to come up with a vim key mapping.<br />
<br />
Put this (all one line) in your .vimrc file and you should be able to do Quicklooks of your files right from vim simply by hitting <b><span style="color: blue;">\v</span></b>:<br />
<br />
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-size: x-small;">map <Leader>v :write<cr>:sil !/usr/bin/qlmanage -p % > /dev/null &<cr>:redraw!<cr></span></b><br />
<br />
It works from both MacVim as well as the terminal version. You don't have to use it solely with markdown files. In fact you can use it with any file that Quicklook understands (html for example).bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-12437601000380937012012-05-22T11:29:00.002-04:002012-05-22T11:29:21.535-04:00Fun with gnu screenIn my day job, I spend a fair amount of time working on numerous tasks on the command-line in Linux and OSX. Way back in my college days, I used a program called <a href="http://www.gnu.org/software/screen/" target="_blank">gnu screen</a> (aka 'screen') but recently I've "refound" it. I have a new appreciation for screen and now use it every day.<div>
<br /></div>
<h3>
What is screen?</h3>
<div>
Screen let's you have multiple shell sessions running simultaneously in the same terminal window. Here's some of the features that I personally find appealing:</div>
<div>
<ul>
<li>With a little muscle memory, you can switch rapidly between the sessions.</li>
<li>It has the ability to split up your terminal window into multiple "window-splits" so you can see two or more shells sessions at the same time. The splits are even resizable.</li>
<li>You can 'detach' and 'reattach' to a screen session, allowing you to sign-out, leave and come back later, all the while keeping your terminal sessions active and running.</li>
<li>It's configurable and scriptable.</li>
<li>You can copy and paste between sessions with 'vi' like keystrokes.</li>
</ul>
<div>
<br /></div>
<h3>
How do you get screen?</h3>
</div>
<div>
Well, if you're running ubuntu like me, it's pretty darn easy. You're one command away from having it:</div>
<div>
<br /></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~ $ <b>sudo apt-get install screen</b></span></div>
<div>
<br /></div>
<div>
Pretty simple.</div>
<div>
<br /></div>
<h3>
Getting started</h3>
<div>
Once you have screen installed, simply run it.</div>
<div>
<br /></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~ $ <b>screen</b></span></div>
<div style="text-align: left;">
<br /></div>
<div>
It will bring you to a welcome screen (I'll show you how to disable the welcome screen later).</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-sqTdQx_cydw/T7uOR07DA8I/AAAAAAAAAJ4/wMpMgeNj5R8/s1600/welcome+screen.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="254" src="http://1.bp.blogspot.com/-sqTdQx_cydw/T7uOR07DA8I/AAAAAAAAAJ4/wMpMgeNj5R8/s320/welcome+screen.png" width="320" /></a></div>
<div>
<br /></div>
<div>
At the bottom of the welcome screen, it will say</div>
<div>
<br /></div>
<div style="text-align: center;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">[Press Space or Return to end.]</span></div>
<div>
<br /></div>
<div>
Go ahead and do that. You'll be returned to a bash prompt. It's important to note that screen hasn't terminated, but rather it's spawned another shell prompt within itself and you're sitting at that new prompt. You can now begin using screen via keyboard command sequences. To prove it, press <span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>Ctrl-a</b></span> and then press <span style="color: blue; font-family: 'Courier New', Courier, monospace; font-weight: bold;">"</span><span style="font-family: Times, 'Times New Roman', serif;"> (double quote)</span>. You should get a screen that looks like this:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-UfLbFi864U0/T7uOdeyfxUI/AAAAAAAAAKA/V2NF14bqmbo/s1600/session+list.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="254" src="http://4.bp.blogspot.com/-UfLbFi864U0/T7uOdeyfxUI/AAAAAAAAAKA/V2NF14bqmbo/s320/session+list.png" width="320" /></a></div>
<div>
This is the terminal session list. If you have more than one terminal session running, you'd seem them all listed here. Let's create another terminal session. Press <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a</span></b> and then press <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">c</span></b>. This should take you back to a bash prompt. Your now in a new bash prompt. See see the updated list, press <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a</span></b> and <span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>"</b></span> again.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-mK4oYj50FtY/T7uP51-G89I/AAAAAAAAAKI/eM5IptGTbVs/s1600/session+list+with+2+sessions.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="254" src="http://1.bp.blogspot.com/-mK4oYj50FtY/T7uP51-G89I/AAAAAAAAAKI/eM5IptGTbVs/s320/session+list+with+2+sessions.png" width="320" /></a></div>
<div>
Now there's two bash sessions. You can select the one you'd like to see by using your arrow keys (or vi cursor movement keys) to highlight the one you want and then press enter/return.</div>
<div>
<br /></div>
<div>
When you want to quit out, simply exit out of all your bash prompts. Once the last bash session ends, screen will automatically terminate, leaving where you were when you initially started it.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-FISbJ4ShiaA/T7uQtAUG_4I/AAAAAAAAAKQ/hf680KwzJyU/s1600/screen+terminates.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="254" src="http://4.bp.blogspot.com/-FISbJ4ShiaA/T7uQtAUG_4I/AAAAAAAAAKQ/hf680KwzJyU/s320/screen+terminates.png" width="320" /></a></div>
<h3>
My most used key sequences</h3>
<div>
As you've probably realized by now, screen uses <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a</span></b> plus a key to drive it's functionality. Here's a list of key sequences that I use most often.</div>
<div>
<br /></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>Ctrl-a Ctrl-a</b></span> - Toggles between two sessions. It's kind of like the "last" button on your TV remote.</div>
<div>
<br /></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a c</span></b> - Creates a new terminal session.</div>
<div>
<br /></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a A</span></b> - Let's you set the title of a terminal session. The title is what appears in the session list.</div>
<div>
<br /></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a "</span></b> - Lists the terminal sessions. You can then use arrow keys or vi keys to select which session you wan to go to.</div>
<div>
<br /></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a d</span></b> - Detach screen so you can log off and come back later. Later when you sign back on, just type <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">screen -r</span></b> to reattach to your running screen session.</div>
<div>
<br /></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a :</span></b> - Bring up screen's colon prompt (similar to vi). From the colon prompt you can issue commands directly to screen. (You'll probably use it most resizing window splits).</div>
<div>
<br /></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a [</span></b> - Start copy mode (for copy/paste).</div>
<div>
<br /></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a ]</span></b> - Paste what you copied from copy mode.</div>
<div>
<br /></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a S</span></b> - Split the screen horizontally.</div>
<div>
<br /></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a |</span></b> (pipe) - Split the screen vertically.</div>
<div>
<br /></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a {tab}</span></b> - Move focus from window split to window split</div>
<div>
<br /></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a X</span></b> - Close the current window split.</div>
<div>
<br /></div>
<h3>
Split example</h3>
<div>
Here a quick example of doing a couple window splits. First, start screen and a few terminal sessions:</div>
<div>
<br /></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace; text-align: left;">steve@ubuntu64 ~ $ </span><b style="color: blue; font-family: 'Courier New', Courier, monospace; text-align: left;">screen</b></div>
<div>
<b style="color: blue; font-family: 'Courier New', Courier, monospace; text-align: left;">{return}</b></div>
<div>
<b style="color: blue; font-family: 'Courier New', Courier, monospace; text-align: left;">Ctrl-a c</b></div>
<div>
<b style="color: blue; font-family: 'Courier New', Courier, monospace; text-align: left;">Ctrl-a c</b></div>
<div>
<b style="color: blue; font-family: 'Courier New', Courier, monospace; text-align: left;">Ctrl-a "</b></div>
<div>
<br /></div>
<div>
Your terminal window should look like this:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-YUUaiSZk9gM/T7umMfeoKSI/AAAAAAAAAKc/utPdCOPf4C8/s1600/three+sessions.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="254" src="http://3.bp.blogspot.com/-YUUaiSZk9gM/T7umMfeoKSI/AAAAAAAAAKc/utPdCOPf4C8/s320/three+sessions.png" width="320" /></a></div>
<div>
Now press <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a S</span></b> to split the screen. You should now have an upper split and a lower split. Press <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">{return}</span></b> now to select terminal session 2 in the upper split. In that terminal session run the <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">top</span></b> command.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-ojyaynrteKo/T7unwLhStyI/AAAAAAAAAKk/q5wAc-bDcPQ/s1600/split+1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="254" src="http://4.bp.blogspot.com/-ojyaynrteKo/T7unwLhStyI/AAAAAAAAAKk/q5wAc-bDcPQ/s320/split+1.png" width="320" /></a></div>
<div>
Now press <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a {tab}</span></b>. This will move the cursor down to the bottom split. Then press <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a "</span></b> to bring up the session list.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-S3Lb47JiJRs/T7uoQtAlV-I/AAAAAAAAAKs/gV_KE79YV8M/s1600/split+2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="254" src="http://3.bp.blogspot.com/-S3Lb47JiJRs/T7uoQtAlV-I/AAAAAAAAAKs/gV_KE79YV8M/s320/split+2.png" width="320" /></a></div>
<div>
Select session <b>0</b> and then issue the following command in the lower split: <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">watch ls</span></b>.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-KyqX9_-vI4k/T7uo2WkR7hI/AAAAAAAAAK0/rqmybkyOVvk/s1600/split+3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="254" src="http://2.bp.blogspot.com/-KyqX9_-vI4k/T7uo2WkR7hI/AAAAAAAAAK0/rqmybkyOVvk/s320/split+3.png" width="320" /></a></div>
<div>
Next, let's split the lower window vertically. Press <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a |</span></b> (pipe), <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a {tab}</span></b>, <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a "</span></b>. Select session 1.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-XFq-8kL6ODw/T7upm9RAgpI/AAAAAAAAAK8/Oe9tZg-cw5s/s1600/split+4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="254" src="http://1.bp.blogspot.com/-XFq-8kL6ODw/T7upm9RAgpI/AAAAAAAAAK8/Oe9tZg-cw5s/s320/split+4.png" width="320" /></a></div>
<div>
As you can see, you can keep splitting, sub-splitting, etc. Remember, <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a {tab}</span></b> moves the cursor to the next window split. That's how you move around.</div>
<div>
<br /></div>
<div>
When you're done, you can use <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">Ctrl-a X</span></b> to close the window splits. This does not quit/kill the process running in that split. The process stays in the session list until you exit it normally.</div>
<div>
<br /></div>
<h3>
Scripting</h3>
<div>
Screen reads and processes the <b>~/.screenrc</b> file at startup. Here's mine:</div>
<div>
<br /></div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">vbell off</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">startup_message off</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">defscrollback 1000</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">bindkey -k k1 colon</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">bindkey -k k2 screen</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">bindkey -k F2 windowlist -b</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">hardstatus off</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">caption always '%{Yb}[%=%t%=]'</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">altscreen on</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">shell -${SHELL}</span></div>
</div>
<div style="text-align: left;">
<br /></div>
<div>
You can go through the screen man page for what each line does. You can customize screen however you'd like.</div>
<div>
<br /></div>
<div>
There are two features that combined make startup processing especially interesting:</div>
<div>
<ol>
<li>You can use the <b>-c</b> command line argument to give screen an alternate rc file to process on startup.</li>
<li>Using commands, you can do anything in the rc file that you could do interactively.</li>
</ol>
<div>
Using this notion, I create different screen rc files for different contexts. For example, when I'm working at a certain client, I would like to have screen set up a certain way, with specific terminal sessions setup. I create a bash aliases for each client.</div>
</div>
<div>
<br /></div>
<div>
In my ~/.bash_aliases file I have:</div>
<div>
<br /></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">alias clientX='screen -c ~/screen_profiles/clientX'</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">alias clientY='screen -c ~/screen_profiles/clientY'</span></div>
<div style="text-align: left;">
<br /></div>
<div>
My ~/screen_profiles/clientX file looks like this:</div>
<div>
<br /></div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">altscreen on</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">vbell off</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">attrcolor b ".I"</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">termcapinfo xterm 'Co#256:AB=\E[48;5;%dm:AF=\E[38;5;%dm'</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">defbce "on"</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">defscrollback 1500</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">bindkey -k k1 colon</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">bindkey -k k2 screen</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">bindkey -k F2 windowlist -b</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">screen -t "Home" bash</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">screen -t "Build" ssh <i>my_username@build_server</i></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">screen -t "Dev2" ssh <i>my_username@dev_server</i></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">screen -t "WebLogic" <i>my_username@weblogic_server</i></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">screen -t "Todo/Notes" vi ~/notes/todo.markdown</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">hardstatus off</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">caption always '%{Yb}[%=%t%=]'</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">select 0</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">split -v</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">resize +20</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">focus</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">select 5</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">focus</span></div>
</div>
<div style="text-align: left;">
<br /></div>
<div>
This rc file does the following:</div>
<div>
<ul>
<li>Does some normal setup that I do anyway (key bindings, colors, etc.)</li>
<li>Creates a terminal session titled, "Home" and runs bash in it.</li>
<li>Creates a terminal session titled, "Build" and ssh's me into the build server.</li>
<li>Creates a terminal session titled, "Dev2" and ssh's me into the dev 2 server.</li>
<li>Creates a terminal session titled, "WebLogic" and ssh's me into the weblogic server.</li>
<li>Creates a terminal session titled, "Todo/Notes" and edits my ~/notes/todo.markdown file using vi in it.</li>
<li>Splits the window vertically, resizing the left window to be 20 columns bigger.</li>
<li>Selects the "Home" session for the left window.</li>
<li>Selects the "Todo/Notes" session for the right window.</li>
<li>Puts the cursor focus on the left window.</li>
</ul>
</div>
<div>
So when I come into clientX's office for the day, I simply type <b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">clientX</span></b> at the bash prompt and within a second, I have screen all set up with my splits and automatically logged into all the servers I commonly use. Pretty cool.</div>bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-66349973126409499302012-05-17T10:41:00.003-04:002012-05-18T09:45:46.675-04:00Gitolite... my new shiny pennyAt my current client engagement, I'm serving as the build-guy and all around code librarian. I'm managing 35 or so git repositories... keeping them tidy and organized.<br />
<br />
Since the client seems to heavily utilize consultant help, they have a veritable revolving door of staff. Before my arrival they decided to purchase and use a tool called <a href="http://gitorious.org/" target="_blank">gitorious</a> to aid in managing the repositories and access to them. It's pretty much like a private github. It has all kinds of features including web-based management of users/keys, browse-able git logs, etc. They have it running as an appliance on a virtual server here at the office.<br />
<br />
<br />
<br />
It works pretty well and I've embraced it because it's the "officially-blessed-by-enterprise-architecture" tool.<br />
<br />
It's not without it's drawbacks though:<br />
<br />
<ul>
<li>In it's default configuration, EVERYONE/ANONYMOUS has READ access to ALL repositories.</li>
<li>Gitorious is heavily integrated with git's hooks, so if you want to add your own hooks to a repository, you're pretty much out of luck... unless you like tampering with their code - good luck with that.</li>
<li>If you don't want use their appliance and want to install it yourself, it's not all that straight forward.</li>
<li>The UI isn't really that intuitive. For example if you want to see a list of tags for a repository, you have to drill into a branch rather than seeing them at the repository level.</li>
<li>It costs money if you want to have your own on-site copy.</li>
</ul>
All that said, I started looking around for alternatives for managing medium to large numbers of git repositories (more than 5'ish) both for my own stuff as well as for future clients.<br />
<br />
In looking around, I found a tool called <a href="https://github.com/sitaramc/gitolite" target="_blank">gitolite</a>. Gitolite seem to provide all the features that I want/need and has the benefits of being simple/easy to implement as well as being opensource/free.<br />
<br />
Gitolite doesn't have a user interface. You mange it with... what else... git!<br />
<br />
<h3>
Installation is easy</h3>
If you're running on Ubuntu, you're only a command away from installation:<br />
<br />
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~ $</span><b style="color: blue; font-family: 'Courier New', Courier, monospace;"> </b><span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>sudo apt-get install gitolite</b></span></div>
<div style="text-align: left;">
<br /></div>
That get's app installed and ready to configure.<br />
<h3>
Configuration is easy too</h3>
Instead of housing gitolite on my personal account (which you could easily do... I just choose not to), I created an account for it called git.<br />
<br />
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~ $ </span><b style="color: blue; font-family: 'Courier New', Courier, monospace;">sudo adduser git</b></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Adding user `git' ...</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Adding new group `git' (1002) ...</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Adding new user `git' (1002) with group `git' ...</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Creating home directory `/home/git' ...</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Copying files from `/etc/skel' ...</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Enter new UNIX password:</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Retype new UNIX password:</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">passwd: password updated successfully</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Changing the user information for git</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Enter the new value, or press ENTER for the default</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> Full Name []: Git Repository Home</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> Room Number []:</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> Work Phone []:</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> Home Phone []:</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> Other []:</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Is the information correct? [Y/n] y</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~ $</span></div>
<div style="text-align: left;">
<br /></div>
After creating the account, you'll want to copy your public key to the git account's home directory for the one and only step of initial gitolite configuration.<br />
<br />
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~ $</span><b style="color: blue; font-family: 'Courier New', Courier, monospace;"> sudo cp .ssh/id_rsa.pub /home/git/steveb.pub</b></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~ $<b> sudo chown git.git /home/git/steveb.pub</b></span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~ $</span></div>
<div>
<div style="text-align: left;">
<br /></div>
</div>
<div>
Now we're ready to do the initial configuration. You'll need to become the git account to do it.</div>
<div>
<br /></div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~ $</span><b style="color: blue; font-family: 'Courier New', Courier, monospace;"> su - git</b></div>
</div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Password:</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">git@ubuntu64:~$</span></div>
<div>
<div style="text-align: left;">
<br /></div>
</div>
<div>
Then you'll want to issue the command <b>gl-setup</b> from the bash prompt, passing in the key you just copied.</div>
<div>
<br /></div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">git@ubuntu64:~$</span><b style="color: blue; font-family: 'Courier New', Courier, monospace;"> gl-setup steveb.pub</b></div>
</div>
<div>
<div style="text-align: left;">
<br /></div>
</div>
<div>
It will prompt you to edit the default <b>.gitolite.rc</b> file, but don't change anything, it's fine as it is for now. You may have a need to change it in the future but I haven't so far. Just save and quit out of your editor without changing anything.</div>
<div>
<br /></div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">The default settings in the rc file (/home/git/.gitolite.rc) are fine for most</span></div>
</div>
<div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">people but if you wish to make any changes, you can do so now.</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><br /></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">hit enter...</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">creating gitolite-admin...</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Initialized empty Git repository in /home/git/repositories/gitolite-admin.git/</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">creating testing...</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Initialized empty Git repository in /home/git/repositories/testing.git/</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">[master (root-commit) 09aad52] start</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> 2 files changed, 6 insertions(+)</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> create mode 100644 conf/gitolite.conf</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> create mode 100644 keydir/steveb.pub</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">git@ubuntu64:~$</span></div>
</div>
<div>
<div style="text-align: left;">
<br /></div>
</div>
<div>
That's it. Gitolite is all set up and ready to use!</div>
</div>
<div>
<br /></div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">git@ubuntu64:~$</span><b style="color: blue; font-family: 'Courier New', Courier, monospace;"> exit</b></div>
</div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">logout</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~ $</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><br /></span></div>
<h3>
Using gitolite is a piece-o-cake</h3>
<br />
Managing your git repositories with gitolite is done entirely by manipulating a git repository called <b>gitolite-admin</b>. First let's use gitolite to create a new repository. Step one: clone the <b>gitolite-admin</b> repository from your system:<br />
<br />
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~ $</span><b style="color: blue; font-family: 'Courier New', Courier, monospace;"> cd dev</b></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev $</span><b style="color: blue; font-family: 'Courier New', Courier, monospace;"> git clone git@localhost:gitolite-admin.git</b></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Cloning into 'gitolite-admin'...</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">remote: Counting objects: 6, done.</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">remote: Compressing objects: 100% (4/4), done.</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">remote: Total 6 (delta 0), reused 0 (delta 0)</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Receiving objects: 100% (6/6), done.</span></div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev $</span></div>
<div>
<div style="text-align: left;">
<br /></div>
</div>
<div>
You'll notice that the <b>gitolite-admin</b> clone has a pretty simple directory structure.</div>
<div>
<br /></div>
<div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev $<b> cd gitolite-admin</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev/gitolite-admin(master) $<b> ls -la</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">total 20</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">drwxr-xr-x 5 steve steve 4096 May 17 09:23 .</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">drwxrwxr-x 13 steve steve 4096 May 17 09:23 ..</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">drwxrwxr-x 2 steve steve 4096 May 17 09:23 <b>conf</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">drwxrwxr-x 8 steve steve 4096 May 17 09:24 .git</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">drwxrwxr-x 2 steve steve 4096 May 17 09:23 <b>keydir</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev/gitolite-admin(master) $</span></div>
</div>
</div>
<div>
<div style="text-align: left;">
<br /></div>
</div>
<div>
It has two directories: <b>conf</b> and <b>keydir</b>. The conf directory contains a file called <b>gitolite.conf</b>. You edit the <b>gitolite.conf</b> file to create repositories and manage permissions to those repositories.</div>
<div>
<br /></div>
<div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev/gitolite-admin(master) $<b> ls -l conf</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">total 4</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">-rw-rw-r-- 1 steve steve 92 May 17 09:23 <b>gitolite.conf</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev/gitolite-admin(master) $</span></div>
</div>
</div>
<div>
<div style="text-align: left;">
<br /></div>
</div>
<div>
The <b>keydir</b> directory is where you should place public key files of the users you want to grant access to repositories. You'll notice that the public key you copied for your self is already there!</div>
<div>
<br /></div>
<div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev/gitolite-admin(master) $ <b>ls -l keydir</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">total 4</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">-rw-rw-r-- 1 steve steve 391 May 17 09:23 <b>steveb.pub</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev/gitolite-admin(master) $</span></div>
</div>
</div>
<div>
<div style="text-align: left;">
<br /></div>
</div>
<div>
Let's add that new repository. To do so, simply edit the <b>conf/gitolite.conf</b> file with your favorite text editor.</div>
<div>
<br /></div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev/gitolite-admin(master) $ vi <b>conf/gitolite.conf</b></span></div>
</div>
<div>
<div style="text-align: left;">
<br /></div>
</div>
<div>
Add the following lines to it:</div>
<div>
<br /></div>
<div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">repo gitolite-admin</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> RW+ = steveb</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">repo testing</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> RW+ = @all</span></div>
</div>
<div>
<div style="text-align: left;">
<b style="color: blue; font-family: 'Courier New', Courier, monospace;">repo<span class="Apple-tab-span" style="white-space: pre;"> </span>my_cool_new_repository</b></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>RW+<span class="Apple-tab-span" style="white-space: pre;"> </span>= steveb</b></span></div>
</div>
</div>
<div>
<div style="text-align: left;">
<br /></div>
</div>
<div>
The two new lines in the file establish that you want to add a new repository named <b>my_cool_new_repository</b> and that you want to grant the user with the public key file <b>steveb.pub</b> full <b>READ/WRITE/REFEDIT</b> permissions.</div>
<div>
<br /></div>
<div>
Save the file. Commit it. Then push it back to the repository origin.</div>
<div>
<div>
<br /></div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev/gitolite-admin(master) $ <b>git add .</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev/gitolite-admin(master) $ <b>git commit -m"Added new repository my_cool_new_repository"</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">[master 393ef15] Added new repository my_cool_new_repository</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> 1 file changed, 3 insertions(+)</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev/gitolite-admin(master) $ <b>git push</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Counting objects: 7, done.</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Delta compression using up to 2 threads.</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Compressing objects: 100% (3/3), done.</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Writing objects: 100% (4/4), 433 bytes, done.</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Total 4 (delta 0), reused 0 (delta 0)</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>remote: creating my_cool_new_repository...</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>remote: Initialized empty Git repository in /home/git/repositories/my_cool_new_repository.git/</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">To git@localhost:gitolite-admin.git</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> 09aad52..393ef15 master -> master</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev/gitolite-admin(master) $</span></div>
</div>
<div>
<div style="text-align: left;">
<br /></div>
</div>
</div>
<div>
Gitolite sees your change to the <b>conf/gitolite.conf</b> file. It then creates a new blank, bare git repository called <b>my_cool_new_repository.git</b>. It also ensures that the <b>steveb.pub</b> file has the appropriate access to the git account (check out the git account's ~/.ssh/authorized_keys).</div>
<div>
<br /></div>
<div>
That's it. Your new repository is ready to clone!</div>
<div>
<br /></div>
<div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev/gitolite-admin(master) $ <b>cd ..</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev $ <b>git clone git@localhost:my_cool_new_repository.git</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Cloning into 'my_cool_new_repository'...</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">warning: You appear to have cloned an empty repository.</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev $ <b>cd my_cool_new_repository/</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev/my_cool_new_repository(master) $ <b>date > file1.txt</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev/my_cool_new_repository(master) $ <b>git add .</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev/my_cool_new_repository(master) $ <b>git commit -m"Added file1.txt"</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">[master (root-commit) 90e1635] Added file1.txt</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> 1 file changed, 1 insertion(+)</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> create mode 100644 file1.txt</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev/my_cool_new_repository(master) $ <b>git push origin master</b></span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Counting objects: 3, done.</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Writing objects: 100% (3/3), 257 bytes, done.</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">Total 3 (delta 0), reused 0 (delta 0)</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">To git@localhost:my_cool_new_repository.git</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"> * [new branch] master -> master</span></div>
</div>
<div>
<div style="text-align: left;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;">steve@ubuntu64 ~/dev/my_cool_new_repository(master) $</span></div>
</div>
</div>
<div>
<div style="text-align: left;">
<br /></div>
</div>
<div>
Adding/removing user access is as simple as adding/removing key files from the <b>keydir</b> directory, updating the <b>conf/gitolite.conf</b> file and pushing.</div>
<div>
<br /></div>
<div>
The gitolife.conf file, while very simple, is incredibly powerful. It supports groups. Also, it will not only allow you to control access to git repositories, but will also allow you to assign permissions to branches and tags with regular expressions. Take a look at the <a href="https://github.com/sitaramc/gitolite/tree/master/doc" target="_blank">documentation</a> for much more info.</div>bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-68132300673550063222012-05-01T11:57:00.002-04:002012-05-18T09:40:32.296-04:00Use automator to create X11 shortcuts in OSXI have a Mac and yet I do a fair amount of work in X11apps on Linux systems. I really like the ssh's -Y flag that let's me run my Linux X11 apps and have their display output (and input) sent back to my Mac.<br />
<br />
<div style="text-align: center;">
<iframe bordercolor="#000000" frameborder="0" height="150" hspace="0" marginheight="0" marginwidth="0" scrolling="no" src="http://ad.doubleclick.net/adi/N7433.148119.BLOGGEREN/B6627866.444;sz=180x150;ord=[timestamp]?;lid=41000000026530730;pid=57365;usg=AFHzDLvO-qX1OMueCN4NvXQW37hlKQgROw;adurl=http%253A%252F%252Fwww.abt.com%252Fproduct%252F57365%252FApple-MD313LLA.html;pubid=543520;price=%241199.00;title=Apple+13.3+Inch+MacBoo...;merc=Abt+Electronics+%26+Appliances;imgsrc=http%3A%2F%2Fcontent.abt.com%2Fmedia%2Fimages%2Fproducts%2FBDP_Images%2Fbig_mbp13blion.jpg;width=110;height=71" vspace="0" width="180"></iframe></div>
<br />
I just stumbled onto this in Automator and thought it might be worth sharing.<br />
<br />
You can create Automator "Application" workflows that look/feel/behave like local OSX applications, but are actually running on remote Linux servers. In this posting, I will walk through creating an Automator workflow for Synaptic, an application package manager for apt based Linux distributions.<br />
<br />
<h3>
Requirements</h3>
You will need to ensure you have an environment that meets the following requirements in order to follow along:<br />
<br />
<ul>
<li>A Mac running OSX (I'm using Lion. I've not tested this with other versions so if you're using another version, your milage may vary.)</li>
<li>You will need X11.app or<a href="http://xquartz.macosforge.org/landing/" target="_blank"> XQuartz.app</a> installed on your Mac. X11.app comes with OSX up to Lion. After Lion, you will need to install XQuartz.app yourself.</li>
<li>Access to a Linux system that is running SSH with X11 tunneling enabled (a default I believe) and gksudo and synaptic installed.</li>
<li>You need to have key-based authentication set up with your account so you are not prompted for a password when logging into the Linux machine via ssh.</li>
</ul>
<b>gksudo</b> is normally not needed. In this case we're running synaptic, a package manager for Linux, which needs elevated privileges in order to add/remove software packages on the Linux machine. If the X11 app you are wanting to run doesn't require elevated privileges you won't need gksudo.<br />
<h3>
Verify that you can log in without as password prompt</h3>
<div>
Before we create the Automator workflow, let's verify that you can log into the Linux system from your Mac without a password prompt. Start a terminal window and issue a command in the following format:</div>
<div>
<br /></div>
<div style="text-align: center;">
<b><span style="color: blue;">ssh username@linux_hostname</span></b></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
If you are prompted for a password you will still need to set yourself up for key-based authentication. Check out any of <a href="https://www.google.com/search?sourceid=chrome&ie=UTF-8&q=setting+up+key-based+authentication+in+ssh" target="_blank">these links</a> and then come back. You should be let in without a password as seen below:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-mRSefpKhku0/T5_uY-BcJII/AAAAAAAAAGw/Dc4LSyBK8SI/s1600/no+password+prompt.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://2.bp.blogspot.com/-mRSefpKhku0/T5_uY-BcJII/AAAAAAAAAGw/Dc4LSyBK8SI/s320/no+password+prompt.png" width="320" /></a></div>
<div style="text-align: left;">
<br /></div>
<div>
<br /></div>
<h3>
Start a new Automator Workflow</h3>
<div>
Once you have confirmed that you can ssh from your Mac into your Linux system, you're ready to get started. Run Automator and start a new "Application' workflow.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-q7S_o4ymd4s/T5_xgK4y3LI/AAAAAAAAAG8/H6bDddBEiLU/s1600/new+workflow.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="276" src="http://2.bp.blogspot.com/-q7S_o4ymd4s/T5_xgK4y3LI/AAAAAAAAAG8/H6bDddBEiLU/s320/new+workflow.png" width="320" /></a></div>
<div>
Then we're going to add a single action to the workflow. In the <b><span style="color: blue;">Actions</span></b> library locate the <b><span style="color: blue;">Utilities</span></b> group and select it. Then drag the<b><span style="color: blue;"> Run Shell Script</span></b> action to your workflow.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-588cd4mpj-I/T5_yKdUGgDI/AAAAAAAAAHE/5Erh5z2VRvA/s1600/run+shell+action.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="276" src="http://1.bp.blogspot.com/-588cd4mpj-I/T5_yKdUGgDI/AAAAAAAAAHE/5Erh5z2VRvA/s320/run+shell+action.png" width="320" /></a></div>
<div>
Once you've done that, click on the <b><span style="color: blue;">Options</span></b> link at the bottom of the action. It will reveal some options. Check the <b><span style="color: blue;">Ignore this action's input</span></b> checkbox.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-tiZ0mn9Zk7M/T5_y2srvEOI/AAAAAAAAAHM/gL2RmZKLB0U/s1600/ignore+input.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="262" src="http://1.bp.blogspot.com/-tiZ0mn9Zk7M/T5_y2srvEOI/AAAAAAAAAHM/gL2RmZKLB0U/s320/ignore+input.png" width="320" /></a></div>
<div>
Finally, replace the cat command with the following ssh command:</div>
<div>
<br /></div>
<div style="text-align: center;">
<b><span style="color: blue;">ssh -Y username@linux_hostname gksudo -g synaptic &</span></b></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-9gHhdhFj5vw/T5_081yD32I/AAAAAAAAAHk/sFqfcTZvZR4/s1600/ssh+syntax.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="262" src="http://1.bp.blogspot.com/-9gHhdhFj5vw/T5_081yD32I/AAAAAAAAAHk/sFqfcTZvZR4/s320/ssh+syntax.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div>
<br /></div>
<div>
Where <i>username</i> is your username and <i>linux_hostname</i> is your Linux host's hostname. Don't forget the <b><span style="color: blue;">&</span></b> at the end. Then save your workflow as <b><span style="color: blue;">Synaptic</span></b>. You can click the run button if you'd like to test it out.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-ZnNI0iVJfpM/T5_0oL1fq0I/AAAAAAAAAHc/wA-9wcVtSd8/s1600/save.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="258" src="http://4.bp.blogspot.com/-ZnNI0iVJfpM/T5_0oL1fq0I/AAAAAAAAAHc/wA-9wcVtSd8/s320/save.png" width="320" /></a></div>
<div>
Once you're satisfied with it, quit out of automator. Now you can run the workflow directly from OSX. Locate it in Finder and double-click on it.</div>
<div class="separator" style="clear: both; text-align: -webkit-auto;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-atsRKLDnlds/T5_3TfhTUSI/AAAAAAAAAHs/Lh2aOnbcKgo/s1600/icon+on+desktop.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-atsRKLDnlds/T5_3TfhTUSI/AAAAAAAAAHs/Lh2aOnbcKgo/s1600/icon+on+desktop.png" /></a> </div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-gYr37p6FtXA/T5_3kKOsBgI/AAAAAAAAAH8/9CoaJXssgrc/s1600/password+prompt.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="204" src="http://3.bp.blogspot.com/-gYr37p6FtXA/T5_3kKOsBgI/AAAAAAAAAH8/9CoaJXssgrc/s320/password+prompt.png" width="320" /></a></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-mEu3MF_S-Wo/T5_3oJqnKrI/AAAAAAAAAIE/zrhRmujAqdo/s1600/synaptic+running+in+X11.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="212" src="http://2.bp.blogspot.com/-mEu3MF_S-Wo/T5_3oJqnKrI/AAAAAAAAAIE/zrhRmujAqdo/s320/synaptic+running+in+X11.png" width="320" /></a></div>
<div>
<br /></div>
<div>
That's it. You can create any number of workflows for individual Linux apps. I have one for LibreOffice, one for synaptic, one for Eclipse and one for gnome-system-monitor.</div>
<div>
<br /></div>
<h3>
Optional - Set icon for your workflow</h3>
<div>
If you're like me, the generic Automator icon on my Synaptic workflow is a little annoying. You can set the icon to whatever you'd like. Here's <i>one way</i> to do it.</div>
<div>
<br /></div>
<div>
First, obtain a copy of the synaptic.png file from your Linux machine and place it on your Mac. I use the following command from a terminal:</div>
<div>
<br /></div>
<div style="text-align: center;">
<b><span style="color: blue;">scp steve@ubuntu32.local:/usr/share/pixmaps/synaptic.png ~/Desktop</span></b></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
This puts the synaptic.png file on my Mac's desktop. Then I double-click the synaptic.png file. It should open up in Preview. In preview, I select <b><span style="color: blue;">Edit->Select All</span></b>, then <b><span style="color: blue;">Edit->Copy</span></b>.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-qR61uxbX4uU/T6AE7CwJlYI/AAAAAAAAAIY/yB-bsfdelWA/s1600/Copying+from+Preview.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="http://4.bp.blogspot.com/-qR61uxbX4uU/T6AE7CwJlYI/AAAAAAAAAIY/yB-bsfdelWA/s320/Copying+from+Preview.png" width="320" /></a></div>
<div>
Next, I control-click on the Synaptic automator workflow and select <b><span style="color: blue;">Get Info</span></b>. This brings up an informational dialog.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-MoBPE8bVk3Y/T6AFc21WvCI/AAAAAAAAAIg/FdWqWwtXL50/s1600/Getinfo+before.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="http://4.bp.blogspot.com/-MoBPE8bVk3Y/T6AFc21WvCI/AAAAAAAAAIg/FdWqWwtXL50/s320/Getinfo+before.png" width="140" /></a></div>
<div>
In the top-lefthand corner of the dialog, there is a small Automator icon. I click on it, causing it to glow blue-ish. While it's glowing, I press <b><span style="color: blue;">Command-V</span></b>. </div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-d8p2Qdt_aME/T6AGF4wYwyI/AAAAAAAAAIo/KdB94WUoLNk/s1600/get+info+after.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="http://1.bp.blogspot.com/-d8p2Qdt_aME/T6AGF4wYwyI/AAAAAAAAAIo/KdB94WUoLNk/s320/get+info+after.png" width="140" /></a></div>
<div>
Finally I close the dialog. I also delete the synaptic.png file as I no longer need it. Now My synaptic Automator workflow has he official synaptic icon.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-DEoNC7wfMTo/T6AGV0sIBoI/AAAAAAAAAIw/NorKbPvI1SU/s1600/new+icon.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-DEoNC7wfMTo/T6AGV0sIBoI/AAAAAAAAAIw/NorKbPvI1SU/s1600/new+icon.png" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
I hope you found this posting helpful.</div>bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-52877926397630923072012-04-27T13:03:00.000-04:002012-04-27T13:03:01.322-04:00Automated python unit testing, code coverage and code quality analysis with Jenkins - part 3This is the third and final posting in a series. Check <a href="http://bhfsteve.blogspot.com/2012/04/automated-python-unit-testing-code.html" target="_blank">here</a> for the first posting and <a href="http://bhfsteve.blogspot.com/2012/04/automated-python-unit-testing-code_20.html" target="_blank">here</a> for the second posting.<br />
<br />
In this posting I will be explaining how to automate the unit testing, code coverage and code quality analysis that we did manually in the first two posts.<br />
<br />
As the title of this series suggests, I use Jenkins for this automation (also known as <u>c</u>ontinuous <u>i</u>ntegration - ci). In keeping with the theme of using apt for everything, I will install Jenkins via apt. That being said, in my real job, I have downloaded a jenkins.war file from the <a href="http://jenkins-ci.org/" target="_blank">Jenkins web site</a> and I start it up from the command line.<br />
<br />
<h3>
Install and start Jenkins</h3>
Jenkins appears in the default apt that ships with Ubuntu, but it's a really old version... so old, in fact, that it won't run the plugins that we'll need in order to do the code quality and code coverage analysis. To get around this, refer to <a href="http://pkg.jenkins-ci.org/debian/" target="_blank">these instructions</a> on the Jenkins web site to update your apt sources and get the latest version Jenkins via apt.<br />
<br />
Here's a quick screen-shot of the instructions you'll need to follow:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-zBE2AyZ5WQU/T5mg5c04vTI/AAAAAAAAAEA/Vh1o6GGsoqo/s1600/apt+instructions.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="149" src="http://4.bp.blogspot.com/-zBE2AyZ5WQU/T5mg5c04vTI/AAAAAAAAAEA/Vh1o6GGsoqo/s320/apt+instructions.png" width="320" /></a></div>
<br />
Once you've got Jenkins installed, check to see if Jenkins is running. If it isn't start it.<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>sudo /etc/init.d/jenkins status</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>sudo /etc/init.d/jenkins start</b></span><br />
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b><br /></b></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-EJ9CK-4nQJk/T5ml9-VBFgI/AAAAAAAAAEM/w172DZVPYNU/s1600/Jenkins+Status+and+Startup.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://4.bp.blogspot.com/-EJ9CK-4nQJk/T5ml9-VBFgI/AAAAAAAAAEM/w172DZVPYNU/s320/Jenkins+Status+and+Startup.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
Once Jenkins is up and running, let's pull it up in a browser. It will be listening on port 8080.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-fnFkl9Lf5D4/T5mmPY7CJgI/AAAAAAAAAEU/ISVroO0FHYE/s1600/Welcome+to+Jenkins.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="247" src="http://4.bp.blogspot.com/-fnFkl9Lf5D4/T5mmPY7CJgI/AAAAAAAAAEU/ISVroO0FHYE/s320/Welcome+to+Jenkins.png" width="320" /></a></div>
<h3>
Install some needed Jenkins plugins</h3>
Jenkins has lots of plugins that will allow it to do lots of interesting things. We will be needing a few of them. To install plugins, click on the <b><span style="color: blue;">Manage Jenkins</span></b> link from the list of links on the left-hand side of the page. Next click on the <b><span style="color: blue;">Manage Plugins</span></b> link from the resulting page. Finally click on the <b><span style="color: blue;">Available</span></b> tab from the Plugin Manager page.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-U9mtDdOs9oc/T5mmltVYluI/AAAAAAAAAEk/rphu50DeWiM/s1600/Available+Plugins.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="247" src="http://3.bp.blogspot.com/-U9mtDdOs9oc/T5mmltVYluI/AAAAAAAAAEk/rphu50DeWiM/s320/Available+Plugins.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
This will show you a list of all the available plugins that Jenkins can use. Find and select (check the checkbox) for the following plugins:<br />
<ol>
<li>Jenkins Cobertura Plugin</li>
<li>Jenkins GIT plugin</li>
<li>Jenkins Violations</li>
</ol>
<div>
After you've check the checkboxes, click the <b><span style="color: blue;">Download now and install after restart</span></b> button at the bottom of the page.</div>
<div class="separator" style="clear: both; text-align: -webkit-auto;">
<br /></div>
Jenkins will install the selected plugins and then take a moment to restart. After Jenkins restarts we need to make a small configuration change in order for git to work properly. When Jenkins is done restarting, click the <b><span style="color: blue;">Manage Jenkins</span></b> link from the left-hand list of links. The click <b><span style="color: blue;">Configure System</span></b> from the resulting page. This will bring up a global Jenkins configuration page. Scroll down (past the Git section) to the <b><span style="color: blue;">Git Plugin</span></b> section. Enter values in the <b><span style="color: blue;">Global Config user.name Value</span></b> and <b><span style="color: blue;">Global Config user.email Value</span></b> fields. Finally, click the <b><span style="color: blue;">Save</span></b> button at the bottom of the page.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-ciu947jJX8s/T5moZZvbiiI/AAAAAAAAAEs/XuiLE9eCJBA/s1600/Git+config.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="247" src="http://3.bp.blogspot.com/-ciu947jJX8s/T5moZZvbiiI/AAAAAAAAAEs/XuiLE9eCJBA/s320/Git+config.png" width="320" /></a></div>
<h3>
Our first job</h3>
When Jenkins comes back, click the <b><span style="color: blue;">New Job</span></b> link from the left-hand list of links. This will bring you to a page where you can enter a job name and select a job type. Enter <b><span style="color: blue;">Project1</span></b> for the job name and select <b><span style="color: blue;">Build a free-style software project</span></b>, then click the <b><span style="color: blue;">OK</span></b> button.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-kbnZQM7Kmoc/T5moohWqtYI/AAAAAAAAAE0/iNZ27rZhC-A/s1600/Naming+the+first+project.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="247" src="http://2.bp.blogspot.com/-kbnZQM7Kmoc/T5moohWqtYI/AAAAAAAAAE0/iNZ27rZhC-A/s320/Naming+the+first+project.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
This will bring you to the Configure page for your new job. This is where you tell Jenkins WHAT, WHEN and HOW to do. Let's get started!<br />
<br />
<h4>
1 - Where to get the source</h4>
Under the <b><span style="color: blue;">Source Code Management</span></b> section, click the <b><span style="color: blue;">Git</span></b> radio button. This will expand more options. For this project we'll be keeping things simple. Simply enter the path to your git repository. Mine's at <b style="color: blue;">/home/steve/dev/project1 </b>(this presumes your git repo is on the same box that Jenkins is running on).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-hDwb_sbJZHk/T5lzDbCT76I/AAAAAAAAADg/adELp3TAeV8/s1600/Source+code.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="163" src="http://2.bp.blogspot.com/-hDwb_sbJZHk/T5lzDbCT76I/AAAAAAAAADg/adELp3TAeV8/s320/Source+code.png" width="320" /></a></div>
<br />
<h4>
2 - When to run the job</h4>
Next go to the <b><span style="color: blue;">Build Triggers</span></b> section and check the <b><span style="color: blue;">Poll SCM</span></b> checkbox. In the resulting <b><span style="color: blue;">Schedule</span></b> field, enter 5 *'s. (<b><span style="color: blue;">* * * * *</span></b>) This makes Jenkins check the git repository for changes once a minute, every minute of every day. If it finds a change in the repository, it will "do a build". We'll be configuring "the build" to run the unit tests and code coverage and quality tests for us.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-DewpmzsyWZo/T5lzXqPVOlI/AAAAAAAAADo/OA2Fxd1YTzY/s1600/Build+triggers.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="91" src="http://3.bp.blogspot.com/-DewpmzsyWZo/T5lzXqPVOlI/AAAAAAAAADo/OA2Fxd1YTzY/s320/Build+triggers.png" width="320" /></a></div>
<br />
<br />
BTW, for larger Jenkins implementations (more than 10 or so Jenkins jobs), the <b><span style="color: blue;">Poll SCM</span></b> option is a terrible idea. I can show you another way using git hooks if there is interest. For now, though, let's just use the <b><span style="color: blue;">Poll SCM</span></b> option since this is just an example project.<br />
<div>
<br /></div>
<br />
<h4>
3 - What to run</h4>
Next, let's configure the job to do what we want it to when it detects a change in the git repository. In the <b><span style="color: blue;">Build</span></b> section, click the <b><span style="color: blue;">Add build step</span></b> button and select <b><span style="color: blue;">Execute shell</span></b> from the resulting popup. In the resulting <b><span style="color: blue;">Command</span></b> field, enter the following text:<br />
<br />
<textarea cols="110" readonly="readonly" rows="5" style="color: blue; font-face: courier; overflow: scroll;">PYTHONPATH=''
nosetests --with-xunit --all-modules --traverse-namespace --with-coverage --cover-package=project1 --cover-inclusive
python -m coverage xml --include=project1*
pylint -f parseable -d I0011,R0801 project1 | tee pylint.out
</textarea><br />
<br />
Let's review what's going on here. When the Jenkins job detects that a source code change has occurred in the git repository, it will clone a copy of the repository and then run the above script against the clone.<br />
<br />
You'll notice that the nosetests command has become a little more complex. I've added a fair number of command-line arguments to it. This is to make sure that the code coverage runs over all the code, not just the code that's involved in the unit tests. It also ensures that nosetests write's it's output to a file that Jenkins can interpret.<br />
<br />
When nosetests runs the code coverage, it generates a .coverage file. Jenkins can't read that. The third line of the script (python -m coverage xml...) converts the .coverage file to an xml format that Jenkins Cobertura plugin can read.<br />
<br />
The last line (pylint...) runs pylint on the project and outputs it in a format that the Violations Jenkins plugin can read. I also have it disabling a couple warnings that I don't care to know about. (You can customize this all you want BTW).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-5nXM-HcOn_U/T5l3apNh2NI/AAAAAAAAAD0/NEuYqxFSu_A/s1600/The+build.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="98" src="http://3.bp.blogspot.com/-5nXM-HcOn_U/T5l3apNh2NI/AAAAAAAAAD0/NEuYqxFSu_A/s320/The+build.png" width="320" /></a></div>
<br />
<h4>
4 - Interpret the results</h4>
Not only can Jenkins detect changes and run our tests. Given the correct plugins are installed (which we did at the beginning), it can interpret the results and display them in charts/trees/etc. For me this is the best part. All the other stuff, I could have done with some clever scripting and cron.<br />
<br />
First stop is coverage. In the <b><span style="color: blue;">Post-build Actions</span></b> section click the <b><span style="color: blue;">Public Cobertura Coverage Report</span></b> checkbox. Then in the <b><span style="color: blue;">Cobertura xml report pattern</span></b> field, enter <b><span style="color: blue;">coverage.xml</span></b>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-VBFAslDmXzw/T5msR1-aUBI/AAAAAAAAAFA/WZorywPiYno/s1600/cobertura+config.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="160" src="http://2.bp.blogspot.com/-VBFAslDmXzw/T5msR1-aUBI/AAAAAAAAAFA/WZorywPiYno/s320/cobertura+config.png" width="320" /></a></div>
<br />
This tells Jenkins to read a file named coverage.xml that will contain testing code coverage information in it. The line listed below (from the build script in step 3) is what creates this file:<br />
<br />
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">python -m coverage xml --include=project1*</span></b><br />
<br />
Next, click the <b><span style="color: blue;">Publish JUnit test result report</span></b> checkbox and enter <b><span style="color: blue;">nosetests.xml</span></b> in the <b><span style="color: blue;">Test report XMLs</span></b> field.<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-9lGdduomP9E/T5muMW1wOJI/AAAAAAAAAFQ/jy9MsDN5tyU/s1600/Junit+setting.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="71" src="http://3.bp.blogspot.com/-9lGdduomP9E/T5muMW1wOJI/AAAAAAAAAFQ/jy9MsDN5tyU/s320/Junit+setting.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
This instructs Jenkins to interpret the a file named nosetests.xml that the nose command from creates:<br />
<br />
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">nosetests --with-xunit --all-modules --traverse-namespace --with-coverage --cover-package=project1 --cover-inclusive</span></b><br />
<br />
Finally, check the <b><span style="color: blue;">Report Violations</span></b> checkbox and enter <b><span style="color: blue;">**/pylint.out</span></b> in the <b><span style="color: blue;">pylint</span></b> field.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-AOjWMoVUImw/T5mut-T6YkI/AAAAAAAAAFY/1RW5QFUK0gA/s1600/Violations.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://3.bp.blogspot.com/-AOjWMoVUImw/T5mut-T6YkI/AAAAAAAAAFY/1RW5QFUK0gA/s320/Violations.png" width="320" /></a></div>
<br />
This instructs Jenkins to interpret the <b>pylint.out</b> file that is generated by the pylint command from the build script:<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>pylint -f parseable -d I0011,R0801 project1 | tee pylint.out</b></span><br />
<br />
At this point, you're done configuring the job. Click the <b><span style="color: blue;">Save</span></b> button. Now you're ready to give it a go!<br />
<br />
<h3>
Run the job</h3>
Click the <b><span style="color: blue;">Build Now</span></b> link on the left-hand list of links. This will manually kick off a run of the job. If no errors. You should see a screen that looks like the one below. If not refresh your page.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-MHx2CQy-2tU/T5qkz34ZOuI/AAAAAAAAAFk/rLP82BjQGcQ/s1600/First+run.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="265" src="http://4.bp.blogspot.com/-MHx2CQy-2tU/T5qkz34ZOuI/AAAAAAAAAFk/rLP82BjQGcQ/s320/First+run.png" width="320" /></a></div>
Not very interesting, is it? That's because we currently have 100% coverage and no test failures! Let's add some code (without tests) and see what happens.<br />
<br />
<h4>
Update the code</h4>
Add the following code to your ~/dev/project1/project1/authentication.py file. Then save and commit it.<br />
<br />
<br />
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;">def logout():</span></b><br />
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;"> print 'You are now logged out.'</span></b><br />
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-jQ1WRNoKVG4/T5qmkZwUJTI/AAAAAAAAAFs/dITFdEjnUwU/s1600/editing+authentication.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://3.bp.blogspot.com/-jQ1WRNoKVG4/T5qmkZwUJTI/AAAAAAAAAFs/dITFdEjnUwU/s320/editing+authentication.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-MXP4d1tRzVU/T5qmoChLKcI/AAAAAAAAAF0/ZmbyJYrJn3g/s1600/commit+logout.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://3.bp.blogspot.com/-MXP4d1tRzVU/T5qmoChLKcI/AAAAAAAAAF0/ZmbyJYrJn3g/s320/commit+logout.png" width="320" /></a></div>
<div>
No go back to your Jenkins window and refresh the Project1 page. You may need to wait a minute. Remember, Jenkins only checks the git repository for changes once a minute. You should see some new info. The top graph is your code coverage. It indicates that you've dropped from 100% line coverage to 92%. This makes sense because we added function, but no test for it.</div>
<div>
<br /></div>
<div>
The second graph is your unit test pass/fail trend. It's all blue because all 3 of our tests are still passing. We'll break that next by adding a test for logout that fails. :-)</div>
<div>
<br /></div>
<div>
The third graph is your code quality report. It's indicating that we had 0 issues in the first build, but now have 1 issue in the second build.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-f_YtJcTOQ64/T5qnlOiE35I/AAAAAAAAAF8/l5Nl_YvRSuM/s1600/jenkins+after+logout+added.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="309" src="http://4.bp.blogspot.com/-f_YtJcTOQ64/T5qnlOiE35I/AAAAAAAAAF8/l5Nl_YvRSuM/s320/jenkins+after+logout+added.png" width="320" /></a></div>
<div>
You can get more info by clicking on the graphs. For example, let's drill into the code coverage graph.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-eqyI1gDMw9E/T5qoEzNPn3I/AAAAAAAAAGE/nHvstzetU6U/s1600/code+coverage+1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="247" src="http://4.bp.blogspot.com/-eqyI1gDMw9E/T5qoEzNPn3I/AAAAAAAAAGE/nHvstzetU6U/s320/code+coverage+1.png" width="320" /></a></div>
<div>
You can browse even deeper. Click on the <b><span style="color: blue;">project1</span></b> link at the bottom.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-zQcq8Atx608/T5qoQUoMISI/AAAAAAAAAGM/UpFs8u08xlw/s1600/code+coverage+2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="314" src="http://4.bp.blogspot.com/-zQcq8Atx608/T5qoQUoMISI/AAAAAAAAAGM/UpFs8u08xlw/s320/code+coverage+2.png" width="320" /></a></div>
<div>
Then click on the <b><span style="color: blue;">authentication.py</span></b> link at the bottom.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-IcosExjBhFE/T5qogZ2QLxI/AAAAAAAAAGU/MoXwJv-IvN0/s1600/code+coverage+3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="283" src="http://2.bp.blogspot.com/-IcosExjBhFE/T5qogZ2QLxI/AAAAAAAAAGU/MoXwJv-IvN0/s320/code+coverage+3.png" width="320" /></a></div>
<div>
As you can see by the red line at the bottom, the print statement at the bottom is never exercised by a unit test. Hmmmm. We'll need to fix that.</div>
<div>
<br /></div>
<h4>
Add a failing test</h4>
<div>
Add this code to the ~/dev/project1/tests/authentication_tests.py file and then commit it to git:</div>
<div>
<br /></div>
<div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;"> def test_logout(self):</span></b></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;"> """Test the logout function...badly."""</span></b></div>
<div>
<b><span style="color: blue; font-family: 'Courier New', Courier, monospace;"> self.assertEqual(0, 1)</span></b></div>
</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-EI1lKppjxVk/T5qqXfZw-6I/AAAAAAAAAGc/mZGs8Mn5aPY/s1600/add+a+failing+test.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://4.bp.blogspot.com/-EI1lKppjxVk/T5qqXfZw-6I/AAAAAAAAAGc/mZGs8Mn5aPY/s320/add+a+failing+test.png" width="320" /></a></div>
<div>
Wait a minute and then go back to the Jenkins Project1 page. It should now have a third build reporting the test failure.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-8YKWzpdkj_E/T5qquCLGvXI/AAAAAAAAAGk/PGkQt0IUYdg/s1600/jenkins+with+a+failing+test.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="312" src="http://1.bp.blogspot.com/-8YKWzpdkj_E/T5qquCLGvXI/AAAAAAAAAGk/PGkQt0IUYdg/s320/jenkins+with+a+failing+test.png" width="320" /></a></div>
<div>
That's it. Hope you found this helpful. Happy coding.</div>
<div>
<br /></div>
<div>
<br /></div>
<br />
<br />
<br />bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-54941348078361878242012-04-20T12:03:00.000-04:002012-04-20T13:33:37.468-04:00Automated python unit testing, code coverage and code quality analysis with Jenkins - part 2This is the second posting in a series. Check <a href="http://bhfsteve.blogspot.com/2012/04/automated-python-unit-testing-code.html" target="_blank">here</a> for the first posting.<br />
<br />
<h3>
Where we gonna do this?</h3>
First let's create a directory and repository to house the source code. Enter the following commands:<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b style="background-color: white;">mkdir -p ~/dev/project1</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b style="background-color: white;">git init ~/dev/project1</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b style="background-color: white;">cd ~/dev/project1</b></span><br />
<br />
It should go something like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-G7d8EVWIgy0/T5FoqiH0TkI/AAAAAAAAAA8/HGOnX8JOzZM/s1600/Create+directory+and+repo.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://4.bp.blogspot.com/-G7d8EVWIgy0/T5FoqiH0TkI/AAAAAAAAAA8/HGOnX8JOzZM/s320/Create+directory+and+repo.png" width="320" /></a></div>
Next, I always add a <b><span style="color: blue;">.gitignore</span></b> file to my repository so unrelated files aren't tracked in source control. Mine contains the following:<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>*.swp</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>*.pyc</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>tags</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>.coverage</b></span><br />
<br />
Once I've created this file, I add it to the git repository with the following commands:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>git add .gitignore</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>git commit -m"Initial commit. Added a .gitignore file."</b></span><br />
<br />
It should look like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-_ukJBgrKivY/T5FqbHVOTUI/AAAAAAAAABE/LeUp--v5LP8/s1600/Adding+.gitignore.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://3.bp.blogspot.com/-_ukJBgrKivY/T5FqbHVOTUI/AAAAAAAAABE/LeUp--v5LP8/s320/Adding+.gitignore.png" width="320" /></a></div>
<h3>
Let's get some source code in place</h3>
Before we get too ahead of ourselves, let's pause for some quick explanation. I don't like to mix my application code with my unit test code. I keep them separate. My application code will eventually need to be deployed somewhere, but the unit test code doesn't. To that end, I put the application code in one python package named after the project and the unit test code in another package called "tests". So let's create the two packages with the following commands:<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>mkdir project1</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>touch project1/__init__.py</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>mkdir tests</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>touch tests/__init__.py</b></span><br />
<br />
This might not be a bad place to do a quick commit to source control:<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>git add .</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>git commit -m"Added two blank packages: project1 and tests"</b></span><br />
<br />
Again, should look like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-G-JF_8muZz0/T5FsFV9N7GI/AAAAAAAAABM/FQ51V9ZRQ6A/s1600/Added+blank+packages.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://2.bp.blogspot.com/-G-JF_8muZz0/T5FsFV9N7GI/AAAAAAAAABM/FQ51V9ZRQ6A/s320/Added+blank+packages.png" width="320" /></a></div>
You could run the test runner, coverage and code analysis tools now, but the output wouldn't be very interesting. Instead, let's move on and add a failing test.<br />
<br />
<h3>
The first test</h3>
If you're a TDD type developer, this shouldn't come as a surprise. We'll write a test first. Then write some code in order to get the test to pass. In the tests directory, create a file named "<b><span style="color: blue;">authentication_tests.py</span></b>" and place the following code in it:<br />
<br />
<br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>#!/usr/bin/env python</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>"""Unit tests for the project1.authentcation module."""</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b><br /></b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>from unittest import TestCase</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>from mock import patch</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>import project1.authentication as auth</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b><br /></b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>class StandAloneTests(TestCase):</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> """Test the stand-alone module functions."""</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b><br /></b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> @patch('__builtin__.open')</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> def test_login(self, mock_open):</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> """Test the login function."""</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> mock_open.return_value.read.return_value = \</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> "george|bosco\n"</b></span><br />
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> self.assertTrue(auth.login('george', 'bosco'))</b></span><br />
<div>
<br /></div>
<div>
This is a test the will test a function named login that will reside project1.authentication module. If you run it now, it should fail because we haven't created an authentication module yet, much less a login function. To run the tests, simply type "<b><span style="color: blue;">nosetests</span></b>" from the project root directory. Nose will scan your whole directory structure looking for unit tests to run.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-BQgLaMUEw_k/T5Fw_NiYm1I/AAAAAAAAABU/02rPhBVL3yQ/s1600/Miserable+Failure.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://1.bp.blogspot.com/-BQgLaMUEw_k/T5Fw_NiYm1I/AAAAAAAAABU/02rPhBVL3yQ/s320/Miserable+Failure.png" width="320" /></a></div>
<h3>
</h3>
<h3>
Let's get the test to pass</h3>
<div>
Before we can move on to the topics of coverage and quality, we need to get the test to pass. To do this, create a file in the project1 directory named "<b><span style="color: blue;">authentication.py</span></b>" and place the following code in it:</div>
<div>
<br /></div>
<div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>#!/usr/bin/env python</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>"""This module provides functions for authenticating users."""</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>def login(username, password):</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> try:</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> user_file = open('/etc/users.txt') </b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> user_buf = user_file.read()</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> users = [line.split("|") for line in user_buf.split("\n")]</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> if [username, password] in users:</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> return True</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> else:</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> return False</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> except Exception, exc:</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> print "I can't authenticate you."</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> return False</b></span></div>
</div>
<div>
<br /></div>
<div>
Now when you run the test, everything should be hunky-dory.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-qA3tQFKi5og/T5F2KkL7tkI/AAAAAAAAABc/8S9GCdrCDm0/s1600/Passing+test.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://2.bp.blogspot.com/-qA3tQFKi5og/T5F2KkL7tkI/AAAAAAAAABc/8S9GCdrCDm0/s320/Passing+test.png" width="320" /></a></div>
<h3>
</h3>
<h3>
Code Coverage</h3>
<div>
You're probably thinking things look pretty good right about now. Let's run the the code coverage tool to see where we stand. To run a coverage test, use this command: "<b><span style="color: blue;">nosetests --with-coverage --cover-package=project1</span></b>". You'll find that we only have 67% coverage and that our test doesn't exercise lines 12-15 of our login method. Yikes.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-2f-siulWliU/T5F3hPoUheI/AAAAAAAAABk/VXQl3n_ik0o/s1600/Coverage+failure.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://4.bp.blogspot.com/-2f-siulWliU/T5F3hPoUheI/AAAAAAAAABk/VXQl3n_ik0o/s320/Coverage+failure.png" width="320" /></a></div>
<div>
You see... In our single test, we're only covering the <i>happy path(tm)</i>. That is, we only check to see what the function does when a user enters a valid username/password. We don't check what happens when they enter an invalid username/password. Additionally, we don't check to see what the function does when it can't read the user file at all. We would need to add more tests for that. Edit the <b><span style="color: blue;">tests/authentication_tests.py</span></b> file and make it look like this:</div>
<div>
<br /></div>
<div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>#!/usr/bin/env python</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>"""Unit tests for the project1.authentication module."""</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>from unittest import TestCase</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>from mock import patch</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>import project1.authentication as auth</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>class StandAloneTests(TestCase):</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> """Test the stand-alone module functions."""</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> @patch('__builtin__.open')</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> def test_login_success(self, mock_open):</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> """Test the login function when things go right."""</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> mock_open.return_value.read.return_value = \</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> "george|bosco"</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> self.assertTrue(auth.login('george', 'bosco'))</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> @patch('__builtin__.open')</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> def test_login_bad_creds(self, mock_open):</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> """Test the login function when bad creds are passed."""</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> mock_open.return_value.read.return_value = \</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> "george|bosco"</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> self.assertFalse(auth.login('george', 'marbleRye'))</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> @patch('__builtin__.open')</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> def test_login_error(self, mock_open):</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> """Test the login function when an error happens."""</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> mock_open.side_effect = IOError()</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> self.assertFalse(auth.login('george', 'bosco'))</b></span></div>
</div>
<div>
<br /></div>
<div>
Notice I've added two more tests: one to test invalid creds and one to test an exception occurring. Now we have 100% coverage.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-DRNRqa_3qwI/T5F6MNPtn4I/AAAAAAAAABs/MbDAqulxLvE/s1600/Coverage+success.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://2.bp.blogspot.com/-DRNRqa_3qwI/T5F6MNPtn4I/AAAAAAAAABs/MbDAqulxLvE/s320/Coverage+success.png" width="320" /></a></div>
<h3>
</h3>
<h3>
Code Quality</h3>
<div>
So we now have tests that pass and 100% coverage. That's awesome! Everything's under control now. What did you say? "How about the quality of that code?" Let's check it out and see. To take a look at code quality use the following command: "<b><span style="color: blue;">pylint -r n project1</span></b>". Ah. Well. We have a few things to clean up. Pylint is notoriously whiny. That said, the things it's complaining about in this case are legitimate if you're concerned about being a good python developer.</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-TJ6UlmsfXzg/T5F8eWcY9mI/AAAAAAAAAB0/htONSrvHEyc/s1600/Quality+attempt+1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://4.bp.blogspot.com/-TJ6UlmsfXzg/T5F8eWcY9mI/AAAAAAAAAB0/htONSrvHEyc/s320/Quality+attempt+1.png" width="320" /></a></div>
<div>
So let's get ourselves into compliance. First let's add a docstring to the project1 package. Edit <b><span style="color: blue;">project1/__init__.py</span></b> and make it look like this:</div>
<div>
<br /></div>
<div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>#!/usr/bin/env python</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>"""</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>Project1 is Steve's example project for the blog.</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>It contains sample code here and there.</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>"""</b></span></div>
</div>
<div>
<br /></div>
<div>
That takes care of the first pylint complaint. Next let's fix up the <b><span style="color: blue;">project1/authentication.py</span></b> file. Edit it and make it look like this:</div>
<div>
<br /></div>
<div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>#!/usr/bin/env python</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>"""This module provides functions for authenticating users."""</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b>def login(username, password):</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> """Log the user in."""</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> try:</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> user_file = open('/etc/users.txt') </b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> user_buf = user_file.read()</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> users = [line.split("|") for line in user_buf.split("\n")]</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> return [username, password] in users</b></span></div>
<div>
<b style="color: blue; font-family: 'Courier New', Courier, monospace;"> except IOError:</b></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> print "I can't authentication you."</b></span></div>
<div>
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><b> return False</b></span></div>
</div>
<div>
<br /></div>
<div>
Now when you run pylint, you should have no complaints:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-sWLw47eptZI/T5F-jG3fRqI/AAAAAAAAAB8/IyBoTc4cV0w/s1600/Quality+attempt+2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://4.bp.blogspot.com/-sWLw47eptZI/T5F-jG3fRqI/AAAAAAAAAB8/IyBoTc4cV0w/s320/Quality+attempt+2.png" width="320" /></a></div>
<div>
If you want a more detailed report from pylint, try using the command, "<b><span style="color: blue;">pylint -r y project1</span></b>" instead. Used that way, it's very much more verbose.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-2UDrj3vQrY8/T5F_qEIAUVI/AAAAAAAAACE/jiP3H39pvos/s1600/Quality+verbose.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="http://3.bp.blogspot.com/-2UDrj3vQrY8/T5F_qEIAUVI/AAAAAAAAACE/jiP3H39pvos/s320/Quality+verbose.png" width="178" /></a></div>
<div>
<br /></div>
<h3>
Let's clean up</h3>
<div>
So here's what we've accomplished so far. We created a project and source code repository. We've created a sample piece of code and some tests for it. We've ensured that we have 100% code coverage and we've analysed our code quality and found it to be in good shape. Let's pause and make sure that all our work is committed to source control.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-LfmeZGr6o68/T5GBMqR82XI/AAAAAAAAACM/doN4re8-rs0/s1600/Commit+code+to+git.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://4.bp.blogspot.com/-LfmeZGr6o68/T5GBMqR82XI/AAAAAAAAACM/doN4re8-rs0/s320/Commit+code+to+git.png" width="320" /></a></div>
<div>
<br /></div>
<div>
That's it for now. In the next post, I will walk through installing Jenkins and configuring it to monitor your source code repository. It will notice when you commit new code and automatically run your unit tests, code coverage and code quality analysis. It can be configured to notify you when things don't look so good. It can also display nice graphs over time of how your code is progressing. Stay tuned.</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-17425345906317136502012-04-20T09:11:00.000-04:002012-04-20T09:12:16.009-04:00Automated python unit testing, code coverage and code quality analysis with Jenkins - part 1As a software development consultant, I do a fair amount of <a href="http://www.python.org/" target="_blank">python</a> development and even though python is a scripting language, I feel that it's still worthy of good coding practices, namely: unit testing, code coverage analysis and code quality analysis. In this series of blog postings I will describe my python development environment and how I engage the tools.<br />
<br />
In this specific blog post, I will talk about the tools I use and how to get them installed/configured. In future postings I'll set up a sample code base with sample unit tests and run the tests manually. Finally in the last posting, I'll talk about using Jenkins to automate the testing.<br />
<br />
<b>Let's get started</b><br />
I tend to do most of my python development on the Linux platform. I love python's cross platform features. While most of my client implementations actually run in Microsoft Windows, I still do my development in Linux.<br />
<br />
For the purposes of this blog series, I will be using <a href="http://www.ubuntu.com/" target="_blank">Ubuntu</a> 11.10, the 64 bit version. This distribution/version isn't necessary, but that what I'm using. Python 2.7 comes with the default installation. If for whatever reason python isn't installed on your system, just type "<b><span style="color: blue;">sudo apt-get install python</span></b>" from the bash prompt:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-W-Bc67CNbwI/T5FZXIkkgoI/AAAAAAAAAAk/GKmsZl0Uyyw/s1600/Install+Python.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://1.bp.blogspot.com/-W-Bc67CNbwI/T5FZXIkkgoI/AAAAAAAAAAk/GKmsZl0Uyyw/s320/Install+Python.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<b>Source control</b></div>
<div class="separator" style="clear: both; text-align: left;">
When we move on to automating the testing/analysis, you'll need to have some source control in place. I use git. Even if you aren't planning on doing the automation part, source control is still a good idea. Again, you can get it with a single command: "<b><span style="color: blue;">sudo apt-get install git</span></b>"</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-zUIG3Lud4dI/T5Fb0VDTrNI/AAAAAAAAAAs/7jUn--_P8ik/s1600/Install+git.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://2.bp.blogspot.com/-zUIG3Lud4dI/T5Fb0VDTrNI/AAAAAAAAAAs/7jUn--_P8ik/s320/Install+git.png" width="320" /></a></div>
<br />
<b>Tools of the trade</b><br />
There are a number of testing, quality and coverage tools out there for python. These are the ones I use:<br />
<br />
Unit testing: <b><span style="color: blue;">unittest</span></b> - Comes with python. It's part of the standard library.<br />
Mocking: <b><span style="color: blue;">mock</span></b> by Michael Foord - It's available in apt.<br />
Test runner: <b><span style="color: blue;">nose</span></b> - Also available in apt.<br />
Code coverage: <b><span style="color: blue;">coverage</span></b> - Also, <i>also</i> available in apt.<br />
Code quality: <b><span style="color: blue;">pylint </span>-</b> Also, <i>also, also</i> available in apt.<br />
<br />
So to get all these great tools you simply need to type one command: "<b><span style="color: blue;">sudo apt-get install python-mock python-nose python-coverage pylint</span></b>"<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-2hShEFGW1mc/T5FeJU_8BUI/AAAAAAAAAA0/wQewqz1nVIg/s1600/Install+tools.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://3.bp.blogspot.com/-2hShEFGW1mc/T5FeJU_8BUI/AAAAAAAAAA0/wQewqz1nVIg/s320/Install+tools.png" width="320" /></a></div>
That's it. If you're following along, you should have a fully ready python development environment. Tune in to the next posting where I'll create a repository, some sample code and sample unit tests. Then I'll manually run the testing, coverage and quality analysis tools against it.<br />
<br />
<br />
<br />bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.comtag:blogger.com,1999:blog-4312481443349522150.post-56080735482472121532012-04-13T11:19:00.000-04:002012-04-13T11:19:49.969-04:00Copy and paste from vimI've added the following to my .vimrc:<br />
<br />
<textarea cols="80">map <Leader>c :w !xclip -sel clip<cr><cr>
map <Leader>p :r !xclip -o -sel clip<cr>
</textarea>
<br />
<div>
<br /></div>
<div>
It allows me to highlight text in a file and then hit '\c' to copy the highlighted text to my X11 clipboard in Ubuntu.</div>
<div>
<br /></div>
<div>
It also allows me to just hit '\p' to paste from the X11 clipboard to the vim buffer.</div>
<div>
<br /></div>
<div>
My .vimrc in OSX has the following to provide the same functionality when I'm on the Mac:</div>
<div>
<br /></div>
<div>
<div>
<textarea cols="80">map <Leader>c :w !pbcopy<cr><cr>
map <Leader>p :r !pbpaste<cr>
</textarea>
</div>
<div>
<br /></div>
<div>
<br /></div>
</div>bhfstevehttp://www.blogger.com/profile/07467841231478317132noreply@blogger.com