Showing posts with label bash. Show all posts
Showing posts with label bash. Show all posts

Wednesday, February 8, 2017

Non-freeze, single file distribution of your python project

In my job, I often have to share/distribute Python code to others on my development team or to others in sibling development teams.  These folks are technical and I can easily rely on them to have a reasonably contemporary version of Python already installed.  Thus I don't really want to completely freeze my Python project (via cxFreeze, py2app, py2exe, etc.).  Nor do I want to send my coworkers through the 'create a virtualenv, pip install -r, etc.).

I want a middle-ground.  I want to just send them a file, knowing that it contains all the required dependencies and will just run, assuming they have Python in their path.

I've known for a while that if you zip up a directory full of Python files and so long as the root of the zip file contains a __main__.py file and you can directly run the zip file via python {filename.zip}.

What I didn't know is that you can concatenate a text file containing the Python shebang ('#!/usr/bin/env python') and a python zip file and the resulting file is runnable.

#!/usr/bin/env python
PK^C^D^T^@^B^@^H^@nLHJb±gL<94>^A^@^@Ò^B^@^@^K^@^\^@__main__.pyUT    ^@^C¯,<9b>Xö,<9b>Xux^K^@^A^D«<82><90>~^D^T^@^@^@u<92>ÑkÛ0^PÆßýWܲ^GÙÐ$},^EÃÂ(<85><90>ìq ^Tûâ<88>Z<92>9<9d>º<95>±ÿ}';ÎJ`Â^O<96>ôéû~ºÓ
ÈY©1^F<93>9£4@^Q&<81><8e>O^X<88>2#tCrÌ^N0©¸<83>ÛÑw°ÌÕÂ#®Ë¾^A^PÅ >«¼h@^P;c6<8d>"^Âý=^\AB!ej<91>  <9e>g¦ñ<95>Vgé<87>ÌqÏ^RC<8a>9<99>aq^O^O<9f>Cv^U^AþXQi<88>ÛÌ®°Üx<90>µó<98>^Gl­ ´Z^B<81>^E·¾&^B^C<83>ÕÚ§
,g^LÃ=kW^Y$r­â Æ]HÉ^UÝ5AÂ[V^FC$y<80>¢a<82>y^V^U{°(Ô<9b>^XW}¤h<8f>aØ}¦^G>o<96>êu<9c>e?k<8e>,àbºÔ/ýëh´¶^LÇp^A^Ye<8b>LÕBt<85>w£hi¿&(^HÛîä<96>tÿ.^@A>
dJuëX

I stumbled upon a tool called pex which will automate generating these shebang/zip files.  While I found pex useful, it seems too much for my needs.  So I started cobbling together my own shebang/zip files by hand as I needed them.

Ultimately, wanted to automate this some, so I ended up creating a simple bash script (which I named 'compile.sh') to speed things up.  Here's that script.  Hope you find this useful.

#!/bin/bash
#
# compile.sh

# Creates a self-runnable, single-file deployable for your python project.
# Your python project's main entry point must be in a file named __main__.py.
# It will embed in the deployable every dependency you've pip-installed in
# your virtualenv .env directory.
#
# The deployable still requires that python is installed on the target system.

if [[ ! -d .env/lib/python2.7/site-packages ]]; then
    echo "This script must be run from a virtualenv root."
    exit 1
fi

if [[ ! -f __main__.py ]]; then
    echo "This will only work if you have a __main__.py file."
    exit 1
fi

if [[ -z "$1" ]]; then
    echo "Usage: $(basename $0) {output_filename}"
    exit 1
fi

TMPDIR="$(mktemp -u /tmp/python.compile.XXXXXXX)"
CURDIR="$(pwd)"
TARGET="$1"

# Create payload zip with our code (remove extra junk)
mkdir -p "$TMPDIR/deps"
zip -9 "$TMPDIR/payload.zip" *
zip -d "$TMPDIR/payload.zip" compile.sh
zip -d "$TMPDIR/payload.zip" requirements.txt
zip -d "$TMPDIR/payload.zip" "$TARGET" > /dev/null

# Gather the virtualenv packages and clean them up
cp -R .env/lib/python2.7/site-packages/* "$TMPDIR/deps"
find "$TMPDIR/deps" -iname '*.pyc' -delete
find "$TMPDIR/deps" -ipath '*pip*' -delete
find "$TMPDIR/deps" -ipath '*easy_install*' -delete
find "$TMPDIR/deps" -ipath '*dist-info*' -delete

# Add the virtualenv packages to the payload
cd "$TMPDIR/deps"
zip -9 -r ../payload.zip *
cd "$CURDIR"

# Assemble the payload into a runable
echo '#!/usr/bin/env python' | cat - "$TMPDIR/payload.zip" > "$TARGET"
chmod +x "$TARGET"

# Cleanup
rm -r "$TMPDIR"

Wednesday, January 22, 2014

Formatting GUIDs with sed

At my current client, I'm often given GUIDs for one reason or another.  Often they are not in the format that I need them to be.  Sometimes they have the dashes and I need them to be dash-free.  Other times, the dashes have been removed and I need them back.

Here's some quick, down and dirty sed commands to swap them back and forth:

To remove the dashes:

$ echo 8EC60070-685F-41DB-C881-EACF9E74E4BD | sed 's/-//g'
8EC60070685F41DBC881EACF9E74E4BD
$

To put them back:

$ echo 8EC60070685F41DBC881EACF9E74E4BD | sed -rn 's/([0-9A-F]{8})([0-9A-F]{4})([0-9A-F]{4})([0-9A-F]{4})([0-9A-F]{12})/\1-\2-\3-\4-\5/p')
8EC60070-685F-41DB-C881-EACF9E74E4BD
$

It goes without saying you can embed this very sed command in a bash script and run it over many rows in a file.

Friday, July 6, 2012

CGI Script to display clone urls


After some discussion, I was able to convince my client to convert from gitorious to gitolite.  One nice feature that people like about gitorious is that the web interface provides an easy way to look up urls for cloning repositories.

In my mind, that's a pretty legitimate need.  To that end, I threw together this bash script, which acts as a cgi and dropped it in the cgi-bin directory of the server that's running gitolite.  Hope you find this helpful.

#!/bin/bash

REPOSITORY_DIR="/home/git/repositories/"
URL_PREFIX="git clone git@internal-build-server:"

cat <<DONE
Content-type: text/html

<html>
<head>
<title>Repository URLs</title>
<link rel="stylesheet" type="text/css" href="/index.css" />
</head>
<body>
<div id="page_container">
<h1>
Repository URLs
</h1>
<p>
Repository URLs on this server follow a specific pattern.  The pattern is
as follows:
</p>
<center>
git@internal-build-server:<i><font color="darkblue">{category}</font></i>/<i><font color="darkblue">{project}</font></i>
</center>
<p>
These URLs are both pull and push URLs.  You do not need separate URLs
for pulling and pushing.  Access control will be handled by a
server git update hook that is provided by gitolite.
</p>
<p>
In an effort to make life a little easier in locating your URLs, this script
enumerates URLs for the repositories located on this machine below.
</p>
DONE

CATEGORIES=$(find $REPOSITORY_DIR -type d -maxdepth 1 -mindepth 1 -not \
-iname '*.git' | sed -e "s|$REPOSITORY_DIR||g")

for CATEGORY in $CATEGORIES; do
echo "<h2>Category: $CATEGORY</h2>"
CAT_REPOSITORIES=$(find $REPOSITORY_DIR$CATEGORY -type d -iname \
'*.git' | sed -e "s|$REPOSITORY_DIR||g" -e 's/.git$//g')
for REPOSITORY in $CAT_REPOSITORIES; do
echo "$URL_PREFIX$REPOSITORY<br />"
done
done

ROOT_REPOSITORIES=$(find $REPOSITORY_DIR -type d -maxdepth 1 -mindepth 1 \
-iname '*.git' | sed -e "s|$REPOSITORY_DIR||g" -e 's/.git$//g')
echo "<h2>Uncategorized Repositories</h2>"
for REPOSITORY in $ROOT_REPOSITORIES; do
echo "$URL_PREFIX$REPOSITORY<br />"
done

cat <<DONE
<br />
</div>
</body>
</html>
DONE

Monday, June 25, 2012

Bash script for pulling/fetching multiple git clones


In my current assignment, I'm acting as the main build guy for a number of projects that use git for source control.  As such, I find it very useful to keep all my git clones up to date whether I'm actively developing in them or not.  Additionally, I need to review the changes other developers are committing, so I'd like to get a summary of recent git activities.

Over time, I've put this little bash script together to help me with that.  I've included the script in this posting so I can remember later what/why I did this.  Disclaimer, I wrote this script and run this script in bash on Linux (not via git-bash in Windows).  Also, I'm using Zenity for some nice UI look/feel.

  1 #!/bin/bash
  2
  3 pushd ~/dev/repos > /dev/null
  4
  5 # The log file
  6 PULL_LOG="$(mktemp)"
  7
  8 # Get a list of all the clones in this directory.
  9 CLONES=$(find -maxdepth 2 -mindepth 2 -type d -name ".git" | sed -e 's|\./||' -e 's|/\.git||')
 10
 11 # Get a list of all the branches in clone/branch format
 12 ALL_BRANCHES=$(for clone in $CLONES; do cd $clone; for branch in $(git branch -l | sed 's/\s\|\*//g'); do echo $clone/$branch; done; cd ..; done)
 13
 14 # Count the branches
 15 BRANCH_COUNT=$(echo $ALL_BRANCHES | sed 's/ /\n/g' | wc -l)
 16
 17 # Start the log file
 18 echo "Pull log for $(date)" >> $PULL_LOG
 19 echo "--------------------------------------------------------------------------------" >> $PULL_LOG
 20
 21 # Function for pipping output to zenity progress dialog
 22 function pull_clones() {
 23     clone_counter=0
 24     for clone in $CLONES; do
 25         echo "Pulling branches for clone $clone" >> $PULL_LOG
 26         echo "--------------------------------------------------------------------------------" >> $PULL_LOG
 27         cd $clone
 28         echo "# Fetching changes for clone $clone"
 29         git fetch origin 2>> $PULL_LOG
 30         for branch in $(git branch -l | sed 's/\s\|\*//g'); do
 31             echo "# Merging branch $clone/$branch"
 32             echo "Merging branch $branch" >> $PULL_LOG
 33             git checkout $branch 2> /dev/null
 34             git merge origin/$branch >> $PULL_LOG
 35             echo | awk '{print count / total * 100}' count=$clone_counter total=$BRANCH_COUNT
 36             let clone_counter=clone_counter+1
 37         done
 38         cd ..
 39         echo >> $PULL_LOG
 40     done
 41 }
 42
 43 # Do it
 44 pull_clones | zenity --progress --title='Pulling development clones' --width=512
 45 zenity --text-info --filename=$PULL_LOG --title="Pull log" --width=500 --height=450
 46
 47 #Clean up
 48 rm $PULL_LOG
 49
 50 popd > /dev/null



Thursday, May 24, 2012

Some light string manipulation in bash

This post is more for me than you.  I had to do a little bash work this morning and thought I'd keep a record of some of the syntax and concepts that I used in case I have to do more in the future.

There are two files here: a function library and a script that consumes it.  The function library is used to do different kinds of string justification (left justify, right justify and centering).  Kind of like 'echo' on steroids.  This could have easily been done in python, perl or ruby, but I wanted it to stay in bash this time to keep the calling scripts uniform.

Here's the sample file that consumes the library:

#!/bin/bash

. echo_helper

x='Steve is here!'
left_justify "$x" 40 .
right_justify "$x" 40 .
center_justify "$x" 40 .

echo $(pad_string 40 -)

x='Steve was here.'
left_justify "$x" 40 .
right_justify "$x" 40 .
echo $(center_justify "$x" 40 .)

And here's what the output should look like:

Steve is here!..........................
..........................Steve is here!
.............Steve is here!.............
----------------------------------------
Steve was here..........................
.........................Steve was here.
............Steve was here..............

Finally, here's the echo_helper library code:



#!/bin/bash



# Syntaxes to remember:
#
# String length of a variable:  ${#varname}
# String offsetting of a variable: ${varname:offset} or ${varname:offset:length}


# Usage: pad_string LENGTH {PADDING_CHARACTER=' '}
function pad_string {
local line=''
local length=$1
local padding_character=' '
if [[ "$2" != "" ]]; then
padding_character=$2
fi
while((${#line} < $length)); do
line="$line$padding_character"
done
echo -n "$line"
}


# Usage: left_justify MESSAGE LENGTH {PADDING_CHARACTER=' '}
function left_justify {
local message=$1
local length=$2
local padding_character=' '
if [[ "$3" != "" ]]; then
padding_character=$3
fi
echo "$message$(pad_string $length-${#message} $padding_character)"
}


# Usage: right_justify MESSAGE LENGTH {PADDING_CHARACTER=' '}
function right_justify {
local message=$1
local length=$2
local padding_character=' '
if [[ "$3" != "" ]]; then
padding_character=$3
fi
echo "$(pad_string $length-${#message} $padding_character)$message"
}


# Usage: center_justify MESSAGE LENGTH {PADDING_CHARACTER=' '}
function center_justify {
local message=$1
local length=$2
local padding_character=' '
if [[ "$3" != "" ]]; then
padding_character=$3
fi
local half_length=$(expr $length / 2)
local half_message_length=$(expr ${#message} / 2)
local padding_length=$(expr $half_length - $half_message_length)
padding="$(pad_string $padding_length $padding_character)"
message="$padding$message$padding"
if [[ ${#message} > $length ]]; then # handle even/odd length issue
echo "${message:1}"
else
echo "$message"
fi
}

Tuesday, May 22, 2012

Fun with gnu screen

In my day job, I spend a fair amount of time working on numerous tasks on the command-line in Linux and OSX.  Way back in my college days, I used a program called gnu screen (aka 'screen') but recently I've "refound" it.  I have a new appreciation for screen and now use it every day.

What is screen?

Screen let's you have multiple shell sessions running simultaneously in the same terminal window.  Here's some of the features that I personally find appealing:
  • With a little muscle memory, you can switch rapidly between the sessions.
  • It has the ability to split up your terminal window into multiple "window-splits" so you can see two or more shells sessions at the same time.  The splits are even resizable.
  • You can 'detach' and 'reattach' to a screen session, allowing you to sign-out, leave and come back later, all the while keeping your terminal sessions active and running.
  • It's configurable and scriptable.
  • You can copy and paste between sessions with 'vi' like keystrokes.

How do you get screen?

Well, if you're running ubuntu like me, it's pretty darn easy.  You're one command away from having it:

steve@ubuntu64 ~ $ sudo apt-get install screen

Pretty simple.

Getting started

Once you have screen installed, simply run it.

steve@ubuntu64 ~ $ screen

It will bring you to a welcome screen (I'll show you how to disable the welcome screen later).


At the bottom of the welcome screen, it will say

[Press Space or Return to end.]

Go ahead and do that.  You'll be returned to a bash prompt.  It's important to note that screen hasn't terminated, but rather it's spawned another shell prompt within itself and you're sitting at that new prompt. You can now begin using screen via keyboard command sequences.  To prove it, press Ctrl-a and then press " (double quote).  You should get a screen that looks like this:

This is the terminal session list.  If you have more than one terminal session running, you'd seem them all listed here.  Let's create another terminal session.  Press Ctrl-a and then press c.  This should take you back to a bash prompt.  Your now in a new bash prompt.  See see the updated list, press Ctrl-a and " again.

Now there's two bash sessions.  You can select the one you'd like to see by using your arrow keys (or vi cursor movement keys) to highlight the one you want and then press enter/return.

When you want to quit out, simply exit out of all your bash prompts.  Once the last bash session ends, screen will automatically terminate, leaving where you were when you initially started it.

My most used key sequences

As you've probably realized by now, screen uses Ctrl-a plus a key to drive it's functionality.  Here's a list of key sequences that I use most often.

Ctrl-a Ctrl-a - Toggles between two sessions.  It's kind of like the "last" button on your TV remote.

Ctrl-a c - Creates a new terminal session.

Ctrl-a A - Let's you set the title of a terminal session.  The title is what appears in the session list.

Ctrl-a " - Lists the terminal sessions.  You can then use arrow keys or vi keys to select which session you wan to go to.

Ctrl-a d - Detach screen so you can log off and come back later.  Later when you sign back on, just type screen -r to reattach to your running screen session.

Ctrl-a : - Bring up screen's colon prompt (similar to vi).  From the colon prompt you can issue commands directly to screen.  (You'll probably use it most resizing window splits).

Ctrl-a [ - Start copy mode (for copy/paste).

Ctrl-a ] - Paste what you copied from copy mode.

Ctrl-a S - Split the screen horizontally.

Ctrl-a | (pipe) - Split the screen vertically.

Ctrl-a {tab} - Move focus from window split to window split

Ctrl-a X - Close the current window split.

Split example

Here a quick example of doing a couple window splits.  First, start screen and a few terminal sessions:

steve@ubuntu64 ~ $ screen
{return}
Ctrl-a c
Ctrl-a c
Ctrl-a "

Your terminal window should look like this:

Now press Ctrl-a S to split the screen.  You should now have an upper split and a lower split.  Press {return} now to select terminal session 2 in the upper split.  In that terminal session run the top command.

Now press Ctrl-a {tab}.  This will move the cursor down to the bottom split.  Then press Ctrl-a " to bring up the session list.

Select session 0 and then issue the following command in the lower split: watch ls.

Next, let's split the lower window vertically.  Press Ctrl-a | (pipe), Ctrl-a {tab}, Ctrl-a ".  Select session 1.

As you can see, you can keep splitting, sub-splitting, etc.  Remember, Ctrl-a {tab} moves the cursor to the next window split.  That's how you move around.

When you're done, you can use Ctrl-a X to close the window splits.  This does not quit/kill the process running in that split.  The process stays in the session list until you exit it normally.

Scripting

Screen reads and processes the ~/.screenrc file at startup.  Here's mine:

vbell off
startup_message off
defscrollback 1000
bindkey -k k1 colon
bindkey -k k2 screen
bindkey -k F2 windowlist -b
hardstatus off
caption always '%{Yb}[%=%t%=]'
altscreen on
shell -${SHELL}

You can go through the screen man page for what each line does.  You can customize screen however you'd like.

There are two features that combined make startup processing especially interesting:
  1. You can use the -c command line argument to give screen an alternate rc file to process on startup.
  2. Using commands, you can do anything in the rc file that you could do interactively.
Using this notion, I create different screen rc files for different contexts.  For example, when I'm working at a certain client, I would like to have screen set up a certain way, with specific terminal sessions setup.  I create a bash aliases for each client.

In my ~/.bash_aliases file I have:

alias clientX='screen -c ~/screen_profiles/clientX'
alias clientY='screen -c ~/screen_profiles/clientY'

My ~/screen_profiles/clientX file looks like this:

altscreen on
vbell off
attrcolor b ".I"
termcapinfo xterm 'Co#256:AB=\E[48;5;%dm:AF=\E[38;5;%dm'
defbce "on"
defscrollback 1500
bindkey -k k1 colon
bindkey -k k2 screen
bindkey -k F2 windowlist -b
screen -t "Home" bash
screen -t "Build" ssh my_username@build_server
screen -t "Dev2" ssh my_username@dev_server
screen -t "WebLogic" my_username@weblogic_server
screen -t "Todo/Notes" vi ~/notes/todo.markdown
hardstatus off
caption always '%{Yb}[%=%t%=]'
select 0
split -v
resize +20
focus
select 5
focus

This rc file does the following:
  • Does some normal setup that I do anyway (key bindings, colors, etc.)
  • Creates a terminal session titled, "Home" and runs bash in it.
  • Creates a terminal session titled, "Build" and ssh's me into the build server.
  • Creates a terminal session titled, "Dev2" and ssh's me into the dev 2 server.
  • Creates a terminal session titled, "WebLogic" and ssh's me into the weblogic server.
  • Creates a terminal session titled, "Todo/Notes" and edits my ~/notes/todo.markdown file using vi in it.
  • Splits the window vertically, resizing the left window to be 20 columns bigger.
  • Selects the "Home" session for the left window.
  • Selects the "Todo/Notes" session for the right window.
  • Puts the cursor focus on the left window.
So when I come into clientX's office for the day, I simply type clientX at the bash prompt and within a second, I have screen all set up with my splits and automatically logged into all the servers I commonly use.  Pretty cool.