Improving My Git Workflow

Through various projects I’ve been working on in the last couple of years I’ve been digging deeper into Git. While many of Git’s basic operations are pretty straightforward, for someone getting used to Git it soon becomes apparent that some operations aren’t as simple as one might first expect, especially when using Git on the command line rather than with a GUI client.

I’ll expand on the topic later, but I wanted to create this stub article pointing to the GitHub repo I published a few days ago. It contains my annotated Git aliases (many of which I use as a quick reference as I haven’t yet committed all of them to memory), and a collection of helper scripts, many of which I have written myself, while few others, like git-wtf, I have not, but have included them for convenience. Also check out the README in the repo for alias dependencies and for other favorite Git extensions I frequently use.

Besides the enhanced Git command line, gitk with its visual blame tool (accessible from gitk, and also directly from the command line with gui blame when gitk is installed), Atlassian SourceTree, Scooter Software’s BeyondCompare (for diffs and merges), and tig for repository viewing on the command line are now solidly part of my everyday Git toolkit.

OpenVPN with FreeRADIUS: How To Use the CN from the User Cert as the Login Name (i.e. the reverse of “username-as-common-name”)

I recently set up handful of OpenVPN servers to provide access to various LAN and AWS VPC resources. Initially I had just the certificate validation configured, but I felt slightly uneasy about not having a password. Especially in the environments where multiple people need access to a resource, in the event one of them no longer should have access – such as when leaving an organization – the only way to block such user would be to add their cert into the CRL. While that should be done anyway when a user’s privilege needs to be revoked, a password would provide a more immediate and easy way to make such changes.

The next step was to install FreeRADIUS which proved to be a very easy task. I’m initially running it with just text-based back-end and will later add MySQL, perhaps with daloRADIUS GUI to make user administration even easier. On Ubuntu/Debian there is a package “openvpn-auth-radius”, which makes it possible to add FreeRADIUS authentication to OpenVPN server with one simple line:

(Of course, the client side also needs the auth-user-pass statement in their OpenVPN client configuration.)

But there is a problem: The user cert can be that of Bob while the login username/password is that of Alice, and the login would still be valid. Apparently I’m not the only one who has thought about this. While I didn’t want to hack the pam auth plugin, the post had enough clues to make a simple bash script (below) that sets the username based on the common-name from the validated user’s certificate:

To use this script, simply save it to /etc/openvpn/, make it executable, and edit the file to add the shared secret for the RADIUS server from /etc/freeradius/clients. Finally, add the following lines in your OpenVPN server configuration that already authenticates the users by their certificates:

tmp-dir /dev/shm
auth-user-pass-verify /etc/openvpn/ via-file

Now the login name for RADIUS authentication is taken from the CommonName (CN) of the user’s certificate and, in fact, the username that the user enters when prompted for auth-user-pass username/password is ignored, only the password is significant.

The bottom line of this script: It utilizes RADIUS to provide a server-side password validation for the certificate’s CN. A user can always remove the password protection from their private key, so this approach functions as an extra layer of security while making it easier to quickly revoke user’s access to a resource.

Note: For this to work, the CommonName set in user certificates obiously must be a valid RADIUS login name. A user can’t modify the CN in their certificate (unless they’re NSA since they apparently have access to RSA-keys, too :( ), so they’re locked to use that specific username.

Also note that I wrote this script on Ubuntu and did not necessarily observe portability, so you may need to modify the script some for other platforms. It is primarily intended as an example (although it does work), as finding something like this would have saved me a few hours of work.

Introducing duplicity-nfs-backup, or How to Use duplicity-backup Safely with NFS/CIFS Shares

After completing nfs_automount script a bit over a week ago, I soon realized rdiff-backup I had planned to use with the now-nearly-guaranteed-to-be-online NFS shares would not work. I then turned to my other favorite *NIX server backup solution, duplicity with wrapper script. It utilizes gzip-based archives, which works much better with NFS/CIFS shares. Besides the other odd problems with rdiff-backup and NFS, it resolves the more obvious issue with conflicting users/permissions between the client and the NFS share host as duplicity doesn’t maintain a direct mirrored copy of the files being backed up.

The only problem was that since duplicity creates incrementals, and I generally like to keep backups around for several months, the incrementals are really never needed beyond couple of weeks. Beyond that in my applications the day-by-day backups are overkill, and should be pruned. Duplicity provides an option to do so (“remove-all-inc-of-but-n-full”), but hadn’t implemented it, so I first contributed a patch to zertrin’s project. Then I proceeded to write a wrapper for the wrapper to add the extra pre-backup checks, and duplicity-nfs-backup was born.

So what is duplicity-nfs-backup? It is a wrapper script designed to ensure that an NFS/CIFS-mounted target directory is indeed accessible before commencing with backup. While can be used to back up to a variety of mediums (ftp, rsync, sftp, local file…), duplicity-nfs-backup is specifically intended to be used with NFS/CIFS shares as backup targets.

The script that was the impetus for writing duplicity-nfs-backup, nfs_automount, attempts to keep the NFS shares online at all times, but the client system can’t always help with such situations. What if the target system becomes unreachable due to a network problem? Or what if a disk, or a filesystem mount fails on the target while the share is still available? In any of these cases duplicity-backup/duplicity would back up into an empty mountpoint. duplicity-nfs-backup adds the necessary checks to ensure that this won’t happen, and it also issues log/syslog warnings when a backup fails due to a share that has gone M.I.A.

I mentioned earlier that duplicity-nfs-backup is “a wrapper for the wrapper.” Paraphrasing zertrin, it is important to note that duplicity-nfs-backup IS NEITHER duplicity, NOR is it duplicity-backup! It is only a wrapper script for duplicity-backup, also written in bash.

This means that you will need to install and configure duplicity and before you can utilize duplicity-nfs-backup. I also recommend that you would make use of nfs_automount as it significantly improves the chances that the NFS target share will be online when duplicity-nfs-backup attempts to access it.

This script is intended to be run from crontab. duplicity-nfs-backup takes no arguments, simply set the configuration parameters in duplicity-nfs-backup.conf and you’re done!

Like nfs_automount, duplicity-nfs-backup is also distributed under MIT license.

Clone or download duplicity-nfs-backup from my GitHub repository, and let me know if you come across any problems (or also if it works fantastically and saves the day! :)). Pull requests are always welcome.

NFS Automount, The Fourth Iteration (the complete rewrite)

** Note: This post has been significantly altered on 18 July 2013 from the original, posted a few days earlier.

A few days ago I released the fourth iteration of the NFS Automount script, with some minor changes to the previous version from December 2011. The earlier versions were released May 2011 (first CentOS Linux version), and July 2010 (originally written for FreeBSD).

Upon releasing the fourth version I realized the script was becoming brittle, the logic was, well, somewhat illogical, and minor refactoring would not help. Hence this complete rewrite of the script, now called “nfs_automount”, was born. It is conceptually based on the older versions, and I also borrowed some ideas from AutoNFS script on Ubuntu’s Community Wiki.

Like the earlier version, the goal of this script is to provide static (i.e. /etc/fstab-like) NFS mounts, while at the same time supporting cross-mounts between servers.

The other non-fstab alternative is to lazy-mount NFS shares with autofs (where available), but with it NFS shares are not continually maintained. When a remote share is accessed, it takes a few moments for it to become accessible as autofs mounts the share on-demand. While autofs times out a mounted share after some time of inactivity, it does not unmount the share before the timeout has lapsed in the event the remote server becomes inaccessible. While on-demand mounting may save some bandwidth, it is not suitable for all applications. Furthermore, when a system has one or more active mounted shares off of a server that goes offline, unexpected behavior is often observed on the client server until the now-defunct NFS shares are unmounted, or the remote server becomes available once again.

nfs_automount offers a solution:

  • The NFS shares are not statically defined in /etc/fstab so that the system startup is not delayed even when the remote server is not available. As soon as the shares become available they’re automatically mounted. If multiple servers cross-mount NFS shares from each other, and the servers are turned on at the same time, nfs_automount ensures that all mounts are established as soon as the shares become available.
  • The shares are monitored at a frequency you define, for example, every 60 seconds. If a share has become dismounted, stale, or their exporting server has become inaccessible, nfs_automount takes action to correct the situation: dismounted and stale shares are attempted to be remounted (stale shares are first immediately unmounted), and shares whose remote NFS service has disappeared are unmounted to prevent impact on the client system stability. Once a remote NFS service returns online, or definition of a previously stale share is reinstated, any shares that were unmounted as a result of those conditions are remounted.
  • The script is intended to run as a daemon (an upstart job script is provided for Ubuntu), and it reads its configuration from /etc/nfs-automount.conf where you can conveniently define the shares to be mounted and monitored along with some other options. You can also set ‘RUNTYPE’ option to ‘cron’, and run the script from crontab if you so choose.
  • You can define the shares to be mounted either as Read/Write, or Read Only. Of course, a share will be Read Only regardless of this setting if it has been exported as Read Only on the remote server.
  • An option to define a remote check file is provided. If provided in the configuration for a share, its unreachability can alert of a problem on the exporting server, such as a failed filesystem mount, even when the NFS share is otherwise working correctly. You can easily expand this feature to add additional functionality.
  • Provides clear logging which provides alerts by default, and more informative detail if you turn ‘DEBUGLOG’ setting to ‘true’.
  • Written in bash script with modular and clear syntax.
  • Tested on Ubuntu 12.x (should also work on Debian) and CentOS 6.x (should also work on RedHat). The service installation instructions (available on GitHub) have been written for Ubuntu, so if you’re installing the script for CentOS/RedHat, you will need to alter the installation steps somewhat. FreeBSD is no longer explicitly supported, but I believe it should work with minor modifications. I have not tested with Solaris or other *NIX environments. If you try, please post comments here!
  • Can be easily run as a service (upstart script is provided), or from crontab; the script works with crontab with just a single configuration switch change.
  • Distributed under MIT license.

Rather than posting the code (now 400+ lines) here, I have created a repository on GitHub from where it is easy to download or clone.

Enjoy! :)

Flexible LAN Name Server System with Partial Zone Overrides using BIND and Unbound

(“TL;DR”? Don’t bother! ;))

Since the early 2000’s I have been using BIND to provide name service for the LANs that I set up and/or maintain. Some time ago it became necessary to find a solution for a use case where an internal name server needed to be able to override and/or add some subdomains to an externally authoritative zones, while resolving the rest [of that zone] from its authoritative source (and, of course, provide internal resolution for the LAN’s internal zone(s)).

The only resolver that provides partial (“transparent”) override of an external zone is Unbound. However, Unbound is only an iterative resolver, and while it provides the option to inject “local zone data” which behaves as if it were authoritative (for all practical purposes), it is not such. Unbound is very efficient in what it does, and it’s secure, but to provide a fully functional name server it needs to be coupled with an authoritative server. I spent some time trying out NSD, considering whether I could adopt it as the auth zone server, especially since it’s also made by NLnet Labs like Unbound. HOwever, after some testing I opted to keep BIND as the authoritative server, mainly because in the LAN use the goal was to select the most versatile components – features trumped [possible] security benefits. In addition to my overall familiarity with BIND, I missed the ACLs, and the $GENERATE statement, both which modern BIND offers.

The next steps were to make BIND and Unbound to play well together, and to figure out how Unbound could be mirrored since it doesn’t natively include IXFR style zone transfers, or in fact any type of replication. If I were to use Unbound to partially override externally authoritative zones, I didn’t want to have to manually keep the overriding zone data in sync on multiple servers.

Below is a diagram of the complete system. For clarity’s sake only one mirror (“DNS2”) is illustrated; adding more mirrors is a trivial task.

(click the image for the full size version; a PDF version is also available)

Once I had set up BIND master and BIND slave, I set up the standard IXFR zone transfer between them, and tested that it works. Since it’s outside of the scope of this article, I let you research/figure it out on your own (if you’re not already familiar with the process). Once I got it working I set BIND to listen on a LAN address on a non-standard port 55. This is because Unbound that will be serving the DNS queries from the LAN will be listening on the default DNS port 53. BIND’s port 55 accepts queries from the Unbound server (in this case, the same server), and also from other LAN segments whose respective resolvers might forward queries for the zones this BIND installation is authoritative for. In some cases BIND also provides (reverse) resolution externally (access to internal zone resolution or recursion is prohibited by an ACL), and in such cases the firewall NAT translates the external port 53 to port 55 on BIND.

With Unbound configured I toggled its default setting with do-not-query-localhost: no since BIND instance resides on the same server. Without that directive Unbound won’t query authoritative zones on BIND at Now stub zones on Unbound are able to complete queries for zones for whom BIND is the authoritative server. The Unbound stub entry looks like this:

You will notice the “##[terminator##” segment in the end of the stub zone file above. Since I like to break out my stub, forward, and “extended” (override) zones into individual config files, the “server:” statement terminates the local-zone segment allowing other content to follow in the unbound.conf from where these are included.

A extended, or “override” zone file (the impetus for this whole excercise), is equally simple:

With the above simple example the zone “” will resolve normally along with its any an all defined subdomains (, etc.), except for “” which is now valid on your LAN with an IP While just an example, such overrides tend to be more useful when applied, for example, to company domains where you need to add subdomains on the internal LAN for intranet, development, etc. while relying on the default external authoritative source for the remainder of the zone.

Now for the interesting part: Implementing the Unbound mirroring. Since Unbound doesn’t provide IXFR or other methods for zone transfer/replication, I have created a handful of shell scripts that take care of the task. At the time of writing of this I’m running this setup on Ubuntu 12.04LTS servers and hence the following prerequisite utilities are available: iwatch, lockfiles-progs, and rsync.

On the master server service “unbound-push-service” is installed. When changes occur (unbound is restarted), the service detects the change and executes another script (trigger-unbound-rsync) which with the help of rsync synchronizes Unbound zone data to the mirror(s). On the mirror(s) a service “unbound-restart-service” is installed. The service utilizes iWatch to detect content changes in the Unbound zone data folder. When changes are detected, “trigger-unbound-restart” is triggered. After a wait for a few seconds (to ensure that the transfer is complete), the script command “unbound-trigger-command” is run and Unbound is restarted, hence making the updated zone data live.

For the above to work the prerequisite utilities must be installed, a rsync (ssh) key needs to be configured between the master and the mirror(s), the monitoring services must be installed on source and the mirror, and the Unbound zone files must be separated to a folder of their own. I’ll step you through the process below with examples and necessary scripts along the way.

First, make sure the prerequisite utilities are installed on both the master and the mirror (note: all tasks below assume you’re running as a root; if not, add sudo as needed):

Copy the following utility scripts to the master (I use /opt/unbound-push/ as the folder, but you can choose a location that suits you best):



Copy the following utility scripts to the mirror (I use /opt/unbound-trigger/ as the folder, but you can choose a location that suits you best):




So far so good. Now that all the prerequirements are in place; let’s do some configuration. The following assumes the above mentioned locations (/opt/unbound-push on the master and /opt/unbound-trigger on the mirror) for the script/service files.

On the Master:

  1. Symlink /opt/unbound-push/unbound-push-service from /etc/init.d:
    ln -s /opt/unbound-push/unbound-push-service /etc/init.d
  2. Install the service:
    update-rc.d unbound-push-service defaults
  3. Create an RSA key specifically to push content via rsync to the mirror server:
    ssh-keygen -f /root/.ssh/dns-unbound-sync.id_rsa
    ** Choose no password to protect the private key! **
  4. Copy the public key to the mirror server via scp
  5. Add to (or create if the file doesn’t exist) /root/.ssh/config:
  6. Double-check rsync configuration in /opt/unbound-push/trigger-unbound-rsync for the key file name, target IP (IP may be preferable over a domain name since this is part of the DNS fabric; this needs to work regardless even if resolution is not [yet] working), etc.

And on the mirror:

  1. Symlink /opt/unbound-trigger/unbound-restart service from /etc/init.d:
    ln -s /opt/unbound-trigger/unbound-restart-service /etc/init.d
  2. Install the service:
    update-rc.d unbound-restart-service defaults
  3. Modify the unbound user’s shell with vipw: /bin/false -> /bin/bash
  4. Make sure /etc/unbound and its contents are owned by unbound.unbound:
    chown -R unbound.unbound /etc/unbound
  5. Create “.ssh” directory for unbound user, make sure it is owned by unbound.unbound, and that its permissions are set to 700:
  6. Move the public RSA key for “unbound” user from the path where you transferred it to from the Master server above (see “On the Master, #4”), make sure it’s owned by unbound.unbound, and set its permissions to 600. The target file name, “authorized_keys2” is significant.
  7. If /etc/ssh/sshd_config limits which users can log in, the transfer user (which is “unbound” if you followed the above steps) must be allowed to log in.

Mirroring configuration is now complete! Since the scripts above monitor specific directory structure under /etc/unbound, I should quickly review it before we proceed to start the services and test the zone mirroring.

As I mentioned earlier, I have local data (stub, forward, and extended zone) broken into separate files. The structure under /etc/unbound looks like this:

/etc/unbound/ – main configuration
/etc/unbound/conf.d – user configuration (local data includes, access control list); monitored and synced to mirror
/etc/unbound/zonedata – local data files; monitored and synced to mirror

Here’s an example unbound.conf:

The above configuration includes couple of files:

/etc/unbound/conf.d/access-control.conf (here you define IPs/networks that are allowed to query/recurse through this Unbound instance):

/etc/unbound/conf.d/zones.conf (here you include the stubs/forwarders/overrides; you could also instead include the files directly from the zonedata folder, but at the time of writing of this Unbound doesn’t yet support inclusion by wild card, though it is coming. I find explicit inclusions safer, though).


/etc/unbound/zonedata/ (NOTE: the ‘nodefault’ entries for local-zones are significant!):

/etc/unbound/zonedata/ (e.g. a zone from another ‘internal’ network such as another LAN):

/etc/unbound/zonedata/ (override remote zone partially):

In the zonedata directory I have a reminder to include files that are added there:

.. and a quick template for the file types:

Finally, let’s start the monitoring/triggering services and do some testing. On Master execute service unbound-push-service start, and on the slave execute service unbound-restart-service start.

Test SSH connectivity from master to slave. When logged in as root, you should be able to connect to the mirror as the unbound user simply by typing “ssh”. Test rsync from master to mirror (build your own rsync command, or use one from trigger-unbound-rsync file.

Once the above tests are completed successfully, test the DNS mirroring by adding an empty file in /etc/unbound/zonefiles on the master, and then restart unbound; now observe: 1) rsync push on the master, 2) initiation of the unbound-trigger-command with 10 second delay on the mirror, and ultimately 3) changing PID of the Unbound service on the mirror as it is automatically restarted, hence bringing the mirrored changes live also on the secondary server.

If it doesn’t appear to be working, logs are your friend. Start by looking at a small transfer log on the master at /opt/unbound-push/push.log, then review ssh/syslogs on both servers, and finally unbound/BIND logs on both systems.

Whoa! That was lots of content! Maybe I should’ve published this as a book? 😀 I hope this proves useful for someone down the line. I have the above configuration running on half a dozen networks, and once set up it has been very peformant and highly stable.

Integrating Shorewall and sshguard on Ubuntu

I’ve been getting familiar with Shorewall since last fall, and quite like it. I also use sshguard to prevent incessent knocking on SSH and FTP ports. Or I thought I was using sshguard, after all, it was running. Then I realized that I had set up and tested sshguard way before I started using Shorewall, and as my understanding of how Shorewall functions gradually incrased, it eventually occurred to me that Shorewall likely overwrites the ‘placeholder’ chain sshguard needs to function.

I was looking for an easy way to view and occasionally delete sshguard rules, and came across Mika’s post that outlined two handy command line aliases that accomplish just that. While starting to test these, I noticed the ‘sshguard’ chain was nowhere to be found.

My first attempt was to load the sshguard chain with iptables-restore in pre-up, but that disabled the entire network! 😀 I suppose sshguard chain needs to come after Shorewall chains/rules have been initialized. While I’m not sure if this is the best way to get it working, this does work: I added two lines from sshguard sample ruleset in /etc/rc.local:

Importing Installer ISOs to ESXi 5.x Datastore

VMware ESXi 5.x service console no longer offers ftpget command. But all is not lost.. :) Follow the steps below to transfer your favorite OS installer ISO to the datastore from where you can conveniently mount it as if it were DVD/CD-ROM:

  1. Download 64-bit ncftp binary distribution to your local (Windows) workstation.
  2. Open vSphere, highlight the hostname, select the the desired storage unit from the Storage list under Resources (I usually use the datastore on the “system” disk for this purpose), and right click on it to open the context menu, then choose “Browse Datastore”.
  3. With the root highlighted, click on the “Upload files to this datastore” (the fourth icon from the left, with green “up” arrow). Select the ncftp binary package (that you download in step 1) from your local Windows hard drive.
  4. After making sure SSH login is enabled on the Yellow and Grey screen (Troubleshooting Options > Enable SSH), log in to the service console via SSH.
  5. Move the ncftp binary archive to /opt from the Datastore root, decompress it, and cd to the ncftp folder:
    cd /opt
    mv /vmfs/volumes/yourdatastore/ncftp* .
    tar zxvf ncftp*
    cd ncftp*
  6. Open ESXi firewall egress (outbound) rules so that the FTP client can access the outside FTP server:
    esxcli network firewall ruleset set -r ftpClient -e true
  7. With your favorite installer ISO available on an FTP server (perhaps on the distribution server, or on your local FTP server), you can now access it, like so:
    ./ncftpget -v -E -u ftpuser -p password /vmfs/volumes/mydatastore ‘/remotefilename.iso’

    Command from left to right:
    ncftp command (requires path, here “./” since you’re in the ncftp directory)
    -v for progress indicator
    -E for NOT passive mode (omitting it makes connection passive)
    -u ftpuser for remote ftp server
    -p password for remote ftp server (note: this is NOT sent encrypted!) is the ftp host name or IP address
    /vmfs/volumes/mydatastore is the path where you want to store the data on the local system
    /remotefilename.iso is the remote file (with full remote path if needed) to be transferred; the local file name will be the same

  8. Optinally, if you want to close the firewall egress rule for the FTP client, you can do so by issuing the following command:
    esxcli network firewall ruleset set -r ftpClient -e false

Now, when starting OS installation in ESXi you can highlight a created blank server instance, click on the CD-ROM icon on the toolbar (“CD/DVD drive 1” > “Connect to ISO image on a datastore…”), and then navigate to and select the ISO image you just transferred to your favorite datastore!