I had some Ruby code that utilized File::SEPARATOR and File::PATH_SEPARATOR to run on both unix and windows, so I wanted to stub these values to test for both platforms. There are couple examples out there, building on each other. This example adds a feature that saves and recalls the former value and this example builds on that to support class constants. Both expect Activerecord, so there’s a little working around that added here. I’m ripping this directly from my spec_helper.rb before I throw it away because it feels over-engineered and complicated.

def with_warnings(flag)
  old_verbose, $VERBOSE = $VERBOSE, flag
  yield
ensure
  $VERBOSE = old_verbose
end

# http://missingbit.blogspot.com/2011/07/stubbing-constants-in-rspec_20.html
def parse_constant(constant)
  source, _, constant_name = constant.to_s.rpartition('::')

  [constantize(source), constant_name]
end

def with_constants(constants, &block)
  saved_constants = {}
  constants.each do |constant, val|
    source_object, const_name = parse_constant(constant)

    saved_constants[constant] = source_object.const_get(const_name)
    with_warnings(nil) {source_object.const_set(const_name, val) }
  end

  begin
    block.call
  ensure
    constants.each do |constant, val|
      source_object, const_name = parse_constant(constant)

      with_warnings(nil) { source_object.const_set(const_name, saved_constants[constant]) }
    end
  end
end
####################

# File activesupport/lib/active_support/inflector/methods.rb, line 209
def constantize(camel_cased_word)
  names = camel_cased_word.split('::')
  names.shift if names.empty? || names.first.empty?

  constant = Object
  names.each do |name|
    constant = constant.const_defined?(name) ? constant.const_get(name) : constant.const_missing(name)
  end
  constant
end

Then you can perform:

  it "does something when running on Windows" do
    with_constants "::File::PATH_SEPARATOR" => ";" do
      # code
    end
  end

I had a need to grab all of the Github repositories for Cookbooks, which is a Github user maintained by the Chef community for collecting many cookbooks in one place for development. All of these cookbooks should be on the Opscode Community site, which is where you should go if you’re browsing for cookbooks to use yourself. But I needed to grep through a large number of cookbooks to develop statistics on Chef Cookbook usage patterns, so I needed All The Things.

#!/usr/bin/env ruby
# 2012-01-11 Bryan McLellan <btm@loftninjas.org>
# Fetch the list of repositories from a Github user and 'git clone' them all

require 'rubygems'
require 'json'
require 'net/http'

url = "http://github.com/api/v2/json/repos/show/cookbooks"
dir = "cookbooks"

if File.basename(Dir.getwd) != dir
 if File.exists?(dir)
   puts "Target directory of '#{dir}' already exists."
   exit 1
 end

 Dir.mkdir(dir)
 Dir.chdir(dir)
end

resp = Net::HTTP.get_response(URI.parse(url))
data = resp.body

result = JSON.parse(data)

result['repositories'].each { |repo|
 puts "Fetching #{repo['url']}"
 system "git clone #{repo['url']}"
}

If you have an iPhone or iPad, check out the game Fat Cats!

Here’s an idea for Web browsers. When the browser window loses focus, stop everything. Freeze all threads, animated images, scripts, plugins… everything. This would be most useful as a setting that could be enabled and disabled at will. Sometimes I’d just like the browser to just simmer down while my attention is elsewhere. No need to be AJAXing around and stuff.

Thoughts?

Here’s a Public Service Announcement for those of you who wish to buy an iPhone and might want to use it for a different carrier than the one the phone is locked with.

You can’t.*

If you buy an At&t iPhone, Apple says Thou Shalt Only Use Thy iPhone with At&t, and makes it very difficult to use the phone with, say, T-Mobile. You’ll get to use the phone, but At&t and Apple really still own part of it, the part that says how the phone can be used.

Strange, isn’t it? It’s like you bought a roll of duct tape, but it only works on ducts.

Other ideas:

1. buy an unlocked iPhone instead. iPhones hold their value well, especially unlocked ones. They cost a lot more so the upfront cash required is high, but an unlocked phone gives you the freedom to change carriers anytime for any reason. When you figure out the price of the phone, don’t just look at the initial price, look at the cost of the contract over the life of the contract and check prices on similar used unlocked phones on eBay.
2. don’t buy an iPhone. By purchasing an iPhone you’re telling Apple that you’re willing to pay the price of your freedom. Apple pushes hard on its customers, saying what they can and can’t do. They sure do make awesome technology, but at the cost of personal freedom.

Learn more about freedom at https://www.eff.org.

* If you are brave, patient, lucky, and desperate, you may be able to bypass the electronic measures in place which lock the phone to At&t.

Spam comment ≠ Mark comment as spam

I do not want to spam this comment. That work has already been done.

Visual feedback for your moodbar scanner:

Try it out.

And yes, this is a pseudo-graphical application of AWK. I know it’s weird.

A recent question on Ask Ubuntu has prompted me to try to better understand how my sound card works. I don’t really know where to begin, but a little poking around the ALSA Intel HDA driver in use on my ThinkPad X60 has revealed a fascinating presentation of how the laptop’s audio codec is represented by the driver (click to expand):

$ cat /proc/asound/Intel/codec#0
Codec: Analog Devices AD1981
Address: 0
Function Id: 0x1
Vendor Id: 0x11d41981
Subsystem Id: 0x17aa2025
Revision Id: 0x100200
No Modem Function Group found
Default PCM:
    rates [0x7f]: 8000 11025 16000 22050 32000 44100 48000
    bits [0xe]: 16 20 24
    formats [0x1]: PCM
Default Amp-In caps: ofs=0x00, nsteps=0x03, stepsize=0x27, mute=0
Default Amp-Out caps: ofs=0x3d, nsteps=0x3f, stepsize=0x05, mute=1
GPIO: io=4, o=0, i=0, unsolicited=1, wake=0
  IO[0]: enable=0, dir=0, wake=0, sticky=0, data=1, unsol=0
  IO[1]: enable=0, dir=0, wake=0, sticky=0, data=0, unsol=0
  IO[2]: enable=0, dir=0, wake=0, sticky=0, data=0, unsol=0
  IO[3]: enable=0, dir=0, wake=0, sticky=0, data=1, unsol=0
Node 0x02 [Audio Output] wcaps 0x30311: Stereo Digital
  Control: name="IEC958 Playback Con Mask", index=0, device=0
  Control: name="IEC958 Playback Pro Mask", index=0, device=0
  Control: name="IEC958 Playback Default", index=0, device=0
  Control: name="IEC958 Playback Switch", index=0, device=0
  Control: name="IEC958 Default PCM Playback Switch", index=0, device=0
  Control: name="IEC958 Playback Source", index=0, device=0
  Device: name="AD198x Digital", type="SPDIF", device=1
  Converter: stream=0, channel=0
  Digital: Enabled
  Digital category: 0x0
  PCM:
    rates [0x60]: 44100 48000
    bits [0x2]: 16
    formats [0x5]: PCM AC3
  Delay: 3 samples
  Connection: 2
     0x01* 0x04
Node 0x03 [Audio Output] wcaps 0x441: Stereo
  Device: name="AD198x Analog", type="Audio", device=0
  Converter: stream=0, channel=0
  Power states:  D0 D3
  Power: setting=D0, actual=D0
  Processing caps: benign=1, ncoeff=70
Node 0x04 [Audio Input] wcaps 0x100511: Stereo
  Device: name="AD198x Analog", type="Audio", device=0
  Converter: stream=0, channel=0
  SDI-Select: 0
  PCM:
    rates [0x7f]: 8000 11025 16000 22050 32000 44100 48000
    bits [0x6]: 16 20
    formats [0x1]: PCM
  Power states:  D0 D3
  Power: setting=D0, actual=D0
  Connection: 1
     0x15
Node 0x05 [Pin Complex] wcaps 0x400187: Stereo Amp-In Amp-Out
  Control: name="Master Playback Volume", index=0, device=0
    ControlAmp: chs=3, dir=Out, idx=0, ofs=0
  Control: name="Master Playback Switch", index=0, device=0
    ControlAmp: chs=3, dir=Out, idx=0, ofs=0
  Amp-In caps: ofs=0x00, nsteps=0x03, stepsize=0x27, mute=0
  Amp-In vals:  [0x00 0x00]
  Amp-Out caps: ofs=0x3d, nsteps=0x3f, stepsize=0x05, mute=1
  Amp-Out vals:  [0xb4 0xb4]
  Pincap 0x0001173f: IN OUT HP EAPD Detect Trigger ImpSense
    Vref caps: HIZ 50 GRD 80
  EAPD 0x0:
  Pin Default 0xc4014110: [Both] Line Out at Ext Right
    Conn = 1/8, Color = Green
    DefAssociation = 0x1, Sequence = 0x0
    Misc = NO_PRESENCE
  Pin-ctls: 0x40: OUT VREF_HIZ
  Unsolicited: tag=00, enabled=0
  Connection: 2
     0x03 0x0e*
Node 0x06 [Pin Complex] wcaps 0x400185: Stereo Amp-Out
  Amp-Out caps: ofs=0x3d, nsteps=0x3f, stepsize=0x05, mute=1
  Amp-Out vals:  [0x80 0x80]
  Pincap 0x0000001f: OUT HP Detect Trigger ImpSense
  Pin Default 0x422140f0: [N/A] HP Out at Ext Front
    Conn = 1/8, Color = Green
    DefAssociation = 0xf, Sequence = 0x0
  Pin-ctls: 0xc0: OUT HP
  Unsolicited: tag=00, enabled=0
  Connection: 2
     0x03 0x0e*
Node 0x07 [Pin Complex] wcaps 0x400104: Mono Amp-Out
  Amp-Out caps: ofs=0x3d, nsteps=0x3f, stepsize=0x05, mute=1
  Amp-Out vals:  [0x80]
  Pincap 0x00000010: OUT
  Pin Default 0x591311f0: [N/A] Speaker at Int ATAPI
    Conn = ATAPI, Color = Black
    DefAssociation = 0xf, Sequence = 0x0
    Misc = NO_PRESENCE
  Pin-ctls: 0x40: OUT
  Connection: 1
     0x0f
Node 0x08 [Pin Complex] wcaps 0x400083: Stereo Amp-In
  Control: name="Mic Boost", index=0, device=0
    ControlAmp: chs=3, dir=In, idx=0, ofs=0
  Amp-In caps: ofs=0x00, nsteps=0x03, stepsize=0x27, mute=0
  Amp-In vals:  [0x00 0x00]
  Pincap 0x00001727: IN Detect Trigger ImpSense
    Vref caps: HIZ 50 GRD 80
  Pin Default 0xc4a1502e: [Both] Mic at Ext Right
    Conn = 1/8, Color = Red
    DefAssociation = 0x2, Sequence = 0xe
  Pin-ctls: 0x24: IN VREF_80
  Unsolicited: tag=00, enabled=0
Node 0x09 [Pin Complex] wcaps 0x400187: Stereo Amp-In Amp-Out
  Amp-In caps: ofs=0x00, nsteps=0x03, stepsize=0x27, mute=0
  Amp-In vals:  [0x00 0x00]
  Amp-Out caps: ofs=0x3d, nsteps=0x3f, stepsize=0x05, mute=1
  Amp-Out vals:  [0xbf 0xbf]
  Pincap 0x00001737: IN OUT Detect Trigger ImpSense
    Vref caps: HIZ 50 GRD 80
  Pin Default 0x418130f0: [N/A] Line In at Ext Rear
    Conn = 1/8, Color = Blue
    DefAssociation = 0xf, Sequence = 0x0
  Pin-ctls: 0x20: IN VREF_HIZ
  Unsolicited: tag=00, enabled=0
  Connection: 2
     0x03* 0x0e
Node 0x0a [Pin Complex] wcaps 0x400301: Stereo Digital
  Pincap 0x00000010: OUT
  Pin Default 0x414411f0: [N/A] SPDIF Out at Ext Rear
    Conn = RCA, Color = Black
    DefAssociation = 0xf, Sequence = 0x0
    Misc = NO_PRESENCE
  Pin-ctls: 0x40: OUT
  Connection: 1
     0x02
Node 0x0b [Audio Selector] wcaps 0x300101: Stereo
  Connection: 6
     0x03 0x0c 0x09 0x0e* 0x05 0x18
Node 0x0c [Audio Mixer] wcaps 0x200101: Stereo
  Connection: 2
     0x1e 0x1f
Node 0x0d [Audio Selector] wcaps 0x30010c: Mono Amp-Out
  Control: name="Beep Playback Volume", index=0, device=0
    ControlAmp: chs=1, dir=Out, idx=0, ofs=0
  Control: name="Beep Playback Switch", index=0, device=0
    ControlAmp: chs=1, dir=Out, idx=0, ofs=0
  Amp-Out caps: ofs=0x0f, nsteps=0x0f, stepsize=0x0b, mute=1
  Amp-Out vals:  [0x80]
  Connection: 2
     0x10* 0x16
Node 0x0e [Audio Mixer] wcaps 0x200101: Stereo
  Connection: 8
     0x0d 0x11 0x12 0x13 0x1a 0x1b 0x1c 0x1d
Node 0x0f [Audio Mixer] wcaps 0x200100: Mono
  Connection: 1
     0x0b
Node 0x10 [Beep Generator Widget] wcaps 0x700000: Mono
Node 0x11 [Audio Selector] wcaps 0x30010d: Stereo Amp-Out
  Control: name="PCM Playback Volume", index=0, device=0
    ControlAmp: chs=3, dir=Out, idx=0, ofs=0
  Control: name="PCM Playback Switch", index=0, device=0
    ControlAmp: chs=3, dir=Out, idx=0, ofs=0
  Amp-Out caps: ofs=0x17, nsteps=0x1f, stepsize=0x05, mute=1
  Amp-Out vals:  [0x97 0x97]
  Connection: 1
     0x03
Node 0x12 [Audio Selector] wcaps 0x30010d: Stereo Amp-Out
  Control: name="Mic Playback Volume", index=0, device=0
    ControlAmp: chs=3, dir=Out, idx=0, ofs=0
  Control: name="Mic Playback Switch", index=0, device=0
    ControlAmp: chs=3, dir=Out, idx=0, ofs=0
  Amp-Out caps: ofs=0x17, nsteps=0x1f, stepsize=0x05, mute=1
  Amp-Out vals:  [0x80 0x80]
  Connection: 1
     0x08
Node 0x13 [Audio Selector] wcaps 0x30010d: Stereo Amp-Out
  Amp-Out caps: ofs=0x17, nsteps=0x1f, stepsize=0x05, mute=1
  Amp-Out vals:  [0x80 0x80]
  Connection: 1
     0x09
Node 0x14 [Power Widget] wcaps 0x500500: Mono
  Power states:  D0 D3
  Power: setting=D0, actual=D0
  Connection: 13
     0x0d 0x0e 0x0f 0x10 0x13 0x14 0x15 0x16 0x17 0x18 0x19 0x1a 0x1d
Node 0x15 [Audio Selector] wcaps 0x30010d: Stereo Amp-Out
  Control: name="Capture Volume", index=0, device=0
    ControlAmp: chs=3, dir=Out, idx=0, ofs=0
  Control: name="Capture Switch", index=0, device=0
    ControlAmp: chs=3, dir=Out, idx=0, ofs=0
  Control: name="Capture Source", index=0, device=0
  Amp-Out caps: ofs=0x00, nsteps=0x0f, stepsize=0x05, mute=1
  Amp-Out vals:  [0x80 0x80]
  Connection: 8
     0x0c* 0x09 0x0e 0x0f 0x19 0x05 0x18 0x17
Node 0x16 [Pin Complex] wcaps 0x400000: Mono
  Pincap 0x00000020: IN
  Pin Default 0x59f311f0: [N/A] Other at Int ATAPI
    Conn = ATAPI, Color = Black
    DefAssociation = 0xf, Sequence = 0x0
    Misc = NO_PRESENCE
  Pin-ctls: 0x20: IN
Node 0x17 [Pin Complex] wcaps 0x400081: Stereo
  Pincap 0x00000027: IN Detect Trigger ImpSense
  Pin Default 0x599311f0: [N/A] Aux at Int ATAPI
    Conn = ATAPI, Color = Black
    DefAssociation = 0xf, Sequence = 0x0
    Misc = NO_PRESENCE
  Pin-ctls: 0x20: IN
  Unsolicited: tag=00, enabled=0
Node 0x18 [Pin Complex] wcaps 0x400187: Stereo Amp-In Amp-Out
  Amp-In caps: ofs=0x00, nsteps=0x03, stepsize=0x27, mute=0
  Amp-In vals:  [0x00 0x00]
  Amp-Out caps: ofs=0x3d, nsteps=0x3f, stepsize=0x05, mute=1
  Amp-Out vals:  [0xbf 0xbf]
  Pincap 0x00001737: IN OUT Detect Trigger ImpSense
    Vref caps: HIZ 50 GRD 80
  Pin Default 0x41a190f0: [N/A] Mic at Ext Rear
    Conn = 1/8, Color = Pink
    DefAssociation = 0xf, Sequence = 0x0
  Pin-ctls: 0x24: IN VREF_80
  Unsolicited: tag=00, enabled=0
  Connection: 2
     0x03* 0x0e
Node 0x19 [Pin Complex] wcaps 0x400001: Stereo
  Pincap 0x00000020: IN
  Pin Default 0x9933e120: [Fixed] CD at Int ATAPI
    Conn = ATAPI, Color = White
    DefAssociation = 0x2, Sequence = 0x0
    Misc = NO_PRESENCE
  Pin-ctls: 0x20: IN
Node 0x1a [Audio Selector] wcaps 0x30010d: Stereo Amp-Out
  Amp-Out caps: ofs=0x17, nsteps=0x1f, stepsize=0x05, mute=1
  Amp-Out vals:  [0x80 0x80]
  Connection: 1
     0x05
Node 0x1b [Audio Selector] wcaps 0x30010d: Stereo Amp-Out
  Amp-Out caps: ofs=0x17, nsteps=0x1f, stepsize=0x05, mute=1
  Amp-Out vals:  [0x80 0x80]
  Connection: 1
     0x17
Node 0x1c [Audio Selector] wcaps 0x30010d: Stereo Amp-Out
  Amp-Out caps: ofs=0x17, nsteps=0x1f, stepsize=0x05, mute=1
  Amp-Out vals:  [0x80 0x80]
  Connection: 1
     0x18
Node 0x1d [Audio Selector] wcaps 0x30010d: Stereo Amp-Out
  Control: name="CD Playback Volume", index=0, device=0
    ControlAmp: chs=3, dir=Out, idx=0, ofs=0
  Control: name="CD Playback Switch", index=0, device=0
    ControlAmp: chs=3, dir=Out, idx=0, ofs=0
  Amp-Out caps: ofs=0x17, nsteps=0x1f, stepsize=0x05, mute=1
  Amp-Out vals:  [0x19 0x19]
  Connection: 1
     0x19
Node 0x1e [Audio Selector] wcaps 0x30010d: Stereo Amp-Out
  Amp-Out caps: ofs=0x00, nsteps=0x00, stepsize=0x00, mute=1
  Amp-Out vals:  [0x00 0x00]
  Connection: 1
     0x08
Node 0x1f [Audio Selector] wcaps 0x30010d: Stereo Amp-Out
  Amp-Out caps: ofs=0x00, nsteps=0x00, stepsize=0x00, mute=1
  Amp-Out vals:  [0x80 0x80]
  Connection: 1
     0x18

As their names suggest, these “nodes” are all connected on a directed graph. It’s difficult to follow the graph as written above, but Cytoscape can be used to produce a more manageable visualization:

This should be a valuable reference.

A few weeks ago a new user on Ask Ubuntu asked how to close command-line applications in Linux. My snarky answer prompted another user to mention me in his, but I felt I had been misrepresented. I’d like to share my response here since I think it well illustrates a view I hold regarding the role of advanced software tools in Ubuntu:

djeikyb, I appreciate that you make the distinction between basic desktop usage and advanced command-line text editing. Ubuntu’s strength over other Linux distributions is in its ability to make the Linux desktop accessible by everyday computer users, whom to its credit are now increasingly permitted the unique luxury of never needing to know what editors like Vim even are.

It is with this special freedom in mind that I make the decision to work as high above traditional Linux command-line methods as I have the patience for during everyday computer use. I make the effort to utilize only simple and accessible tools largely because I can afford to. It would be inaccurate to say that I simply don’t like command-line text editors. To the contrary, I happen to be fond of Vim with its VimOutliner plugin, and I am indebted to it for its influence on the Vimperator/Pentadactyl project.

However, I view my personal use of these tools as a deviation from typical computer use. I am of the opinion that it would be a healthy development for the Ubuntu community to less readily suggest to newcomers that casual use of advanced tools with steep learning curves is the norm among everyday users.

Letseatlunch, if your goal is to become proficient in Vim, then by all means I encourage you to continue your pursuit of it. I only mean to communicate that, while, importantly, Ubuntu offers many such advanced tools that other operating systems lack altogether, learning to use them should never be a requirement for using Ubuntu.

The Faenza application icons fit nicely with Ubuntu 11.04′s Unity launcher, but the rest of the Faenza icon theme deviates too far from the default Ubuntu look-and-feel for my tastes. To use only the application icons from Faenza, create a “meta-theme” that symbolic links to Faenza’s application icons and inherits everything else from the default icon theme:

1. Create the folder ~/.icons/faenza-apps.
2. Symbolic link ~/.icons/faenza-apps/apps to /usr/share/icons/Faenza/apps.
3. Copy /usr/share/icons/Faenza/index.theme to ~/.icons/faenza-apps/index.theme.
4. In ~/.icons/faenza-apps/index.theme, change the name:
   Name=Faenza for Applications
5. append ubuntu-mono-dark’s inheritances:
   Inherits=ubuntu-mono-dark,Humanity-Dark,gnome,hicolor
6. delete unwanted directories:
   Directories=apps/16.apps/22.apps/24.apps/32.apps/48.apps/scalable
7. and delete all directory stanzas except for the [apps] stanzas.
8. Choose the new theme in Applications ▸ Appearance ▸ Customize ▸ Icons.

Alternatively, just download the following file and drag-and-drop it into the Applications ▸ Appearance window:

• faenza-apps.tar.gz (0.4 MB)

The end result:

I’m currently experimenting with a low-effort technique for mimicking the benefits of a hybrid solid state/hard disk drive.

My approach is to use a union mount with multiple writable branches to combine folders from both an SSD and HDD into a single home folder in the root filesystem. The result is a unified home folder that contains a mixture of files from each drive in a way that is transparent to the operating system:

Newly created files are stored on the same device as their parent folder. Files created at the root of the home folder are typically configuration files and so default to the SSD.

As the screenshot reveals, I have also drafted a Nautilus extension that detects which storage back end each file uses and labels those which use the SSD.

Implementation

The branches of the union mount exist externally as folders within the root filesystem. I chose to place them within /home/.ssd and /home/.hdd. On my system / already uses the SSD so /home/.ssd can simply exist as a regular folder, while /home/.hdd is mounted separately. /etc/fstab contains:

# / containing /home/.ssd (Physical SSD)
UUID=... / ext4 noatime,discard,errors=remount-ro 0 1
# /home/.hdd (Physical HDD)
UUID=... /home/.hdd ext4 errors=remount-ro 0 2

The default file creation policy in aufs is to store new files within the same branch as their parent folder, with files created in the root directory of the union filesystem defaulting to the first listed branch. As long as the SSD branch is listed first, no other policy needs to be explicitly defined:

# /home/ak from SSD and HDD mounts
none /home/ak aufs noauto,br:/home/.ssd/ak=rw:/home/.hdd/ak=rw 0 0

It is critical that this entry in /etc/fstab be mounted only after the previous two lines so that the incorporated branches contain mounted filesystems and not empty mount points. Until I learn how to specify a particular mount order in /etc/fstab, I have disabled automatic mounting of the union mount (noauto above) and instead mount it during /etc/rc.local:

# Workaround to mount aufs only *after* its branches exist
mount /home/ak

Back end detection is accomplished with a custom emblem in ~/.local/share/icons/hicolor/24x24/emblems and a Nautilus-Python extension saved as ~/.nautilus/python-extensions/ssd-emblem.py that checks for parallel files in /home/.ssd:

import os.path
import nautilus

class SsdEmblem(nautilus.InfoProvider):
    def __init__(self):
        pass

    def update_file_info(self, file):
        filepath = os.path.realpath(file.get_location().get_path())
        if filepath.find('/home/ak/') == 0 and os.path.exists(filepath.replace('/home/ak/', '/home/.ssd/ak/',1)) == True:
            file.add_emblem("ssd")

Thoughts

After using this for a week I’m pleased with the results. There is a noticeable improvement in login times compared with my previous configuration that stored the entire home folder on a HDD.

I would like there to be an easy way to switch the storage back end of individual files and folders, ideally via a Nautilus context menu item. This looks possible via UDBA, but there are several potential quirks involved that I haven’t spent the time to thoroughly investigate.

Obviously, more work would need to be done for this to be scalable to multi-user environments or to be compatible with encrypted home folders. I have no plans to develop these areas.

To begin familiarizing myself with R, I’ve been experimenting with binned kernel density estimates using data from Ask Ubuntu.

The resulting plots aren’t very meaningful, but they’re quite pretty.

Rockbox has the ability to generate Audioscrobbler logs for later submission to Last.fm. A custom udev rule can be combined with a command-line Last.fm scrobbler to automate the process of submitting tracks every time the digital audio player is connected to the computer.

The udev rule is designed to call a script whenever a Rockbox device is connected:

# /etc/udev/rules.d/80-custom.rules

ACTION=="add", SUBSYSTEM=="block", ATTR{partition}=="1", ATTRS{vendor}=="Rockbox ", ATTRS{model}=="Internal Storage", RUN+="/opt/rockbox/upload-scrobbler-log"

Last.fm credentials can be stored in GNOME Keyring and accessed using gkeyring:

gkeyring.py --set --name 'http://www.last.fm/' --password 'YOUR_PASSWORD_HERE' -p username_value=YOUR_USERNAME_HERE,origin_url=http://www.last.fm/

The script will wait for the device to be mounted, retrieve the Last.fm credentials from GNOME Keyring, display a graphical notification using notify-send, call Laspyt to submit the scrobbler log, and then display a report:

#!/bin/bash
# /opt/rockbox/upload-scrobbler-log
# WARNING: This script must finish quickly because udev is effectively paused while it is running

# Run all this in the background and detach from the parent process
{

	ubuntuuser="ak"
	logfile="/tmp/upload-scrobbler-log.log"
	backupscrobblerlog="/tmp/scrobbler.log.backup"
	gkeyring="/opt/rockbox/gkeyring.py"
	laspyt="/opt/rockbox/laspyt.py"

	echo "Script starting at $(date)" >> "$logfile"

	dapdescription="$(echo -e "$ID_FS_LABEL_ENC")"
	echo "Detected $dapdescription at $DEVNAME" >> "$logfile"

	# Wait for filesystem to be automounted
	echo "Waiting up to 20 seconds for automount..." >> "$logfile"
	counter=0
	while [ $counter -lt 20 ]; do
		grep -q "$DEVNAME" /proc/mounts && break
		sleep 1
		let counter++
	done

	# Abort if timed out
	if [ $counter -ge 20 ]; then
		echo "Unable to locate mountpoint; aborting" >> "$logfile"
		exit 1
	fi

	# Determine mountpoint
	mountpoint="$(echo -e "$(grep $DEVNAME /proc/mounts | awk '{print $2}')")"
	echo "Detected mountpoint at $mountpoint" >> "$logfile"

	# Identify log file
	scrobblerlog="$mountpoint/.scrobbler.log"
	echo "Using scrobbler log file $scrobblerlog" >> "$logfile"

	# Get credentials from keyring
	echo "Getting credentials from keyring..." >> "$logfile"
	credentials="$(sudo -H -u $ubuntuuser \
		env $(grep -v "^#" /home/$ubuntuuser/.dbus/session-bus/$(cat /var/lib/dbus/machine-id)-0) \
		$gkeyring -p origin_url=http://www.last.fm/ --output username_value,secret)"
	username="$(echo "$credentials" | awk '{print $1}')"
	password="$(echo "$credentials" | awk '{print $2}')"
	if [ -z "$username" -o -z "$password" ]; then
		echo "Failed to get credentials; aborting" >> "$logfile"
		exit 1
	else
		echo "Got credentials for $username" >> "$logfile"
	fi

	# Back up scrobbler log
	echo "Backing up scrobbler log to $backupscrobblerlog" >> "$logfile"
	cp "$scrobblerlog" "$backupscrobblerlog"

	# Upload scrobbler log
	timezone="$(date +%:::z)"
	echo "Uploading scrobbler log file with timezone offset $timezone..." >> "$logfile"
	sudo -H -u $ubuntuuser \
		env DISPLAY=$(w -s $ubuntuuser | grep -m 1 " :" | awk "{print \$3}") \
		notify-send --icon "/usr/share/icons/Humanity/devices/48/multimedia-player.svg" \
			"Uploading scrobbler log" \
			"Submitting $(grep -v "^#" "$scrobblerlog" | wc -l) tracks from $dapdescription to Last.fm as $username"
	sudo -H -u $ubuntuuser \
		$laspyt --file "$scrobblerlog" --timezone "$timezone" --clear \
			--user "$username" --password "$password" \
			2>&1 | sed 's/\x1B\[[0-9;]*[mK]//g' >> "$logfile"

	# Report
	echo "Done" >> "$logfile"
	sudo -H -u $ubuntuuser \
		env DISPLAY=$(w -s $ubuntuuser | grep -m 1 " :" | awk "{print \$3}") \
		zenity --text-info --title "Scrobbler log upload" --filename "$logfile" &

} & disown

exit 0

Details from a run:

and confirmation on Last.fm:

Dot stickers make an excellent tool for customizing the Apple aluminum keyboard.

I don’t listen to a wide range of music. I tend to stick to just a handful of artists, listening to the albums I love over and over.

This gives the distribution of what artists I listen to a well-defined long tail, and I think Last.fm’s histogram doesn’t quite do it justice. With the assistance of the Last.fm API, the R wordclould package, a code sample, and some inspiration, I’ve generated a new visualization:

I think this more clearly illustrates the sheer difference in scale between the few dozen artists that I listen to regularly and the thousand others I’ve had opportunity to evaluate.

Larry Cafiero and Joe Brockmeier are two big voices for technological freedom. They’re both pretty fired up about RMS’s f-you epitaph of Jobs.

Generally you want the figurehead of a public foundation to be, uh, attractive. Intellectually, maybe even physically. Right? Not only does the cause itself have to make sense, these people need to attract other people to their cause. And they usually “say the right things”, smile, wear a suit, whatever. But I always thought these requirements only applied to other causes (besides Free Software).

Certainly RMS lacking those traits didn’t keep me from FLOSS. I heard about RMS and the proprietary printer a while back, and that’s all it took to get me hooked on FLOSS. I could identify immediately because I write software, and proprietary code is a pain. His cause just makes sense, even if he doesn’t. But I’ve been justifying his abnormal behavior because, well, he started something new! Something important. He knew it was important, and dedicated his life to this thing that many, many folks never even know exists. Something that affects all our lives, every day, more and more. Software must support our Freedom, or we are not free.

So he won me over, but I’m a nerd. I’m used to eccentrics in my field. Truth wins, period. And I still don’t know if it matters if RMS is a polished, smiley, public-friendly dude or not. Would Free Software be farther along today if RMS were kinder, more respectful, or somehow a better “public figure”? Would DRM have never been allowed to exist? Would the government pass laws that software for implanted medical devices be Free?

Link checkers spider through your website and make sure that links work. I want an awesome link checker. Ideally, it would espouse as many of these attributes as possible:

• easy to learn
• easy to configure/customize
• example config: don’t hit URLs on other servers
• sensible default behaviors
• example: respects robots.txt and ‘nofollow’ link attributes
• scriptable / embeddable
• useful from command line
• useful from within CI servers like Jenkins
• recurses (parses HTML, follows links)
• and smartly avoids checking the same pages twice
• fast
• thrifty with memory
• pluggable
• example plugin: run jslint on all JavaScript
• example plugin: validate HTML 5
• example plugin: validate CSS
• example plugin: compute accessibility score
• example plugin: JUnit XML output
• example plugin: OpenDocument spreadsheet output
• example plugin: Excel output
• example plugin: CSV output
• example plugin: JavaScript engine
• example plugin: follow hashbang URLs
• beautiful source code
• FLOSS

Please check out my post Warren Buffet’s Long Bet on the BreadVault blog.

All the employment suckage of late had me thinking of how many jobs we've (and by we I mean the corporate jackoffs) let go to foreign countries via outsourcing, NAFTA etc. In light of this I want to remind us of the "Buy American" movement of yesteryear. Yes it lost steam, but for the wrong reasons. We've got to remember quality of course! But buying products made in the U.S. and even better, locally (produce is a great example) will go a long way towards helping our economic recovery.

So, here are some products I have found are "made in America" that I like

• KIND PLUS nature bars - these are super-yummy nature bars
• More to come!

Permalink | Leave a comment  »

I’m liking Henri Sivonen’s Validator.nu service. I’ve got it running locally, and it works well. I can use it as a web service and validate HTML from within Vim, using quickfix to rapidly resolve errors. My Jenkins CI server uses the same validator via phpunit tests.

Warning: it took me a very long time to get it running locally. Technically easy (just run a build script), but it downloads tons of libraries and files before it can do its job.

My new job is going well! I can’t wait to post more about it. Sign up if you’d like to hear more as we share more.

Visit http://breadvault.com to sign up.

Virtual machines don’t produce a lot of entropy on their own. Typically the need for additional entropy triggers talk of hardware based entropy generators or network based entropy distribution protocols. Sometimes you just need a little bit of entropy for a moment.

$ sbuild-update --keygen
Generating archive key.

Not enough random bytes available.  Please do some other work to give
the OS a chance to collect more entropy! (Need 279 more bytes)

Disk tends to be one of the only remaining sources of entropy on virtual systems. I usually do something like this:

$ while true ; do cat /proc/sys/kernel/random/entropy_avail  ; \
    sudo find / > /tmp/find.log ; sync ; done

This numbers printed should go up and down as your application consumes the entropy. Hit CTRL+C when you’ve got enough. This is probably a bad source of entropy, but the world is inherently dangerous.

http://fedex.com/Tracking?action=track&cntry_code=us&tracknumber_list=TNUM

Replace TNUM with your tracking number.

I run “web tests” on a remote server. I use Selenium to act like a person interacting with a website, viewing and entering data. Selenium is pretty awesome, it can drive a real web browser like Firefox.

Even better is to have these web tests run automatically every time I commit code. I use Jenkins for this. Jenkins even fires up a headless desktop so Selenium can run Firefox.

When a web test breaks (especially in some way I can’t reproduce on my local desktop), sometimes it helps to actually see what Jenkins sees as it runs the test. Here’s a quick guide for doing so on an Ubuntu GNU/Linux server.

1. Connect to the remote server using SSH. Install VNC server:
   

   sudo apt-get install vnc4-server

2. On the remote server, become the user tests run as. For example:
   

   sudo su - ci

3. Set a password for the VNC server using the vncpasswd command.
4. Start headless X server by running vncserver. Note the given display. If example.com:1 is included in the output of vncserver, the display is :1.

5. Figure out which port the VNC server is using. I usually do something like

   sudo netstat -nape | grep '^tcp.*LISTEN.*vnc.*'

   Here’s some example output:

   tcp        0      0 0.0.0.0:6001            0.0.0.0:*               LISTEN      107        3099855     13233/Xvnc4     
   tcp6       0      0 :::5901                 :::*                    LISTEN      107        3099858     13233/Xvnc4

   By trial and error, I figured out that 5901 was the port I should use.

6. Port-forward VNC to your local machine.

   1. Disconnect from the server.
   2. Reconnect, including -L10000:localhost:5901 on your SSH command line.
   3. Leave this connection open.

7. On your local machine, connect a VNC client to localhost:10000. An X terminal should be displayed.

vncserver -kill :1

I joined a startup, and they gave me a MacBook Pro. It was bound to happen eventually; all the cool kids use MBPs and startups are cool, right?

The great period of adaption began, as I learned I couldn’t have simple technology like sloppy focus. One of the greatest inconveniences is the keyboard. I have a hard time using the keyboard on the laptop because special keys are in different places than I’m used to. Even with a Unicomp Spacesaver M (for those of us attached to the Model M), some change is apparent, like Apple using “delete” when they mean “backspace” (The Unicomp uses “delete ->” when they mean “delete”).

Most frustrating of this set of issues is that in Firefox the home and end keys go to the top and bottom of the page, whereas you have to use cmd+left and cmd+right to go to the beginning and end of a line in a textbox. However sometimes these keys represent page forward and page back, and sometimes they don’t (usually in a flash app, I believe). The solution is to install the keyconfig extension. After you restart firefox, you will find it in the Tools menu where you can disable “GoBackKb” and “GoForwardKb”. Then these keys work as expected in a text box and you no longer find yourself going back a page unintentionally, possibly losing a textbox full of input along the way.

Chef uses a special key pair to create new clients called the “validation client.” If you lose this file, or perhaps you end up with an empty CouchDB database and no longer have this client in the database, you could get a 401 Unauthorized error when trying to use it.

$ sudo chef-client
[Thu, 14 Jul 2011 11:44:44 +0000] INFO: *** Chef 0.10.2 ***
[Thu, 14 Jul 2011 11:44:44 +0000] INFO: Client key /etc/chef/client.pem is not present - registering
[Thu, 14 Jul 2011 11:44:44 +0000] INFO: HTTP Request Returned 401 Unauthorized: Failed to authenticate. Ensure that your client key is valid.
[Thu, 14 Jul 2011 11:44:44 +0000] FATAL: Stacktrace dumped to /var/cache/chef/chef-stacktrace.out
[Thu, 14 Jul 2011 11:44:44 +0000] FATAL: Net::HTTPServerException: 401 "Unauthorized"

Removing your validation key on the server, which is typically stored on the filesystem at /etc/chef/validation.pem and restarting the chef-server will create a new key pair both on disk and in the database.

$ ls -l /etc/chef/validation.pem
-rw-r--r-- 1 root root 1676 2011-07-14 11:44 /etc/chef/validation.pem
$ sudo rm /etc/chef/validation.pem
$ sudo /etc/init.d/chef-server restart
 * Restarting chef-server
 ~ Killing pid 10783 with INT
 ~ In 12051
   ...done.
$ ls -l /etc/chef/validation.pem
-rw------- 1 chef chef 1679 2011-07-14 11:55 /etc/chef/validation.pem

The same process works with the webui key pair, which knife uses as the default “admin” key to create initial knife clients.

$ ls -l /etc/chef/webui.pem
-rw------- 1 chef chef 1675 2011-07-14 11:31 /etc/chef/webui.pem
$ sudo rm /etc/chef/webui.pem
$ sudo /etc/init.d/chef-server restart
 * Restarting chef-server
 ~ Killing pid 12051 with INT
 ~ In 12091
   ...done.
$ ls -l /etc/chef/webui.pem
-rw------- 1 chef chef 1675 2011-07-14 11:57 /etc/chef/webui.pem
$ sudo /etc/init.d/chef-server-webui restart
 * Restarting chef-server-webui
 ~ Killing pid 10945 with INT
 ~ In 12129
   ...done.

If you’ve also lost your key for your knife client, you will need to create another one. Use a new client name unless you’re sure that the server does not still contain a registration for the previous client. After creating the new client, you can delete the old one from the server using ‘knife client delete MY_OLD_CLIENT’ by replacing MY_OLD_CLIENT with the name of the former client.

$ sudo knife configure --initial
Overwrite /home/ubuntu/.chef/knife.rb? (Y/N) y
Please enter the chef server URL: [http://ip-10-204-150-209.ec2.internal:4000]
Please enter a clientname for the new client: [ubuntu] new_ubuntu
Please enter the existing admin clientname: [chef-webui]
Please enter the location of the existing admin client's private key: [/etc/chef/webui.pem]
Please enter the validation clientname: [chef-validator]
Please enter the location of the validation key: [/etc/chef/validation.pem]
Please enter the path to a chef repository (or leave blank):
Creating initial API user...
Created client[new_ubuntu]
Configuration file written to /home/ubuntu/.chef/knife.rb

Provided with the new validation.pem, your node should be able to register now, as long as there is not still a client by the same name. If there is, you will need to delete that client first. Note that on Opscode Hosted Chef, you currently will need to delete the node as well, because the default permissions only allow the client that created the node to modify it.

$ knife client list
  chef-validator
  chef-webui
  ip-10-204-150-209.ec2.internal
  new_ubuntu
  ubuntu
$ knife client delete ip-10-204-150-209.ec2.internal
Do you really want to delete ip-10-204-150-209.ec2.internal? (Y/N) y
Deleted client[ip-10-204-150-209.ec2.internal]
$ sudo chef-client
[Thu, 14 Jul 2011 12:04:24 +0000] INFO: *** Chef 0.10.2 ***
[Thu, 14 Jul 2011 12:04:26 +0000] INFO: Client key /etc/chef/client.pem is not present - registering
[Thu, 14 Jul 2011 12:04:27 +0000] INFO: Run List is []
[Thu, 14 Jul 2011 12:04:27 +0000] INFO: Run List expands to []
[Thu, 14 Jul 2011 12:04:27 +0000] INFO: Starting Chef Run for ip-10-204-150-209.ec2.internal
[Thu, 14 Jul 2011 12:04:27 +0000] INFO: Loading cookbooks []
[Thu, 14 Jul 2011 12:04:27 +0000] WARN: Node ip-10-204-150-209.ec2.internal has an empty run list.
[Thu, 14 Jul 2011 12:04:28 +0000] INFO: Chef Run complete in 0.623124 seconds
[Thu, 14 Jul 2011 12:04:28 +0000] INFO: Running report handlers
[Thu, 14 Jul 2011 12:04:28 +0000] INFO: Report handlers complete

Some more books I’ve read lately. I guess I really like Sci Fi. Most of these are free from Feedbooks. Thank you, Feedbooks! I like the Feedbooks versions because they include handy recommendations at the end of the books. These recommendations led me to many of the books below.

• Treasure Island by Robert Louis Stevenson – Awesome. Must read.
• Brave New World by Aldous Huxley – Must read. This came out in 1932, but is as compelling as any of the best science fiction ever written. Ever.
• I, Robot by Cory Doctorow – Short, fun. Must read because I like Doctorow’s take on sentient robots.
• Starfish by Peter Watts – Crazy, intense. Must read.
• Peter and Wendy by J. M. Barrie – Must read.
• The Shallows by Nicholas Carr – Very thick with useful, interesting information. I need to read this a few more times to really soak it all in. Must read because we need to think hard about the technology we consume.
• The Time Machine by H. G. Wells – Like.
• Computers, Ltd. by David Harel – Like. I need to read this one a few more times. Must read for computer nerds, because you’ll understand why some programming problems are unsolvable.
• Philip K. Dick short stories (Beyond Lies the Wub, Beyond the Door, Mr. Spaceship, The Crystal Crypt, The Defenders, The Gun, The Skull, The Variable Man) – These are all short and pretty fun.
• 2BR02B by Kurt Vonnegut – Short, fun. Must read.
• The Dark Fields by Alan Glynn – Pretty good. Probably better than Limitless.
• Lee Child (Jack Reacher stories) – These are always fun. I read the latest one and liked it, but I don’t even remember the name. Doesn’t really matter since they’re all pretty formulaic. Still, good books.
• Jury Service by Cory Doctorow – Crazy fun. Gotta love Doctorow.
• Sherlock Holmes by Arthur Conan Doyle – Awesome.
• Planet of the Damned by Harry Harrison – Must read.
• Deathworld by Harry Harrison – Must read.
• The Machine Stops by E.M. Forster – Pretty good.
• Missing Link by Frank Herbert – Ok.
• Operation Haystack by Frank Herbert – Not bad.
• The Call of Cthulu by H. P. Lovecraft – Ok. I don’t quite get what’s up with the intense cult following.
• Benjamin Button by F. Scott Fitzgerald – Like.
• The Last Universe by William Sleator – Meh.
• Island of Dr. Moreau by H. G. Wells – Must not read. This book was like nails on a chalkboard.

See also

Books I

I’ve finally decided to attempt a real transition to the Mutt email program after years of flirting with the idea and making short, hesitant attempts that amounted to nothing. This time I’m giving it a fair shake as part of an overarching effort to move toward more fully keyboard-driven computer interaction that has involved replacing other components with Vim and Xmonad.

I currently have all of my mail flowing through Gmail in one way or another, accessing it regularly via the web on my laptop, and the official Gmail application for Android. So the first step in this transition is to use mutt as an IMAP client for Gmail, leaving any server-related work for later stages.

Fortunately, Google turned up several on-target results for just that scenario. After some trial-and-error one of them had the goods: Using Gmail with Mutt, the minimal way (IMAP update). The folder names were an especially great detail as it solves the problem of IMAP/Drafts and so forth labels being created in Gmail by prefixing them with Gmail.

I also managed to get Gvim hooked up to it after a little trouble initially. Mutt automatically launched Gvim for new message composition because (I assume) it checks for the VISUAL and EDITOR environment variables (which are set to Gvim and Vim on my system) however the headers were not present in the buffer and saving an attempted message would come to nothing. In fact Mutt would print “Aborted unmodified message” in its status bar as soon as Gvim was launched.

The solution was to set the editor in .muttrc, like so:

set editor="gvim -f"

The -f option tells Gvim that it should stay tethered to the shell it was spawned from rather than forking and detaching from it.

But that’s not all, I also found a few helpful nuggets here: Mark’s Mutt Fan and Tip page. After which the set editor line in my .muttrc file was replaced with:

set edit_headers  
set editor="gvim -f +/^$ ++1"

I also added Mark’s syntax highlighting suggestion:

" set up syntax highlighting for my e-mail  
au BufRead,BufNewFile .followup,.article,.letter,/tmp/pico*,nn.*,snd.*,/tmp/mutt* :set ft=mail

And finally, I happened across an aging mailing list entry with a smart tip: define an email-specific configuration file for Vim/Gvim. Unlike the example given, I’m using -S rather than -u, to preserve the rest of my configuration.

So finally the set editor line in .muttrc looks like:

set editor="gvim -f +/^$ ++1 -S ~/.mutt_vimrc"

A-muttering I will go!

Permalink | Leave a comment  »

Ran into a mildly perplexing problem while trying to install the mysql2 gem under Ubuntu 11.04 today. As will sometimes happen when Ruby gems require compilation of native binaries to run on your system, there is an unspoken dependency requirement of certain development libraries….

$ gem install mysql2
Fetching: mysql2-0.3.2.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing mysql2:
	ERROR: Failed to build gem native extension.

        /home/trevor/.rvm/rubies/ruby-1.9.2-p180/bin/ruby extconf.rb
checking for rb_thread_blocking_region()... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lm... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lz... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lsocket... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lnsl... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lmygcc... no
checking for mysql_query() in -lmysqlclient... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:

[looooong list of options]

To compile the driver successfully I needed to install the MySQL development library first.

$ sudo apt-get install libmysqlclient-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  libmysqlclient-dev
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 3,174 kB of archives.
After this operation, 9,839 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ natty/main libmysqlclient-dev amd64 5.1.54-1ubuntu4 [3,174 kB]
Fetched 3,174 kB in 15s (210 kB/s)                                                                                                                                                           
Selecting previously deselected package libmysqlclient-dev.
(Reading database ... 157888 files and directories currently installed.)
Unpacking libmysqlclient-dev (from .../libmysqlclient-dev_5.1.54-1ubuntu4_amd64.deb) ...
Processing triggers for man-db ...
Setting up libmysqlclient-dev (5.1.54-1ubuntu4) ...

After that, the mysql2 gem installation succeeded without any trouble.

What’s up with OpenLogic? A recent wave of press claims that OpenLogic’s data reveals the top Open Source license. Meh, no way.

I commented on that article (corrections made):

Sean, your headline and article are misleading. These claims may be true for OLEX, but not FLOSS in general. I’ve been downloading FLOSS for over 10 years, and I’ve never heard of OLEX. I download most FLOSS from GNU/Linux distributors (Fedora/Debian/Ubuntu/Canonical). I’m not sure if they even track downloads.

Do the people at Geeknet/sf.net, Google Code, and the FSF support your findings?

Come on, OpenLogic. Are you serious?

I’ve been using Linux for a long time, and using the less utility increasingly over recent years. Discovering that I’d been missing some really great functionality and would never have known but for a twitch made me sit up and think a bit.

Steve Krug’s Don't Make Me Think taught me an amazing word: satisfice. We all do it, all the time. We learn just enough to satisfy our immediate need with exactly what will suffice and no more so we can move on to whatever our real goal is, and that’s exactly what I did with less, resulting in going out of my way to open a file for editing occasionally because I didn’t take a few minutes to read the manual.

For those working on moving debian ruby library packaging to gem2deb, you can exempt specific hits from the slick built in ‘require rubygems’ test by adding the path to debian/require-rubygems.overrides.

For instance, to exempt this:

debian/chef/usr/lib/ruby/vendor_ruby/chef/provider/package/rubygems.rb: require 'rubygems'
debian/chef/usr/lib/ruby/vendor_ruby/chef/provider/package/rubygems.rb: require 'rubygems/version'
debian/chef/usr/lib/ruby/vendor_ruby/chef/provider/package/rubygems.rb: require 'rubygems/dependency'
debian/chef/usr/lib/ruby/vendor_ruby/chef/provider/package/rubygems.rb: require 'rubygems/spec_fetcher'
debian/chef/usr/lib/ruby/vendor_ruby/chef/provider/package/rubygems.rb: require 'rubygems/platform'
debian/chef/usr/lib/ruby/vendor_ruby/chef/provider/package/rubygems.rb: require 'rubygems/format'
debian/chef/usr/lib/ruby/vendor_ruby/chef/provider/package/rubygems.rb: require 'rubygems/dependency_installer'
debian/chef/usr/lib/ruby/vendor_ruby/chef/provider/package/rubygems.rb: require 'rubygems/uninstaller'
debian/chef/usr/lib/ruby/vendor_ruby/chef/provider/package/rubygems.rb: require 'rubygems/specification'
debian/chef/usr/lib/ruby/vendor_ruby/chef/providers.rb: require 'chef/provider/package/rubygems'
Found some 'require rubygems' without overrides (see above).
ERROR: Test "require-rubygems" failed. Exiting.
dh_auto_install: dh_ruby --install /«BUILDDIR»/chef-0.10.0/debian/chef returned exit code 1
make: *** [binary] Error 1
dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2

debian/require-rubygems.overrides should contain:

debian/chef/usr/lib/ruby/vendor_ruby/chef/provider/package/rubygems.rb
debian/chef/usr/lib/ruby/vendor_ruby/chef/providers.rb

http://chrisaitchison.com/2011/05/03/you-are-not-a-software-engineer

Thank you Jesse for the link!

I really like the post, but I’d like to suggest a couple changes to the Chris Aitchison. Maybe the team of people building the software is the garden, and the software product is just a fruit (or maybe people producing the software are gardeners and the garden!). Second, the metaphor works well enough for Web programmers. But folks writing game ROMs (for instance, 8-bit Nintendo games), lunar lander software, and Level-A software for commercial airliners really are Software Engineers.

I still like Free Software, but I decided to change the name of this blog to just… my name. In my feed reader, I found myself renaming other creatively-named blogs authored by one person mainly about their own interests to their name. So I guess this is dogfooding.

Dear reader, I’d love to hear your preference: creative blog names or simple but accurate names?

Blargh == blog. I think I got that from Aaron Patterson.

"LinuxFest Northwest is happening this Saturday in Bellingham Wa". I'll
be there (for my 6th year) with the BitPusher crew, helping run the
GameDen and even shmoozing a bit.

Permalink | Leave a comment  »

Are you a writer? Consider learning a programmable text editor. You don’t need to be a programmer to learn one, you just need a little patience and curiosity. Thanks for picking up my article, Cool Tools!

On a related note, I’ll be giving a talk on Vim at LinuxFest Northwest. Hope you can make it!

I received the following error while working on a Debian sid box:

$ schroot -l
terminate called after throwing an instance of 'std::runtime_error'
  what():  locale::facet::_S_create_c_locale name not valid
Aborted

With debconf + locales already installed, I ran ‘export | grep LANG’ to discover that my locale was set to ‘en_US.UTF-8′. Then I ran ‘dpkg-reconfigure locale’ and checked that locale and set it to the default.

For the most part, this is the same as any post about creating an image for Eucalyptus, but I had a hard time figuring out exactly how to put it all together. You need an up to date Debian sid system nearby to take the kernel and ramdisk from. I found having a sid VM easier than discovering the commands to build a sid initrd on my Ubuntu workstation.

# First, the prerequisites. You need debootstrap and the eucalyptools tools installed.
sudo apt-get install debootstrap euca2ools

# Export your eucalyptus variables to use the tools.
source ~/.euca/eucarc

# Create an empty disk image. You can adjust the count to change the root disk size. 1000 is about a GB.
dd if=/dev/zero of=image count=1000 bs=1M

# Put a filesystem on the new disk image
mkfs.ext3 -F image

# Mount the filesystem
mkdir chroot
sudo mount -o loop image chroot

# Install debian sid to the chroot. Notice that the ssh server and curl are included here
sudo debootstrap --include=openssh-server,curl,vim --arch amd64 sid chroot/ http://ftp.debian.org

# chroot into the image
sudo chroot chroot

# Setup basic networking and disk configurations
echo -e 'auto lo\niface lo inet loopback\nauto eth0\niface eth0 inet dhcp' >> /etc/network/interfaces
echo -e '/dev/sda1 / ext3 defaults 0 1\n/dev/sda2 swap swap defaults 0 0' > /etc/fstab

# Set a default root password if you want
# passwd

# Set up the image to automatically install ssh keys
mkdir /root/.ssh
cat <<EOS > /etc/rc.local
echo >> /root/.ssh/authorized_keys
curl -m 10 -s http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key | grep 'ssh-rsa' >> /root/.ssh/authorized_keys
echo "AUTHORIZED_KEYS:"
echo "************************"
cat /root/.ssh/authorized_keys
echo "************************"
exit 0
EOS

# Leave the image
exit

# Unmount the image
sudo umount chroot

# After you've copied the latest /boot/vmlinuz* and /boot/initrd* from your sid system, upload the kernel + ramdisk
euca-bundle-image --image vmlinuz-2.6.38-2-amd64 --kernel true
euca-upload-bundle --bucket sid --manifest vmlinuz-2.6.38-2-amd64.manifest.xml
euca-register sid/vmlinuz-2.6.38-2-amd64.manifest.xml
euca-bundle-image --image initrd.img-2.6.38-2-amd64 --ramdisk true
euca-upload-bundle --bucket sid --manifest initrd.img-2.6.38-2-amd64.manifest.xml
euca-register sid/initrd.img-2.6.38-2-amd64.manifest.xml

# Prepare the image for upload, use the values given by euca-register above here
euca-bundle-image -i image --kernel eki-XXXXXXXX --ramdisk eri-XXXXXXXX

# Rename to manifest to something descriptive and upload it
mv image.manifest.xml `date +%Y%m%d`.sid.manifest.xml
euca-upload-bundle -b sid -m `date +%Y%m%d`.sid.manifest.xml

# Register the image to get an EMI
euca-register sid/`date +%Y%m%d`.sid.manifest.xml

You should be able to use euca-run-instance on the emi that is returned by the last command. Remember to pass an ssh key (that eucalyptus knows about) using -k. If there are any issues, use euca-get-console-output to monitor the instance startup and tail the eucalyptus/nc.log file on the node controller for any errors. Building the initrd this way is a little hackish, because it is actually generated for your sid system, not for the one running in eucalyptus. Chicken, or the egg?

Interesting article from Ars Technica about certificates being used for man-in-the-middle (MITM) attacks. The quote "Certification revocation is basically useless" sure hits home.

http://arstechnica.com/security/news/2011/03/how-the-comodo-certificate-fraud-calls-ca-trust-into-question.ars/

Permalink | Leave a comment  »

A coworker forwarded me an inspiring article from the Harvard Business Review. I was interested after the first paragraph, and was looking excitedly looking forward to finishing it along with a cookie. But I could not!

I’d get a couple more sentences in before being distracted. I would dutifully stop, blink, then rewind a couple sentences, then repeat. Over and over. The dreaded technology loop!

Hmm, I’d rather read this as an ebook. I wonder how I could convert it. I wonder who else has read this article. Do I have any new emails? Maybe there’s some news on Lybia I should be aware of. What other stuff has this dude written that I should read first? Man, an ebook of this would be nice. Maybe I should just print it?

Come on brain, read! Think! And remember, dangit!

I felt like I was learning to read for the first time. I had to work very hard to actually make it through the article. Then I stopped to think about what I had read, and realized my reading comprehension was for naught.

Why couldn’t I take in the information? Because I was reading a computer screen. When I’m upright at the computer, hands on keyboard & mouse, I’m in computer mode, not reading mode. Great for doing stuff. Sure, I can learn stuff too, but nothing more than a tweet-sized tidbit. And the only pieces I can reliably hang on to are pointers to the “real” information online, somewhere. It’s a useful state of being for some activities, not completive thought.

This is exactly the phenomenon described by Nicholas Carr in The Shallows.

I find myself doing this all the time lately.

me: “Check out the supermoon!”
Eva: “What’s a supermoon?”
me: “No idea. Just saw the headline. It’s probably bigger or something.”
Eva: “Huh?”

Eva and I did an experiment for while where, for one day a week, we didn’t use the internet. At all. For a whole day. Try it! I dare you.

Oh yeah, the article. It’s 6 pages long. How Will You Measure Your Life? by Clayton M. Christensen.

Are you stuck in The Shallows at the computer? Are you able to read long articles on a computer screen? How do you recalibrate your brain when you need to think deeply about something? How does your reading comprehension and ability to concentrate change when you read newsprint? A book on an ebook reader? A printed novel? A Web page? How does your writing change when you use a computer vs. when you write with a pen and paper? Do you ever actually write with a pen and paper?

I want a very stable desktop, so I’m using Ubuntu 10.04 “lucid lynx” LTS (long-term support). I also want features of the latest version of git. I ran these commands to grab the latest stable git release without futzing with any manual downloading or dpkg.

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:git-core/ppa
sudo apt-get update
sudo apt-get install git

Thank you, Anders Kaseorg!

Distribution packages in a release typically lag behind those released by upstream maintainers. This is expected: part of what makes a release stable is intentionally not introducing changes. PPAs (personal package archives) are a handy way to get at packages not already present in the distribution/release you are using. Backports are handy too, but I didn’t see the latest git in there for Ubuntu 10.04.

Back in the old days I used to manually scour some combination of rpmfind, DAG, Dries, ATrpms, rpmforge, (and others!) to find the right packages and dependencies.

TL;DR: If you replace an iPhone with a non-iPhone on the same AT&T account, make sure AT&T “re-features” your account voicemail, whether you want to use it or replace it with Google Voice.

I recently replaced my first-gen Apple iPhone with a Motorola’s new premium Android phone for AT&T, the Atrix 4G. Some time ago I switched voicemail handling from the iPhone’s “visual voicemail” to Google Voice. It turned out to work very well for me so I stuck with it.

Naturally I wanted to continue using Google Voice when I upgraded to the Atrix, so I didn’t change a thing when my service was transferred. Eventually I discovered that unanswered calls to my Google Voice number would go to voicemail, but calls to my old carrier-assigned number would ring forever (or until the caller lost patience and terminated the connection).

Attempts to correct this by setting Google Voice as my voicemail handler with the controls in the Google Voice web application (and the given MMI codes) and adjusting the Call Settings on the phone met with cryptic error messages.

Eventually, a web search provided the important nugget of information that led to a resolution. The voicemail feature on my AT&T account had a lingering setting for iPhone visual voicemail that was preventing manipulation. A few minutes in chat with an AT&T rep had that cleared up!

This may be important to others as well: I made a good number of frustrated attempts to call my carrier-provided phone number and kept hearing the endless ringing, only to find a voicemail from my brother waiting for me when I returned to the computer and phone after walking away for a few minutes. I’d been testing by calling from the only other available phone I had: Google Talk. And yes, Google Talk uses my Google Voice number, so I was just being forwarded back to myself.

Here is a strange one that I fixed but I’m not sure why. Roughly using the SBuildLVM Howto, and the Chef sbuild cookbook, I have an sbuild server. It was working alright for me, but another user was seeing this:

schroot -c lucid
E: 05lvm: File descriptor 3 (socket:[460392]) leaked on lvcreate invocation.
E: lucid-40c0e109-2d5d-4103-bf92-a44288595dcc: Chroot setup failed: stage=setup-start

When he ran with verbose mode, this line was particularly interesting:

E: 05lvm:

When I su’d to his user, it worked fine for me without verbose but failed similarly with the verbose flag.

In the course of debugging, I started trying to redirect output and I found that these changes to /etc/schroot/setup.d/05lvm fixed the problem. Unfortunately I’m running behind on work so I can’t track down the root cause right now.

--- 05lvm.orig	2011-03-10 19:28:17.000000000 +0000
+++ 05lvm	2011-03-10 19:37:54.000000000 +0000
@@ -36,10 +36,10 @@

 	if [ "$AUTH_VERBOSITY" = "verbose" ]; then
 	    lvcreate $VERBOSE --snapshot --name "$CHROOT_LVM_SNAPSHOT_NAME" \
-		"$CHROOT_DEVICE" $CHROOT_LVM_SNAPSHOT_OPTIONS
+		"$CHROOT_DEVICE" $CHROOT_LVM_SNAPSHOT_OPTIONS 2>&1
 	else
 	    lvcreate $VERBOSE --snapshot --name "$CHROOT_LVM_SNAPSHOT_NAME" \
-		"$CHROOT_DEVICE" $CHROOT_LVM_SNAPSHOT_OPTIONS > /dev/null
+		"$CHROOT_DEVICE" $CHROOT_LVM_SNAPSHOT_OPTIONS 2>&1 > /dev/null
 	fi

     elif [ $1 = "setup-stop" ]; then
@@ -57,9 +57,9 @@
 		--pid=$PID || true

 	    if [ "$AUTH_VERBOSITY" = "verbose" ]; then
-		lvremove $VERBOSE -f "$CHROOT_LVM_SNAPSHOT_DEVICE" || true
+		lvremove $VERBOSE -f "$CHROOT_LVM_SNAPSHOT_DEVICE" 2>&1 || true
 	    else
-		lvremove $VERBOSE -f "$CHROOT_LVM_SNAPSHOT_DEVICE" > /dev/null || true
+		lvremove $VERBOSE -f "$CHROOT_LVM_SNAPSHOT_DEVICE" 2>&1 > /dev/null || true
 	    fi
 	else
 	    # The block device no longer exists, or was never created,

This is a followup to my post about Measuring Development Speed. Here’s an updated graph showing the current state of head/master:

Most dramatic change in Struts/jsp LOC is now: 5412 (time period ending Feb 2011)

Most dramatic change in SpringMVC/ftl LOC is still: 3665 (time period ending Sep 2010)

Again, some assumptions/disclaimers:

• these data are not useful for estimating speed of future refactoring work
• all acceptance tests pass in refactored areas
• look & feel of refactored areas is acceptable

Also see this this thread on the dev list about the usefulness of these measurements.

Two and a half years have passed since I wrote about running Munin with fastcgi triggered graphs on Debian etch. Unfortunately, not a lot has changed since then. A revolution in trending would have been nice. When I started here munin was triggering graph generation using CGI and was painfully slow to use. I switched over to cron triggered graph generation and was happy. After a data center migration, drawing the munin graphs for that cluster from cron was taking about 130 seconds. As a precaution I wanted to get this down a bit.

Someone asked me why munin-graph would have caused data loss because munin-update collects the data and I couldn’t remember. I had problems with both munin-update and munin-update taking over five minutes in certain circumstances back then. The latter was primarily from the slow response time of the SNMP queries I was doing against MSSQL servers. That was back during Munin 1.2 as well and a few things have changed since then, most relevant is that you no longer have to patch Munin for fastcgi support.

This time around I used fcgid instead of fastcgi. There are less licensing hurdles for fcgid, which was written to be compatible with fastcgi. Provided you already have munin running, install the prerequsites first.

sudo apt-get install libcgi-fast-perl libdate-manip-perl libapache2-mod-fcgid

The packaging should restart Apache as required to load the new module we just installed, but we need to configure our Munin site a bit to link our CGI script to fcgid. Add this to or update the VirtualHost block for your Apache configuration and reload Apache.

  ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/

  <Directory /usr/lib/cgi-bin/>
    AllowOverride None
    Options ExecCGI -MultiViews +SymLinksIfOwnerMatch
    Order allow,deny
    Allow from all
  </Directory>

  <Location /cgi-bin/munin-fastcgi-graph>
    SetHandler  fastcgi-script
  </Location>

Add the following lines to your munin.conf. This causes the munin-graph that is run from cron to not generate any graphs (noops) and munin-html will update the img src links to use the CGI script to generate the graphs rather than linking directly to files. You’ll need to wait for the cron job to run once or run munin-html yourself to trigger this.

graph_strategy cgi
cgiurl_graph /cgi-bin/munin-fastcgi-graph

Triggering munin-html manually:

sudo -s
sudo -u munin /usr/share/munin/munin-html --debug

Remember that Apache needs to be able to write the graphs out. You will get no graphs and HTTP 500 errors in your Apache logs if the munin-cgi-graph script cannot write the graphs out. My Munin data directory, /var/www/munin/ is owned by ‘munin’ while Apache runs as ‘www-data’. The following commands fix this, but Apache is going to change the user ownership to ‘www-data’ when it saves a file by default, so if you try to switch back to munin-graph via cron, you’ll need to fix permissions again.

sudo chgrp -R www-data /var/www/munin
sudo chmod -R g+w /var/www/munin
sudo chgrp www-data /var/log/munin /var/log/munin/munin-graph.log
sudo chmod g+w /var/log/munin /var/log/munin/munin-graph.log

After the switch to fcgid generated munin graphs, generating all the graphs for a single node would take minutes and was quite painful. I gave the node more CPU resources, but it still took two minutes to draw a page of graphs. I ended up switching back to cron based graph generation. The additional CPU resources cut about forty seconds off the munin-graph time from cron, which is progress. Having the graphs immediately available when you need them is worth the cost of the CPU resources you could otherwise share that you would save from demand based graph generation via CGI. For the time being I intend to keep giving Munin more CPU until I find settle on a better way to do trending.

Note: this list was meant to fit within the requirements of this O’Reilly sweepstake. It’s not a comprehensive list of the books/videos/etc. I’d like to buy/read/view/etc.

• Being Geek (ebook)
• Classic Shell Scripting (ebook)
• Designing Web Interfaces (ebook)
• Designing Web Navigation (ebook)
• Diagnosing and Fixing MySQL Performance Problems (video)
• Great Bash (video)
• HTTP: The Definitive Guide (ebook)
• Mining the Social Web (ebook)
• Mobile Web High Performance (video)
• MySQL High Availability (ebook)
• sed & awk, Second Edition (ebook)
• SSH, The Secure Shell: The Definitive Guide, Second Edition (ebook)

The argument that Chef is difficult to learn because recipes are written in Ruby is a fallacy.

package "vim"

cookbook_file "/home/btm/.vimrc" do
  source "dot-vimrc"
  owner "btm"
  group "btm"
  mode "0644"
end

With the exception of the do/end block, that doesn’t look like a programming language at all and is way easier to grok than some configuration file syntaxes I’ve used. Any tool’s configuration file syntax has a learning curve and refusing to learn a new one means you’re going to be stuck in the past using old tools. Someone may not want to try out nginx today because they already know how to configure Apache, and I understand that up to a point. The tool you know is sometimes easier to use in the less than ideal conditions because you already understand it. I can’t spend all of my time learning new tools anymore than the next person, but frankly if you are unwilling to learn something new, you are in the wrong industry. We are moving fast over here.

Even if you don’t know any Ruby, over time you start reusing other people’s code shortcuts because it is easier to write understandable and flexible code.

# Install useful troubleshooting tools that get regular use
%w{htop dstat strace sysstat gdb tmux tshark}.each do |tool_package|
  package tool_package
end

# Install the correct apache package depending on distribution
package "apache2" do
  case node[:platform]
  when "centos","redhat","fedora","suse"
    package_name "httpd"
  when "debian","ubuntu"
    package_name "apache2"
  end
  action :install
end

Because Chef recipes are written in Ruby and they are compiled on the client rather than the server you can leverage Ruby in very powerful ways. When we want to create databases and grant privileges for a web application, we can use a number of Chef resources, primarily execute, to use existing tools such as mysqladmin. We can also leverage Ruby to access Ruby libraries. Ruby code in a Chef recipe is executed during convergence, but Ruby code in a ruby_block resource is executed along with other resources during compilation and can be notified like any other resource. You can get a better idea of when these steps happen from the Anatomy of a Chef Run page on the wiki. Here is some code I used recently that is quite a bit simpler to read and shorter than using resources to perform all of the steps.

    ruby_block "Create database + execute grants" do
      block do
        require 'rubygems'
        Gem.clear_paths
        require 'mysql'

        m = Mysql.new(mysql_host, "root", mysql_root_password)
        if !m.list_dbs.include?(node[:jira][:database_name])
          # Create the database
          Chef::Log.info "Creating mysql database #{node[:jira][:database_name]}"
          m.query("CREATE DATABASE #{node[:jira][:database_name]} CHARACTER SET utf8")

          # Grant and flush permissions
          Chef::Log.info "Granting access to #{node[:jira][:database_name]} for #{node[:jira][:database_user]}"
          m.query("GRANT ALL ON #{node[:jira][:database_name]}.* TO '#{node[:jira][:database_user]}'@'localhost' IDENTIFIED BY '#{node[:jira][:database_password]}'")
          m.reload
        end
      end
    end

Because Chef makes it easy to model data, you don’t need to write the above code. You can just use what I wrote and change your variable names. If you use it for more than one web_application, you could make it a cookbook definition or LWRP that you could extend as you need more features.

initialize_mysql_db "jiradb" do
  database_name node[:jira][:database_name]
  database_user node[:jira][:database_user]
  database_password node[:jira][:database_password]
end

Unicorn doesn’t have any monitoring hooks. Typically folks either put nginx in front and monitor response time, do some backlog magic and track errors or make guesses based on other available information. I’ve been using a modified version of the unicorn_status munin plugin for a while. It tracks CPU time for a thread and considers that thread idle if it hasn’t changed after sleeping for a second. This doesn’t pan out under load. Still, here it is.

#!/usr/bin/env ruby
#
# unicorn_status - A munin plugin for Linux to monitor unicorn processes
#
#  Copyright (C) 2010 Shinji Furuya - shinji.furuya@gmail.com
#  Copyright (C) 2010 Opscode, Inc. - Bryan McLellan <btm@loftninjas.org>
#    - Specify pid file via environment variable
#    - Do not assume process names
#  Licensed under the MIT license:
#  http://www.opensource.org/licenses/mit-license.php
#

module Munin
  class UnicornStatus

    def initialize
      @pid_file = ENV['UNICORN_PID']
    end

    def master_pid
      File.read(@pid_file).to_i
    end

    def worker_pids
      result = []
      ps_output = `ps w --ppid #{master_pid}`
      ps_output.each_line do |line|
        chunks = line.strip.split(/\s+/, 5)
        pid = chunks[0]
        result << pid.to_i if pid =~ /\A\d+\z/
      end
      result
    end

    def worker_count
      worker_pids.size
    end

    def idle_worker_count
      result = 0
      before_cpu = {}
      worker_pids.each do |pid|
        before_cpu[pid] = cpu_time(pid)
      end
      sleep 1
      after_cpu = {}
      worker_pids.each do |pid|
        after_cpu[pid] = cpu_time(pid)
      end
      worker_pids.each do |pid|
        result += 1 if after_cpu[pid] - before_cpu[pid] == 0
      end
      result
    end

    def cpu_time(pid)
      usr, sys = `cat /proc/#{pid}/stat | awk '{print $14,$15 }'`.strip.split(/\s+/).collect { |i| i.to_i }
      usr + sys
    end
  end
end

case ARGV[0]
when "autoconf"
  puts "yes"
when "config"
  puts "graph_title Unicorn - Status"
  puts "graph_args -l 0"
  puts "graph_vlabel number of workers"
  puts "graph_category Unicorn"
  puts "total_worker.label total_workers"
  puts "idle_worker.label idle_workers"
else
  m = Munin::UnicornStatus.new
  puts "total_worker.value #{m.worker_count}"
  puts "idle_worker.value #{m.idle_worker_count}"
end

And the configuration file:

$ sudo cat /etc/munin/plugin-conf.d/unicorn
      [unicorn_*]
      user root
      env.UNICORN_PID /etc/sv/opscode-chef/supervise/pid

I wrote another plugin today that uses raindrops to collect information about the active and queued connections. It is interesting how greatly active connections fluctuates. Thus, active connections don’t produce a stable munin graph, but having the queue depth recorded is pretty useful for tracking down latency issues.

#!/usr/bin/env ruby
#  Copyright: 2011 Opscode, Inc.
#  Author: Bryan McLellan <btm@loftninjas.org>
#
#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

require 'rubygems'
require 'raindrops'

def collect(port)
  # raindrops requires an array of strings, even if it denies this
  addr = [ "0.0.0.0:#{port}" ]
  stats = Raindrops::Linux.tcp_listener_stats(addr)

  puts "active.value #{stats[addr[0]].active}"
  puts "queued.value #{stats[addr[0]].queued}"
end

if ARGV[0] == "config"
  puts "graph_title Unicorn - connections"
  puts "graph_args -l 0"
  puts "graph_printf %6.0lf"
  puts "graph_vlabel connections"
  puts "graph_category Unicorn"
  puts "active.label active"
  puts "queued.label queued"
  exit 0
end

if $0 =~ /.*_(\d+)/
  # the munin wildcard format of plugin_value
  port = $1
elsif ARGV.size > 0
  port = ARGV[0]
else
  usage = "Usage: #$0 port or #{$0}_port"
  abort usage
end

collect(port)

Usage is the same as any wildcard munin plugin.

1. Install the raindrops gem
2. Drop the above code in “/usr/share/munin/plugins/unicorn_connections_”
3. Create a link from “/etc/munin/plugins/unicorn_connections_UNICORNPORT” to the above script
4. killall -HUP munin-node

Graphs should start showing up in five or ten minutes. You can always test like so:

$ nc localhost 4949
# munin node at unicorn.example.org
fetch unicorn_connections_6880
active.value 5
queued.value 0
.
quit

Of course, I use the Chef and the munin cookbook’s munin_plugin definition, so my application’s cookbook has this additional code:

# required for unicorn_connections_ munin plugin
gem_package "raindrops"

munin_plugin "unicorn_connections_" do
  plugin "unicorn_connections_6880"
  create_file true
end

The trend with init replacements

When you write a number of service resource providers for a configuration management system, you get some intimate experience with the quirks of init systems. A slew of new ones are working their way into stable releases lately which seem primarily motivated by decreasing system startup time by allowing services to be started in parallel. For instance, Ubuntu has been moving to upstart, the latest release of Debian uses insserv, and OS X uses launchd. There is overlap in design, and certainly parallel service execution isn’t the only significant improvement. Since init is a basic building block of our systems, small changes can cause large ripples. In the end we will have some great new functionality, but we’re in a rough patch of transition right now and need to ensure the functionality we rely upon doesn’t get passed over.

Disabling services with Upstart

If you want a service to not start on system startup, but still want to be able to start it, you have to comment out a line in the configuration file. Programmatically editing configuration files, from a script or a configuration management system is difficult to do cleanly. In general you want to avoid minor changes to configuration files because then you have to reconcile the differences when you upgrade the package. There are plans to add support for an override file wherein you can specify that the service is manual, but clearly Ubuntu server users are taking a backseat to desktop users inside Canonical where Upstart is developed.

Restarting services with Upstart

Which is interesting, as Ubuntu server related packages are being migrated to use Upstart. We start to run into additional quirks, such as when you restart a service that isn’t running, Upstart does not start it. We plan to work around this behavior in Chef but others have clearly taken notice.

$ status mysql
mysql start/running, process 548
$ sudo restart mysql
mysql start/running, process 649
$ sudo stop mysql
mysql stop/waiting
$ sudo restart mysql
restart: Unknown instance:

Insserv changes how you specify runlevels

On Debian lenny you could specify service runlevels and priorities as such:

$ sudo update-rc.d apache2 start 20 3 4 5 . stop 80 0 1 .
 Adding system startup for /etc/init.d/apache2 ...
   /etc/rc0.d/K80apache2 -> ../init.d/apache2
   /etc/rc1.d/K80apache2 -> ../init.d/apache2
   /etc/rc3.d/S20apache2 -> ../init.d/apache2
   /etc/rc4.d/S20apache2 -> ../init.d/apache2
   /etc/rc5.d/S20apache2 -> ../init.d/apache2

However on squeeze, update-rc.d is wrapped by insserv, which ignores your request and acts on the LSB headers.

$ sudo update-rc.d apache2 start 20 3 4 5 . stop 80 0 1 2 6 .
update-rc.d: using dependency based boot sequencing
update-rc.d: warning: apache2 start runlevel arguments (3 4 5) do not match LSB Default-Start values (2 3 4 5)
update-rc.d: warning: apache2 stop runlevel arguments (0 1 2 6) do not match LSB Default-Stop values (0 1 6)
$ find /etc/rc* -name '*apache*'
/etc/rc0.d/K01apache2
/etc/rc1.d/K01apache2
/etc/rc2.d/S18apache2
/etc/rc3.d/S18apache2
/etc/rc4.d/S18apache2
/etc/rc5.d/S18apache2
/etc/rc6.d/K01apache2

Insserv does have an option to override the LSB headers, but the update-rc.d wrapper doesn’t use it and you have to be very careful as it fails silently if you use it wrong.

$ sudo insserv -r apache2
$ sudo insserv apache2,start=3,4,5,stop=0,1,2,6
$ find /etc/rc* -name '*apache*'
/etc/rc0.d/K01apache2
/etc/rc1.d/K01apache2
/etc/rc2.d/K01apache2
/etc/rc2.d/S18apache2
/etc/rc3.d/S18apache2
/etc/rc4.d/S18apache2
/etc/rc5.d/S18apache2
/etc/rc6.d/K01apache2

Additional behavior to work around in Chef.

Moving forward

Distributions continue to change the way we interact with init with every release. This is clearly a reasons to use a configuration management tool. You know that you want mysql to never start automatically because your cluster resource manager controls it, but how you achieve that has been changing lately with regularity. You can let your configuration management tool abstract that from you. Still, we need to stay involved in the discussions in the open source communities whose software we use and be proactive citizens.

We’re rewriting the Mifos front-end from Struts 1.0/JSP to SpringMVC/Freemarker. So far it’s been slow going, so we’re trying out several experiments to speed up the process.

I think it’s important to measure the impact of our improvements. I want to know: are we moving faster? Is product quality flourishing? Can we say so quantitatively? The last bit has been the most elusive. It’s like measuring how useful a painting is!

Still, here are some measures which we hope will eventually show how much the improvements are or are not helping. Right now we’re just establishing baseline data.

The red line in the graph represents legacy front-end code that will eventually disappear. The green line is new code, but it really doesn’t matter.

From the chart data, we can say the following by measuring deltas between points: the most dramatic change in Struts/jsp LOC was 4380, for the time period ending Aug 2010.  The most dramatic change in SpringMVC/ftl LOC was 3665, for the time period ending Sep 2010.

Assumptions:

• these data are not useful for estimating speed of future refactoring work
• all acceptance tests pass in refactored areas
• look & feel of refactored areas is acceptable

The 2nd & third bullets are a bit vague. To address this shortfall, we’ll measure the following aspects of quality:

1. Time spent modifying CreateSavingsAccountTest.
2. Number of issues (regressions) caught during CSS walk through.
3. Number of layout or functional bugs logged during testing phase (missed during walk through).
4. Time spent by QA manually testing the feature, including logging issues and retesting bugs fixes.

Finally, developers should know if things are better. This is the most qualitative and possibly the most important “measure”! Certainly, for the migration of our codebase from svn to git, it was the only measure we used, and it was enough.

Thoughts/comments/feedback are appreciated. I’d rather have some kind of simpler measure, like “x increased by 250%”. Do you know of any more effective (and hopefully simpler) means of measuring development speed?

I used gnuplot to generate the graph. Here’s the source.

Nathan and I were discussing yesterday the lack of a good way to visualize all of the updates waiting to be installed across a server cluster. I wrote a another knife script to do this, and Seth Falcon helped me clean it up.

# Knife exec script to search for and describe systems needing updates
# 2011-01-11 - Bryan McLellan <btm@loftninjas.org>

gem "net-ssh", ">= 2.0.23"
require 'net/ssh/multi'

class AptSsh < Chef::Knife::Ssh
  # Override configure_session so we can specify where to get the query
  def configure_session
    @longest = 0 # Set in Chef::Knife::Ssh.run
    q = Chef::Search::Query.new
    @action_nodes = q.search(:node, ARGV[2])[0]
    fqdns = @action_nodes.map { |item| item.fqdn }
    if fqdns.empty?
      Chef::Log.fatal("No nodes returned from search!")
      exit 10
    end
    session_from_list(fqdns)
  end

  def capture_command(command, subsession=nil)
    host_data = Hash.new { |h, k| h[k] = "" }
    subsession ||= session
    command = fixup_sudo(command)
    subsession.open_channel do |ch|
      ch.request_pty
      ch.exec command do |ch, success|
        raise ArgumentError, "Cannot execute #{command}" unless success
        ch.on_data do |ichannel, data|
          host_data[ichannel[:host]] << data
          if data =~ /^knife sudo password: /
            ichannel.send_data("#{get_password}\n")
          end
        end
      end
    end
    session.loop
    return host_data
  end
end

abort("usage: knife exec apt.knife QUERY") unless ARGV[2]
ssh = AptSsh.new
ssh.configure_session

# install apt-show-versions if it isn't installed
install_show_versions = <<EOH
if [ ! -e /usr/bin/apt-show-versions ] ; then
  echo INSTALLING APT-SHOW-VERSIONS ; sudo apt-get install apt-show-versions -y
fi
EOH
ssh.ssh_command(install_show_versions)

apt_data = ssh.capture_command('apt-show-versions -u -b')

apt_data.each do |host, data|
  puts "#{host} - #{data.count("\n")} updates, #{data.scan("-security").length} of which are security updates"
  data.each_line do |line|
    puts "  #{line}"
  end
end

# Prevents knife from trying to execute any command line arguments as addtional script files, see CHEF-1973
exit 0

Given a search query, this provides an output of:

$ knife exec apt.knife role:dev
webui-dev.example.org - 6 updates, 6 of which are security updates
  libc-bin/lucid-security
  libc-dev-bin/lucid-security
  libc6/lucid-security
  libc6-dev/lucid-security
  libc6-i686/lucid-security
  libc6-xen/lucid-security
monitoring-dev.example.orgs - 6 updates, 6 of which are security updates
  libc-bin/lucid-security
  libc-dev-bin/lucid-security
  libc6/lucid-security
  libc6-dev/lucid-security
  libc6-i686/lucid-security
  libc6-xen/lucid-security
rabbitmq-dev.example.org - 6 updates, 6 of which are security updates
  libc-bin/lucid-security
  libc-dev-bin/lucid-security
  libc6/lucid-security
  libc6-dev/lucid-security
  libc6-i686/lucid-security
  libc6-xen/lucid-security
couchdb-dev.example.org - 7 updates, 7 of which are security updates
  libc-bin/lucid-security
  libc-dev-bin/lucid-security
  libc6/lucid-security
  libc6-dev/lucid-security
  xulrunner-1.9.2/lucid-security
  xulrunner-1.9.2-dev/lucid-security
  xulrunner-dev/lucid-security

Lets say that you didn’t want to upgrade the couch box, you could modify the search query to not include that box and run again to confirm. Then reuse that search string to trigger the update.

$ knife exec apt.knife "role:dev NOT hostname:couchdb-dev"
$ knife ssh "role:dev NOT hostname:couchdb-dev" "sudo apt-get upgrade -y"

We have a table in our corporate Confluence wiki that looks something like this. It was a product of a few quick notes to allow the team to build out VMs in parallel, distributed across a number of virtual hosts, and not rely on luck for proper resource utilization. The number fields are the amount of gigabytes of RAM allocated to the guests. As long as the total didn’t exceed a magic number for the entire host, we could keep building and the team remained unblocked. It got the job done, but it is no way to keep track of guests and resources. First, wiki’s have a tendency to get out of date and rot. It takes a fair amount of work to know what needs to be updated and keep it that way on a daily basis. Also, tables in Confluence are not all that great. They are far from Excel. The total row contains no formula to autosum the column, and you find yourself regularly switching between editor modes depending on how you are entering data, such as by hand or using cut and paste.

So, what if your “back of the napkin” calculations could be sourced from real data? This is usually unrealistic because you don’t know what data you need until you need it, so it hasn’t been captured. But we do capture a lot of data about nodes in Chef, so it is sitting there waiting for you to have that bright idea. In this case, I wanted to reconcile the memory usage on the VM hosts. I could ssh to each host, and collect this information from libvirt by hand, and put it in a spreadsheet somewhere or add it up myself for Confluence. But what happens when a teammate builds another server tomorrow? Will they update the documentation? Is that a step we want to keep doing by hand, as we build and destroy VMs on a regular basis? Is it a step we should be doing by hand, these days?

Chef::Log.level= :fatal
printf "%-10s %-12s %-8s %s\n", "host", "guest", "MB RAM", "Run List"
search(:node, 'role:virt').each do |host|
  total_mem = 0
  host[:virtualization][:domains].each do |domain,attribs|
    begin
      guest = nodes.show(domain)
    rescue
      guest = search(:node, "hostname:#{domain}")[0]
    end
    run_list = guest.run_list if guest
    printf "%-10s %-12s %-8s %s\n", host.name, domain, attribs[:memory] / 1024, run_list
    total_mem += attribs[:memory]
  end
  printf "%-10s %-12s %-8s %s\n", host.name, "TOTAL", total_mem / 1024, ""
end

This example is a knife exec script. If you saved this to a file named virt_ram.knife, then you could run it with knife exec virt_ram.knife. While Chef has full blown APIs you can interface with, that can raise the cost of a small project higher than its worth. With knife exec, small proof of concept projects done on the side of your desk are approachable with ease.

Let us take a moment to step through the code.

1 — Set the Chef log level to fatal to surpress warnings generated my line 7 when we look a non-existent node.
2 — Print out a header describing the columns of data we are going to generate
3 — Search chef for all of the nodes with the role “virt” and loop through them, naming the node object ‘host’
5 — Each virtual host object contains a hash of domains in host[:virtualization][:domains]. Step through these assigning the key to ‘domain’ and the value (another hash) to ‘attribs’
6-10 –  Look to see if we have a node in Chef whose name matches the domain name in libvirt. If not, rescue and trap that failure and try to search for a node with that hostname. Your node names in chef don’t have to be your hostnames or fqdns. At Opscode we use short unique identifiers such as EC2 instance IDs, portions of randomly generated GUIDs, and asset tracking numbers.
11 — If we did find a matching node, get its run_list. This really explains what a host does at Opscode, as we tend two only have two or three meta roles applied to a node. Usually one represents the environment it is in, such as “prod” or “dev” and the other is its role like “webserver” or “couchdb”
12 — Print out the information we known about this guest
13 — Then add the memory used by that guest to the running total for the host.
15 — Finally, print out the total memory we’ve calculated for that host.
16 — Go back around and do it all again for the next host.

$ knife exec virt_ram.knife
host guest        MB RAM   Run List
vm1  rv-735a342e  2048     role[prod], role[web]
vm1  rv-8ef1f3d1  4096     role[prod], role[database]
vm1  rv-eb574386  512      role[prod], role[dns]
vm1  TOTAL        6656
vm2  rv-91ba412e  2048     role[prod], role[web]
vm2  rv-8e342d11  4096     role[prod], role[database]
vm2  rv-e3829f86  512      role[prod], role[dns]
vm2  TOTAL        6656
vm3  cobbler1     1024
vm3  rv-e3829f86  512      role[prod], role[dns]
vm3  TOTAL        1536

This data is made up, but I’ve shown on vm3 something that I found in my own infrastructure; there were guests left over from testing that weren’t named properly and never made it into the chef server. I wouldn’t know they were there if I hadn’t done an audit of the servers this way. This exemplifies the Chef philosophy that it should help you do what you want, not model what it thinks you should be doing. This isn’t a carefully engineered reporting feature built around a common practice of virtualization management. This is a script I hacked on with Dan’s endless helpful guidance while I was waiting for an rsync to finish. I know others have written similar scripts to reconcile EC2 instances by comparing Chef and EC2 via Fog.

I love it. Do you have some spare time? What do you need? Chef will get you there.

Bachelors, health nuts, frugal folks, check this out.

Grrrraaaaaains!

Some rainy night, do this:

1. Buy a bunch of bulk spelt, red winter wheat, and barley from the bulk section of your grocery store.
2. Add 2/3 cup of each grain to a large pot (2 cups / 500 mL total grains).
3. Add 8 cups (2 L) water.
4. Bring to and keep boiling uncovered for 45 minutes.

Throw it in the fridge. Heat some up in the morning and dress it like oatmeal. I like honey and whole milk. Try stuff like fresh/dried fruit, brown sugar too. Freezes well.

45 minutes means the wheat and spelt will be chewy. That’s how I like it.

Beeeeans!

1. Add 2 cups (500 mL) raw pinto beans to a crockpot.
2. Cover with 3-4 inches (8-10 cm) water.
3. Cook on low for about 8 hours overnight.

Dress with like sliced fresh raw tomatoes, salsa, cottage cheese, sour cream, salt and pepper. Leftovers keep well for up to a week in the fridge, or freeze ‘em.

YUM!

Thanks Forest M and Pam W for the inspirations!

Odds & Ends

These grains/legumes/whatever are so tasty on their own. Seriously, why the heck do we need raisin bran and corn flakes?!

What’s your favorite breakfast?

For more fun, check out oat groats, amaranth, quinoa, and kasha.

Suggestions/corrections/feedback welcome. Yes Patrick, even your lovely trolling is welcome.

The right birthday card for a Minecraft alpha gift code.

Here is a PDF template,

and the source SVG from Inkscape. There is a commentary on Reddit with leads to a few other similar projects.

Knife’s exec sub-command makes it easier to interact with a Chef server from the command line. Let’s assume I’ve created a data bag named cluster as follows:

{
  "id": "www1",
  "cats": "lol",
  "hostname": "www1.example.org"
}
{
  "id": "www2",
  "cats": "lol",
  "hostname": "www2.example.org"
}
{
  "id": "www3",
  "cats": "plz",
  "hostname": "www3.example.org"
}

If I wanted to get a list of hostnames for each data bag item where the value of ‘cats’ is ‘lol’, I would run:

$ knife exec -E "search(:cluster, 'cats:lol').each {|host| puts host[:hostname] }"
www2.example.org
www1.example.org

Granted, I could get this data from the search sub-command as well:

$ knife search cluster cats:lol
{
  "start": 0,
  "total": 2,
  "rows": [
    {
      "id": "www2",
      "cats": "lol",
      "hostname": "www2.example.org"
    },
    {
      "id": "www1",
      "cats": "lol",
      "hostname": "www1.example.org"
    }
  ]
}

However, it is hard to manipulate the result of this data. For instance, if I wanted to to check the status of ntp on each of these nodes as a “one-off command”, I could run:

$ knife ssh -m \
"`knife exec -E "search(:cluster, 'cats:plz').each {|host| puts host[:hostname] }" | xargs`" \
'/etc/init.d/ntp status'
www1.example.org  * NTP server is running
www2.example.org  * NTP server is running

The quoting can get pretty tricky fast. Instead, if you leave off the -E flag to knife exec, you can pass a script file to knife where you can write clearer scripts, which makes it easier to do more.

# Script contents
$ cat /tmp/knife.exec
targets = Array.new
search(:cluster, 'cats:lol').each do |host|
  targets << host[:hostname]
end
puts targets.join(' ')

# Execute the script
$ knife exec /tmp/knife.exec
www2.example.org www1.example.org

What if you needed to reconcile your hardware support contracts with the systems currently deployed? It is no problem to get a list of hardware with chef and knife.

# Script contents
$ cat /tmp/dell.exec
search(:node, 'dmi_system_manufacturer:Dell*').each do |node|
  puts node[:dmi][:system][:serial_number] + "\t" + node[:fqdn]
end

# Execute the script
$ knife exec /tmp/dell.exec
XJS1NF1 www1.example.org
XJS1NF2 www2.example.org
XJS1NF3 www3.example.org

These are pretty simple examples, but hopefully you can see how easy it is with Chef to use knife scripts to create reports, collect data, and execute one-off commands.

Typically directions for downloading a i386 version of a library for a x86_64 system link to a specific deb package and tell you to download it with wget. A new release of that package often breaks the link, so I wanted to document how to do this using apt. Unfortunately, it looks like apt won’t download a single deb if it can’t resolve dependencies, but aptitude will, so we use them together.

I use a separate sources.list here just to speed up the process, as we need to correct apt when we’re finished.

# Download 32bit list files from the mirror specified in /tmp/sources.list
apt-get -o=APT::Architecture="i386" -o=Dir::Etc::sourcelist="/tmp/sources.list" -o=Dir::Etc::sourceparts="/dev/null" update
# Download the single library. Set libstdc++5 to whatever library you want
aptitude -o Apt::Architecture=i386 download libstdc++5
# Return apts lists to their preconfigured state
apt-get update
# Optionally, install the package
dpkg --force-architecture -i libstdc++5_1%3a3.3.6-20~lucid1_i386.deb

Note that if you install the package, it would overwrite the 64bit version of the library if it is installed. 32bit packages meant for 64bit systems, like the ia32-libs package, install to /lib32 and /usr/lib32 to avoid this. You could also extract the package with ‘dpkg -x libstdc++5_1%3a3.3.6-20~lucid1_i386.deb’ and copy the libraries to where you like, then run ‘ldconfig’. The getlibs tool will try to repack debs more appropriately for you, if you like.

The backlight of this Canon PowerShot SD400 doesn’t work anymore, but the rest of the LCD is fine:

I can order a new backlight for about $20 via eBay. The old one is worthless now, so I had a look inside. It disassembled into several layers:

On the left is a stack of thin plastic sheets that form the topmost layers of the backlight. They serve to diffuse light from the layer below.

On the right is thick plastic layer stacked on top of a reflective metal backing. Three tiny LEDs are affixed to the right edge of the plastic layer and shine directly into it from the side. This layer turned out to be birefringent with some curious optical properties:

You can see its purpose when the laser shines sideways into its edge. The whole surface lights up evenly.

Sometimes errors don’t float to the top of stacks well.

Our virtualization stack is pretty automated wherein we have a custom script that uses vmbuilder to create the guest, register it with libvirt, create first boot scripts that will have it register with a chef server, and start the VM. We saw this error today libvirtError: monitor socket did not show up.: Connection refused, and I commented that my memory contained a lot of libvirt/kvm errors, and many resolutions, but the two don’t always stay connected. I checked the libvirt logs in /var/log/libvirt and even ran libvirt with LIBVIRT_DEBUG=1 libvirtd -v. When I tried running kvm by hand using the syntax in the logs, but with the -net options removed from the command line, kvm just spouted Aborted. After starting at it for a bit, I noticed that instead of -m 1024 KVM was trying to run with -m 1073741824 (1024^3). This was due to a small conversion bug in our custom script.