Every year my family does a gift exchange. Names are chosen out of a hat to randomly assign pairs. Certain rules are followed:
I decided to give it the old college try in Python. Drawing names following the rules certainly posed more of a challenge than I anticipated! But I think I figured it out. At least, I figured out the “lazy programmer” way to solve it, not the statistician/np-complete/combinatorial/elegant mathy way to solve it.
This script can be used to “draw names” for a Christmas gift exchange.
It’s very brute-force, and works differently than a real-life drawing (what the Pierce fam currently does) in the following ways:
Runtime for the full draw is generally under .03 seconds. Here’s an example run for 2010:
$ ./gift_exchange.py Amika gives to Jim D.J. gives to Jose Grandma gives to Ruby Grandpa gives to Suzy Jim gives to Grandma Jose gives to D.J. Ruby gives to Grandpa Suzy gives to Amika
I didn’t do extensive testing, just ran it 10 million times or so and watched to see that it didn’t crash or hang.
Feedback/bug reports/patches/forks welcome!
Today is Whyday! One year ago today, the man behind _why suddenly completely removed the persona and all of his works from the net.
While we’re all still confused and sad in the wake of it, _why was a magnificent inspiration and his exaggerated playfulness and sense of whimsy were a breath of fresh air in the programming world and a perfect complement to the Ruby sensibility.
Though Ruby was his primary vehicle, his lessons, intentional and otherwise, transcend platforms. Have fun. Teach. Learn. Don’t be afraid to do something completely bizarre, for no apparent reason. Do this in whatever language on whatever platform is effective for those goals.
Cheers, _why. You’re welcome back any time.
Follow #whyday on Twitter to see what everyone’s doing in the spirit of _why today!
After a brief respite over the next few weeks, I’ll be joining Opscode in September. Not only am I excited about working with such a great group of people, but also for the incredible opportunity of getting to work on problems whose solutions are already beginning to permanently change how we build systems.
After all, we should be solving life’s more difficult problems, not passing our days as a cog in a machine of repetitious activity. When skilled and respected humans become mere automatons of deployment tasks; we’ve slipped into a dismal place. There is boundless room out there for innovation, but we need a dependable platform on which to build.
I’m joining Opscode to help craft this reality. I want to help you find meaning in these tools; how they will make your life easier. Don’t confuse this end as simply being able to work faster, it’s about working better.
So I just read Zed Shaw’s new post about health concerns for programmers and it sparked a new notion that I was able to quickly follow through on. In his post he describes the bad habits and neglectful behavior common to programmers that can lead to health problems, mild and severe.
None of it is particularly revelatory, but he does a great job of lining up a comprehensive listing of these problems and some suggested solutions. However you feel about Zed, it’s worth a read.
For myself, the biggest problem I have with correcting these mistakes (and surely I’m not alone) is awareness. Sure, bursts of pain in your lower back are an effective reminder that you need to address your posture, but how do you avoid getting to that point?
Scheduled, repeating tasks in Remember the Milk (my “to do” manager of choice) seemed inappropriate because they are too distracting. A “to do” creates a task for me and pulls my attention more completely away from what I’m working on. What I decided I needed were passive reminders to pull my shoulders away from my earlobes or gulp down some dihydrogen monoxide.
I did a quick search on the Ubuntu repositories for anything of the kind (you never know how far ahead people have gone with an idea before you arrive at it) but didn’t find anything, so I just did the following.
Yes I know “Bin” is odd, but by default all of the home subdirectories in Ubuntu are capitalized, so I followed that convention.
The reminder scripts are intentionally primitive. I don’t need another project to manage and wanted to solve this fast. Here’s an example:
#!/bin/sh notify-send "REMINDER: POSTURE" "Take a deep breath. Slowly release it while stretching your spine and relaxing your shoulders."
And its corresponding cron entry:
10,20,40,50 * * * * env DISPLAY=:0.0 /home/trevor/Bin/reminders/posture.sh
“env DISPLAY=:0.0” is how the cron job knows where to send the graphical results of your command. If you only have one display you can leave off the “.0” but I often have an external monitor attached and that directs the output to the primary display.
I do not fire the posture reminder at 0 or 30 minutes because that’s when I have another reminder telling me I should stand up and stretch for a moment. My expectation is that a pile of reminders all on top of each other would result in all of them being ignored.
Unfortunately these reminders are somewhat easy to miss due to their rapid expiration and complete lack of required acknowledgement. While notify-send does indeed support timeouts or setting an urgency level above or below the norm, the daemon Ubuntu uses to handle notifications does not, as this astonishing and disgusting bug report informs.
Strangely, despite notify-osd documentation to the contrary, the notifications expired at 10 seconds regardless of urgency. No more, no less.
Despite that, it seems to be meeting my goals: keeping my hydrated, and reminding me to avoid squirrel pose and shoulder earrings!
The Systems Administrator is likely the closet technological trade to skilled manual labor. They troubleshoot complex systems that others take for granted, until they fail, with a deceptive ease. Explaining to another how they had a hunch to look at a certain part of the system is either a retrospective tale of why it made sense, or a sarcastic nod to magic. This tale attempts to work out how one could have deduced the solution, but even if someone assembled a collection of symptoms and solutions into a step-by-step guide, they would not be able to replace the role of a Systems Administrator. Like an automotive mechanic can detect a blown head gasket from the smell of the oil, a Systems Administrator can sense infrastructure issues from how systems are behaving. And like a fondness for a make of automobile, we grow attached to Linux distributions that have treated us well and editors whose dark secrets we can manipulate skillfully.
I once had a student who didn’t understand why we couldn’t repair board-level hardware issues ourselves as easily as replacing a stick of memory, as their uncle was capable of repairing any engine problems by opening up the hood and quite literally “jiggling some wires.” A mystic knowledge exists in both worlds that is challenging to articulate to the layman. It can be difficult enough to explain a single component, but when a part of a system falls over and causes cascading failures in other parts of a system, outsiders are tempted to believe that they’ve just learned a truth about the solution. That is, that when certain symptoms occur, it is always caused by the failure of a particular part and that this part should be restarted to ’solve’ the problem. Yet, the experienced know that this only resolves the symptoms and the still problem lurks, now with fewer hints as to its origin.
The future is already here – it is just unevenly distributed. — William Gibson
The trouble with paradigm shifts is that they aren’t necessarily direct improvements on existing technology with a clear lineage. Critics ask why the new ways are better than that which they replace, and we struggle to draw the path that led us to this new place of understanding. The struggle is because instead of making a choice at a clear intersection of a path, we stepped through the bushes to another path not as obviously traveled. This alternate path may lead us to the same end, but its course has an entirely different shape.
To further exacerbate the problem, new innovations stand on the shoulders of giants. Some people have been convinced of the merits of leveraging cloud computing on a strictly financial basis, and have missed the tenants of Undifferentiated Heavy Lifting (UHL), where running servers and building networks may not be ones core business and ultimately a distraction. Some have yet to grasp the concept of treating systems, even built on internal hardware, as disposable, still accustomed to legacy processes of maintaining a system for the lifetime of the hardware.
It is essential to realize that these new technologies are not minor improvements to business as usual. Like the birth of globalization changing business around the world, nursed by the multi-modal shipping container’s head fake as just another way of moving cargo, todays innovations will surely reshape the face of operations permanently, in substantial and non-incremental ways.
Tarus Balog is the CEO of the OpenNMS Group, a company which funds the development of OpenNMS: FLOSS enterprise network monitoring software. OpenNMS lets you know when your machines go down, among other things. I use OpenNMS at work to keep Mifos infrastructure up and running: build servers, cloud, databases, etc.
Our corner of the Grameen Foundation focusing on Technology for Microfinance has a lot to learn from OpenNMS! They’re also FLOSS, and they’re profitable.
I talked with Tarus the other day. Turns out he’s a really cool guy. I’ve been following his blog closely for quite a while now, so I was thrilled with the chance to pick his brain. The point of the call was to find out why and how they run their developer summits: in-person meetings where coding, alignment, teambuilding and planning are plentiful. OpenNMS hosts these yearly, and we’d like to do them for Mifos.
We talked for a while, so I’ll summarize. I’ll cover the dev summit stuff first, then more about FLOSS business.
Adam: what instigated the first summit?
Tarus: I live on a farm in the middle of NC, wanted to get in touch with other devs. I invited them to hang out at my place and code for a week. The result was camaraderie and commonality of purpose. In other words: the hard stuff. We also got a lot of useful coding done. There were only 5 people, yet it was an awesome time.
A: How do you think things will be different if you didn’t have these summits
T: we had to cancel last year’s, and we felt the pain. We missed each other. We usually have dev tasks that are especially suited to summits that we didn’t do.
A: can anyone come?
T: yes. We sponsor active contributors. Full-time paid employees are pretty much required to go. Others must pay $1400 (includes room + board). This is modeled after debconf ( http://debconf.org ).
A: who comes?
T: 1/3 full-timers, 1/3 sponsored volunteers, 1/3 folks who pay their own way. Even with the folks who pay, dev summits are expensive and can run $20-$30k.
A: what’s the format?
T: loose/barcamp/vague. This works much better for us than a dictated organized agenda
A: What are your definitions of “success” for a summit? How do you know when you’ve had a successful summit?
T: we don’t really have a measure, but the value is apparent from the team cohesiveness, fun had, code produced, and productivity following the summits.
A: can I come?
T: yep
A: what do I need to do to run a successful summit?
T: plan it well. Make it fun. Buy food. Have someone go onsite and set up logistics: rooms, wifi, whiteboards, projectors, etc. Do stuff together. Discourage newbies (they need to pay to attend): don’t make it a training session: make it a *developer* summit. Publish prereqs: compile the code, run tests, study the schema, have your laptop ready, etc.
A: where are your devs?
T: global.
A: how do you reward volunteers and retain contributors?
T: LOTS of ways! “order of the X polo”: exclusive group of contributors are in the “order of the green polo”, these folks are the steering committee. OpenNMS is business driven, but the OGP drives the tech. They stick around because they are invested and we value what they say. They’re the experts. OpenNMS Group sells services, not software: training, consulting, installation, custom dev. Volunteers and contributors are invited to (and do!) do the cool stuff, the fun stuff, the most important stuff. We guarantee the code will not suck, and it will always be Free (as in freedom). also see “order of the blue polo”: that’s the one you should sign up for, Adam!
A: will do. Can you say more about what custom development you’ll have to do?
T: yes, but not just anything. Ex: someone wanted RT integration. Instead, we built a generic event-based framework and allowed them to hook into it. I don’t sell “custom dev” to a customer, I make them *sponsors* of features that make it into OpenNMS. Salespeople must have technical knowledge to make this work. The custom features aren’t what make them competitive anyway, it’s their business strategy, their processes, their people. We also sell commercial licenses for OEM arrangements. They pay for (us to do) custom dev, and we get the code, too.
I’ll be there, too!
Thanks to Adam Feuer for encouraging me to approach Tarus/OpenNMS.
I figured out a semi-cool setup to provide alternate views if the client is found to be, for instance, a Mobile Safari browser on an iPhone. I wanted to share my findings since I’ve seen related questions asked here or there, and I have a question at the end about a potential improvement.
First, I made a filter:
detectMobile (controller:'*', action:'*') { before = { if (request.getHeader('user-agent') =~ /(?i)iphone/) { request['isMobile'] = true } else { request['isMobile'] = false } return true // keep processing other filters and the action } after = { } afterView = { } }
Next, I added the following to a controller:
def afterInterceptor = { model, modelAndView -> if (request['isMobile']) { modelAndView.viewName = modelAndView.viewName + "_m" } }
For that controller, if a request comes in from a mobile device, a view ending in “_m” is loaded, instead of the default view. The view “list.gsp” would have a counterpart mobile view “list_m.gsp”, and that’s what would be served to iPhone clients. That’s it, so far.
One bit that would be handy would be to add this closure to every controller, rather than have to copy and paste it to controllers that support mobile views. I’ve been meaning to see how the rest plugin dynamically adds methods to every controller in case that is similar to what I’m trying to do, but if anyone knows offhand what I need to do (or read) to automatically get the closure in every controller, please let me know.
Are you busy clicking away at a slide deck for your next presentation? Want to try something a little different?
Non-nerdy folks following this blog, you’ll probably want to tune out now. Or not! I tried to write this for a wide audience, so come on down the rabbit hole if you dare.
Sometimes I give talks, and I like having a slide deck for these talks. It’s a useful and familiar idiom for most audiences, and it serves as a kind of roadmap as I walk through live demonstrations and deliver a message.
OpenOffice.org Impress, Powerpoint, or Google Docs all help you create a presentation. Sometimes these tools are fine, but sometimes they get in the way. Here’s an example.
I keep notes in plain text files. My notes look something like this:
making bread is fun smells great when baking yeast does the tricky work for you
Indented lines are subtopics of lines with less indenting. Hierarchical. Not fancy.
If I wanted to make this into a presentation, I could cut and paste it into some presentation-building software. Then format it. Then choose a nice background, save it, create PDFs, etc. But if I add to my notes, I have to also change the presentation. And I don’t want to open up Impress and start having to click fifty buttons and be compelled to pick a fancy background and diamond-swipe slide transition. Why can’t I just say “notes, become a presentation!”
For help, let’s turn to markup. When I say markup, I mean annotations to text that provide meaning. HTML is a kind of markup, for instance, but it doesn’t look great for a slideshow without a ton of work.
I dig markup. It’s generally readable. It encourages focusing on content/meaning over presentation/look & feel. It leverages powerful tools with consistent, deterministic results. Since markup is plain text, it can be usefully tracked/shared in version control.
There are lots of extensible, feature-rich markups, but I only considered lightweight markups to help with creating my presentation.
Then came the breakthrough: my notes are already markup!
Sure, my presentation should look nice. But the important part about my presentation is the content. Trying to put aside text size and spacing, graphics, animations, backgrounds, and bullet-point styles can be difficult while creating a presentation using tools like Impress. Markup can help isolate you from these details while still allowing customization of presentation.
There are several tool that generate presentations from markup, such as S5, Beamer, and MagicPoint. S5 looks cool, and I like that it does everything in a Web browser. But I don’t like authoring HTML so much. I’ve used Beamer, but I don’t like writing LaTeX either. MagicPoint looks cool too, but I didn’t know about it until, well, just now.
I decided to write my own. I designed a very lightweight markup for generating input for Beamer. Then I let Beamer/LaTeX take care of the grunt work of pretty-printing the content, putting images in the correct place, and creating presentation slides and presenter notes.
Create a plain text file:
Save it as “food.txt”. Indent lines 2 and 3 with a tab. It’s the same source code as above. Just make sure the whitespace at the start of the lines is a tab character.
Install prerequisites.
Run “otl2beamer”, then “pdflatex”, like so:
otl2beamer.py food.txt > food.tex pdflatex food.tex
Among the generated files will be food.pdf, the presentation.
The indentation in the plain text file has meaning. Here’s the basic markup syntax:
It does more, too. If anyone’s interested, I’ll document the other features.
I wouldn’t recommend using a markup-based presentation tool for a single presentation, but if you give talks often and are looking for a good way to manage your content, give it a shot! The setup time is significant, but the ongoing time savings, focus on content, and process improvement might just be worth it.
These are odds and ends an editor would likely toss had I an editor and were this ever sent to print.
If you’re saying “who needs slides, anyway?” but still want to show the audience something, you could do what Damian did: present your talk in a text editor. He wrote a talk on Vim, presented in Vim. I couldn’t convince him to release his presentation source, but I imagine it was just a gigantic Vim script.
I’ve been dying to use AsciiDoc for something.
I’ve created a Makefile to simply use of otl2beamer, as well as some magic for the Vim text editor to run the Makefile (and process the result) within Vim. Lastly, here’s a more extensive example presentation.
Here’s an interesting bit I stumbled upon while playing with Amazon Web Services (AWS) Elastic Load Balancing (ELB): HTTP requests must have their lines terminated with CRLF and not just a line feed. When using netcat to test a web server by speaking HTTP, it only sends LFs by default (\n). While RFC 2616 specifies:
… a bare CR or LF MUST NOT be substituted for CRLF within any of the HTTP control structures …
Using netcat to connect to a web server typically works just fine. I’m inputting the HTTP requests by hand and [ENTER] is where I hit the enter key.
$ nc www.google.com 80 GET / HTTP/1.0[ENTER] [ENTER] HTTP/1.0 200 OK Date: Fri, 09 Apr 2010 20:07:25 GMT Expires: -1 [snip]
This works against Apache. However when connecting to an Apache server through ELB, one must run netcat with the -C option to send a CRLF instead of a lone LF upon return.
$ nc -C elb.example.org 80 GET / HTTP/1.0[ENTER] [ENTER] HTTP/1.1 302 Found Content-Type: text/html; charset=iso-8859-1 Date: Fri, 09 Apr 2010 20:09:39 GMT Location: http://elb.example.org/404/ Server: Apache Vary: Accept-Encoding Content-Length: 290 Connection: Close
Sans the -C option, the connection simply hangs. Which asks the question, what is Amazon doing with your HTTP traffic in between?
I’m often a bit surprised by the lack of substance about trending that leaks out on the Internet. I mean, everybody is doing it. Right? Munin is a great introduction to trending due to its simplicity in getting started and the wealth of plugins.
I’m a believer of collecting as much data as possible and sorting it out later. Without data, you can only speculate wildly at what it might have said. So will others, so it’s nice having a response; often they won’t. I don’t need to be looking at the disk latency or available entropy for dozens of servers every day, but the time saved by being able to look at these graphs when something occurs and make correlations between trends is revolutionary to how you will spend your day. When having too much data can feel overwhelming, it’s time to post-process it into something more bite-size.
Still, I run operations for a web product and there is data I do want to see every day, both to monitor the health of the product and plan capacity for upcoming growth. Aggregating data for multiple systems and creating a sort of executive trending console helps accomplish this.
The best way to get familiar with munin is to install it on a debian or ubuntu workstation. Installing the ‘munin’ (server) and ‘munin-node’ (client) packages will be enough to generate some graphs about your local machine. Go ahead and run:
sudo su munin -s /bin/bash -c 'time /usr/bin/munin-cron'
Then point your browser at file:///var/cache/munin/www/index.html.
Aggregate graphs are created by munin-graph from existing data in the RRDs collected by munin-update. There are two types of aggregates: sum and stack. Sum will show you the total of multiple data sets. The Munin wiki uses the aggregate current between two UPS’s as an example. Sum is most useful when the data sets are relatively meaningless individually. For instance if you wanted to know the total current CPU usage in a 50-node cluster, each node is not particularly interesting alone, but the sum would be. Stack provides the data sets visually stacked on a single graph. The Munin wiki uses the total entropy between two systems as their example, which isn’t particularly interesting. I’ll use some similarly uninteresting examples, but later I’ll show one that produces a stack comparing data in multiple datacenters.
Lets look at a simple example /etc/munin/munin.conf file with an aggregate graph similar to what is in the munin.conf man page:
[localhost.localdomain] address 127.0.0.1 use_node_name yes [localdomain;Totals] update no load.graph_title 2xload load.double.stack one=localhost.localdomain:load.load two=localhost.localdomain:load.load
This will create a graph that shows the local systems load twice in a graph by stacking the same value.
Munin separates hosts by domain in more ways than just the html index that munin-html puts out. By default hosts are put into a “group” by their domain name. If an aggregate graph attempts to reference data values from a host in another group, munin may not find it and fail to clearly notify as to why. You can manually place a node in a group as we do above where we put the virtual host “Totals” in the “localdomain” group by entitling the section “[localdomain;Totals]” on line 5. Your groups can be called anything, they don’t have to be a domain name.
The “update no” directive on line 6 tells munin-update to skip this section, or hos since these graphs are created entirely from data collected from other hosts. Please note that you typically still need to run munin-update before munin-graph to get configuration changes to aggregate graphs to appear in the graph. Munin appears to bailout on drawing a graph if it sees no new data for that graph pretty early in the process.
Typically failures in this area of configuration result in a new graph not being created but munin-graph appearing to run successfully otherwise. Note that graph_title is required. If you see an error that looks like:
2010/04/08 18:43:46 [RRD ERROR] Unable to graph /var/cache/munin/www/localdomain/Totals/load-year.png : opening '': No such file or directory
This is because munin was unable to find a data set, or specifically the RRD file, based on the value you specified. Both of the following lines cause this error and the graph to not be drawn:
load.double.stack one=localhost.localdomain:load.load two=localhost.localdomainX:load.load load.double.stack one=localhost.localdomain:load.load two=localhost.localdomain:load.loadX
This syntax evaluates to:
graph.value.stack line=host.domain:plugin.value
Line, also called alias, ends up being the label for that line. Often dashes are inconsistently converted to underscores in Munin. I have a working plugin called ‘foo_js-3_0′, which I have to specify as ‘foo_js_3_0′ in the above syntax.
[localhost.localdomain]
address 127.0.0.1
use_node_name yes
[localdomain;Totals]
update no
load.graph_title 2xload
load.double.sum localhost.localdomain:load.load localhost.localdomain:load.load
load.double.label Double the load
Here is the same example but displayed as a sum. Note that we’ve added ‘load.double.label’, and this is required. This replaces the ‘alias’ or ‘line’ value we were just discussing in stacked graphs, which you will notice is no longer in the configuration line for ‘fieldname.sum’ on line 9.
Here is a proof of concept configuration that I made that counts some javascript calls in different datacenters
# Aggregrates
[example.org;OTS]
update no
contacts no
js-3_0.update no
js-3_0.graph_category example
js-3_0.graph_title CAPI3 OTS Calls
js-3_0.graph_total Total calls per minute
js-3_0.graph_scale no
js-3_0.graph_period minute
js-3_0.graph_args --base 1000 -l 0
js-3_0.graph_order iad irl las
js-3_0.total.graph no
js-3_0.iad.label IAD calls per minute
js-3_0.iad.sum \
iadots02.example.org:example_js_3_0.calls \
iadots01.example.org:example_js_3_0.calls
js-3_0.irl.label IRL calls per minute
js-3_0.irl.sum \
irlots02.example.org:example_js_3_0.calls \
irlots01.example.org:example_js_3_0.calls
js-3_0.las.label LAS calls per minute
js-3_0.las.sum \
lasots02.example.org:example_js_3_0.calls \
lasots03.example.org:example_js_3_0.calls \
lasots06.example.org:example_js_3_0.calls \
lasots04.example.org:example_js_3_0.calls \
lasots05.example.org:example_js_3_0.calls \
lasots01.example.org:example_js_3_0.calls
This creates the below graph. The jagged lines at the left edge are from missing data values while I was working out some of the issues I describe in this post. There are a couple new directives in this configuration. The ‘contacts’ directive on line 4 specifies that if we had munin configured for monitoring (as opposed to trending) we don’t want it to provide any notification based on the graph values for this virtual host. This is the job of munin-limits. The ‘graph_category’ directive allows us to put this graph in a category that we specify, otherwise Munin puts it in ‘other’. This is particularly useful if you have different types of aggregate graphs data such as CPU and Apache related data on the same virtual host. The ‘graph_total’ directive on line 9 isn’t that well documented but provides a simple way to add the black total line you see in the graph and is therefore quite useful. Lines 10-12 control how the graph is drawn and are outside the scope of this post. The ‘graph_order‘ directive seems to give us the ability to control the order in which the fields are drawn on the graph, but is documented as a method to control the order in which the graphs are drawn to specify complex data dependencies.
For fun, here is the Chef template that created this, which allows additional nodes be added automatically, but is still ultimately incomplete.
[example.org;OTS]
update no
contacts no
<% wop_datacenters = [ "iad", "irl", "las" ] -%>
js-3_0.update no
js-3_0.graph_category example
js-3_0.graph_title CAPI3 OTS Calls
js-3_0.graph_total Total calls per minute
js-3_0.graph_scale no
js-3_0.graph_period minute
js-3_0.graph_args --base 1000 -l 0
js-3_0.graph_order <%= wop_datacenters.join(" ") %>
js-3_0.total.graph no
<% wop_datacenters.each do |dc| -%>
js-3_0.<%= dc %>.label <%= dc.upcase %> calls per minute
js-3_0.<%= dc %>.sum \
<% dc_servers = @ots_servers.select { |host| host['hostname'] =~ Regexp.new(dc) }.select { |host| host['hostname'] !~ /pp/ } -%>
<% dc_servers.each_with_index do |host, index| -%>
<%= host['fqdn'] %>:example_js_3_0.calls <%= '\\' unless dc_servers.length - 1 == index %>
<% end -%>
<% end -%>
Debugging munin can be really tough. I keep stopping myself from breaking into explanation of munin’s process, but something as innocent as as an omitted ‘graph_title’ can cause munin to all but silently fail at producing a graph for you. Normally munin runs every give minutes via cron, usually via the ‘munin-cron’ wrapper, but you can run the parts individually to look for issues. These tools create a lockfile when they run so they won’t interfere with the regular process if it is started by cron.
user@localhost:~$ sudo su - munin -s /bin/bash
munin@localhost:~$ /usr/share/munin/munin-update --debug --nofork
munin@localhost:~$ /usr/share/munin/munin-graph --debug --nofork --nolazy
munin@localhost:~$ /usr/share/munin/munin-html --debug
In larger infrastructures, you can limit munin-update and munin-graph to specific host and service combinations while testing. Be wary that these sometimes will appear more successful than they are:
munin@localhost:~$ /usr/share/munin/munin-update --debug --nofork --host nonexistent --service nonexistent
2010/04/08 17:13:23 [DEBUG] Creating new lock file /tmp/munin-update.lock
2010/04/08 17:13:23 [DEBUG] Creating lock : /tmp/munin-update.lock succeeded
2010/04/08 17:13:23 [INFO]: Starting munin-update
2010/04/08 17:13:23 [DEBUG] Creating new lock file /tmp/munin-datafile.lock
2010/04/08 17:13:23 [DEBUG] Creating lock : /tmp/munin-datafile.lock succeeded
2010/04/08 17:13:23 [INFO]: Munin-update finished (0.00 sec)
I’m going to open this up with some blunt honesty. Every time I see someone I like or respect say something admiring about the iPad, I feel a little embarrassed for them. It’s like finding out your parents were duped by a 409 scam, or an acquaintance discussing the medical benefits of proximity to certain crystals.
Anyway, I don’t mean to string you along while I explore my feelings about a chunk of plastic. I just found the time today to finally read through all of the interviews on usesthis.com and was surprised to see the following.
”I’d like it if the Air was about half the size. I don’t know why Apple won’t make something in between the Air and an iPhone.” Paul Graham
“My biggest wish is for Apple to release a big-screen iPod Touch, so I can replace my Macbook Air with a more comfortable reading device.” Joe Hewitt
There’s little doubt that the the iPad will be a commercial success (it already is) but how exactly it will fit into our lives alongside smartphones, netbooks, notebooks, desktops, and all the rest will be interesting, as will seeing what it displaces along the way.
I’m really excited for the Mifos user manual sprint tomorrow. We’re using FLOSS Manuals to write our new user manual. FLOSS Manuals is an exciting way to write a book, it provides a framework for high-bandwidth collaboration, publishing to HTML and print, translating, and more. There are many examples of small teams (less than 10 people) publishing large, excellent books in short time periods (less than a week).
We’ll have a video feed going. I’ll be wearing my track suit.
Check out our coordination wiki page and stop by our IRC channel if you’d like to join in! If you’ve ever edited text on a wiki, are brave enough to learn how, or just want to see me in a track suit, come on by.
I suppose I should explain the title of this post. If we can establish a framework for software development on Mifos like we’re running this sprint, we can break Brooks’s law. Why not? The Ksplice folks did it, so we can too!
In less than obvious fashion, Amazon EC2 Windows Server AMIs reset their hostname on startup due to the Ec2ConfigService. To disable this feature, select ‘EC2ConfigService Settings’ from the start menu, and uncheck the first checkbox under ‘Set Computer Name’
by Andrew Gray (sweetandy@gmail.com) at January 25, 2010 11:36 AM
First, a bit of social commentary.
Sometimes we refer to the way something ought to be accomplished as the RightWay[tm], sarcastically noting that every best practice contains a certain degree of opinion. As we dedicate more time to doing something our way, we become invested in it being the RightWay, and risk becoming defensive about our choices. Adam Jacob calls this “survivorship-bias,” and I’ve spent some time listening to him think about what he feels the risks are, and considering them myself. When we make significant personal investment in a choice, it becomes a personal challenge to remain impartial about the merits of that choice over time.
I’ve previously said that Configuration Management is the act of programatically configuring your systems. Cloud computing says that building servers is undifferentiated heavy lifting; unless your service is building servers, you should really let someone else do it and focus on the product or service you’re actually trying to sell. Configuration Management is the first step in bringing this same ideology to configuring systems. We are not in the business of selling configured servers any more than we are in the business of providing coffee to our employees, we happen to do both. We build our systems to enable our business to conduct business. In my case, operations is enabling our customers to use the web product that we develop.
We are all members of the Configuration Management community, because we believe it is ultimately a better process for building systems. We naturally have different ideas about how that process should execute, which among other differentiating factors is often that “goals are different but are left unstated” in the community. The level of preference here and resulting fragmentation is not any different than holding an opinion over what open source operating system one should use for a specific task. The merits of our choices are worth discussing, but the implication that tools and libraries should all come to the same conclusions about design is like implying that the world only needs one type of hammer.
So, defining the meatcloud as the established notion that having your internet presence grow forms a direct relationship with hiring more people to rack servers, install software, and update configuration files; I asked around a little, why do we think Configuration Management is better?
You don’t need to be a mechanic to drive a car, should you need to be a subject matter expert on Apache to run a webserver? Infrastructure as code shows us how and the resulting communities are starting to implement this. When we spend less time getting the parts working, we can spend more time innovating better solutions with the parts.
Ever rebuild a server that someone built long ago and is no longer with the organization, and find many small parts necessary to make it work that nobody bothered to write down? Placing those small changes and required files in configuration management ensures that even if the code doesn’t run flawlessly on an upgraded operating system, you know everything that went in to making it work. Since you’re configuring the system through configuration management, a lot less falls through the cracks because documentation is often an afterthought to getting the system working.
What is on that system? Everything you told CM to put there. ‘Golden image‘ disk images often leave you in the aforementioned situation where you don’t know how to recreate them, but often you don’t know what else ended up there. Configuration Management allows you to build many copies of the same system easily, from scratch every time.
Did sales tell you they had fifty customers and it turned out to be five hundred? How long will it take you to build the extra servers by hand? How many extra people do you have to put into the meatcloud to get that done in time? Business requirements always change, and slow moving businesses are at a disadvantage to dealing with this. The inability to build and deploy servers fast enough should never be a reason your business fails.
Three applications on one server? Or one application on three servers? Apache doing different jobs on multiple servers? Moving applications between servers and leveraging existing infrastructure code for new projects is easy. We automate tasks that are repeatable, but often scripts are written to accomplish one repeatable task. Here we say, how can we treat configuration as sets of modular tasks that we can mix and match?
There was an innocuous-seeming update to Wine a few days ago, and upon attempting to play the one game I have installed via Wine, Runic Games’ astonishingly awesome dungeon-crawler, Torchlight, I found it suddenly mute. Which is not really how I like to play it. So I was perturbed.
After a few minutes of rechecking all of my audio mixer settings to make sure nothing had put them in disarray while I wasn’t looking, I turned to Google. There was the usual collection of outdated forum entries and unrelated flotsam, but one of the hits prompted me to visit the Wine App DB.
Lo and behold, there was a new addition to Torchlight's entry, leading to this bug report, #21248 "No sound in Torchlight, needs mmdevapi implemented".
I’m not familiar with how Wine development generally works, but I’m disturbed by the contents of that report. The audio is apparently silent because only partially implemented code has been put in place that (I assume) diverts the game’s audio to subsystems that don’t exist yet. It seems bizarre to me that incomplete code would not be checked into a parallel branch of development until it’s complete, but again, I’m not familiar with the Wine development traditions; maybe there’s a good reason.
In any case, after seeing a workaround described in the bug report, “Try adding the mmdevapi = none override in winecfg?”, I set out to discover exactly how one does that. Back in the day that was done in a configuration file, but now it’s in the GUI.
So the exact steps to fix it are:
Open Winecfg
Go to Libraries tab
Enter “mmdevapi” and click Add
Select the new mmdevapi entry and click Edit
Select “disable”, then OK both windows.
Upon restarting Wine, Torchlight should now be its awesome, atmospheric self again.
Well a couple people used the 0.7.16wt1 release, including Thom May who had issues with that version number. Consequently I took out the ‘wt’ from the version, but this is still an unofficial, unsupported release.
I pulled in the changes on Thom’s internal branch and grabbed some more low hanging fruit. Don’t ask what methodology I used. It’s magic.
Grab the github branch, or the debs in my launchpad ppa. I’ve only tested this on Ubuntu Karmic, as it goes.
** Bug
* [CHEF-454] – Centos4 yum provider failure
* [CHEF-616] – rake install in chef-repo breaks if there is no git origin
* [CHEF-637] – duplicate copies of FileEdit: file_edit.rb & fileedit.rb
* [CHEF-639] – git resource fails on subsequent checkouts of the same repostiry
* [CHEF-642] – Services will always issue a WARN when status is not present
* [CHEF-684] – Should be possible for roles to be created without anything in the run_list
* [CHEF-704] – Ruby block device does not have a default action
* [CHEF-722] – Link provider doesn’t understand paths requiring expansion
* [CHEF-843] – FileEdit permission issues
** Improvement
* [CHEF-598] – Upstart service provider
* [CHEF-617] – Install to chef repository on a remote machine
* [CHEF-709] – Support for backup up files in another directory than the original file
* [CHEF-752] – No way to pass parameters to useradd provider
I’ve periodically been having DNS lookup issues with internal domains and isolated them to remote SSL VPN clients connecting to a Cisco ASA 5520 using the Anyconnect SSL VPN client. I eventually got frustrated and troubleshooted the issue by using the command line ‘vpn’ client to initiate a connection on a remote Ubuntu Linux machine while here in the office. nslookup would produce the error “Got recursion not available from x.x.x.x, trying next server” and dig would respond with “status: REFUSED” and “;; WARNING: recursion requested but not available”. I noticed traffic was not making it to the Windows Server 2008 DNS server by watching wireshark and enabling DNS debugging.
Having been acquired six months ago our list of internal domains increased quite a bit. I found the ’split-dns’ setting in the default group access policy set to the old list of internal domains and set this to ’split-dns none’. This resolved the issue. Apparently the client was comparing the query to its list of split-dns domains, and the match was failing so it was sending the resolver (operating system) an error message so it would go through the list of DNS servers until it tried the local server. Rather than trying to make a list of all the possible domain names in the company, I’m going to leave this off since the internal DNS servers have recursion enabled and can handle DNS lookups just fine for the remote clients.
I sent an email out to the chef list today about releasing an unofficial Chef 0.7.16 fork to tide some of us over until 0.8 ships as Opscode does not plan any more 0.7.x releases. I use this internally at $job and pushed it out to github by request. I have no plans to push another release but will if there is demand internally or externally. This includes the security patches in 0.7.16 and a couple bug fixes:
* [CHEF-706] - mount provider fails to mount samba/cifs devices (Device does not exist)
* [CHEF-634] - UI fails silently when unable to save node
* [CHEF-659] - UI expands some escaped characters from JSON, then fails to encode them again
You can grab this tree from github, or use the debs built in my launchpad PPA.
Rather than making additional blog posts as the build for Chef 0.8 alpha has progressed, I’ve been updating this gist each time I test another build cycle. My original instructions are here. There is another blog posts here and some instructions for running under screen as well.
The Chef site starts out with, “Chef is a systems integration framework, built to bring the benefits of configuration management to your entire infrastructure.” There is an important point hidden in that description; Chef is not a CM tool. Of course it can be used as one, and many do, but from its beginning it has been leveraged by others such as Engine Yard inside of their own infrastructure. You can safely bet it will be an integral part of the Opscode platform when released as well.
While I was off dealing with my startup’s acquisition this fall, a chef user wrote knife. It was a command line client for interacting with a chef server. An impressively simple prototype, made possible and powered by the chef libraries and API. This has happened before with chef; for instance a while ago in the 0.6 era, an OS X GUI client called Casserole was written by another chef user with an itch to scratch. However, something has happened with knife that is interesting enough I’d like to specifically point out; it got mainlined and heavily expanded.
This happened for a handful of reasons. For one, it was simply a great idea. The kind of user who would be attracted to chef as a tool is very likely to be a power user who would rather not spend their time clicking around a graphical interface. It’s much easier to script a command line tool where needed, passing data in and out for quick hacks to your infrastructure. The folks at Opscode saw this, agreed, and set out to add full functionality to it for the upcoming 0.8 release.
What I think is most important is the planning a full API from the start. From hearing Adam talk about other tools being “first-class citizens” in chef-land, and knowing his experience writing iClassify as an early open source external node tool for puppet, I know this design was intentional. Using iClassify to tell puppet about your nodes was great, but puppet assumed that you only wanted this tool to classify nodes in the way puppet thought about nodes. Consequentially, when you wanted to to use data in iClassify about your nodes to make decisions about your infrastructure on the fly, you were forced to do it in templates. This created the expected repetition of loading the iClassify library and accessing iClassify in many templates, but also required you at times to do some fancy footwork to get data between templates when you really wanted puppet to know about the data itself.
Reductive Labs recently announced a dashboard for puppet. I was hoping this meant those barriers had been removed. It certainly creates really nice graphs from your puppet report data. However from the README it looks like you’re still pushing limited data into puppet using the external node interface. Reductive is going to have to expand this interface greatly if dashboard is to have any meaningful node integration benefits that we didn’t already have two years ago with iClassify.
Just as you can see some concepts from other configuration management tools in chef, you can see parts of iClassify. It was a great start and it was disappointing that the puppet community didn’t engage it further. Perhaps it was simply before its time, but I believe it was that there were too few doors into puppet-land to let you really take advantage of and grow external tools.
I think this was the lesson that Opscode learned, and consequently chef was born with an API. With it we can accomplish nearly anything we dream up. What is most exciting about this is that we can do whatever everyone else dreams up. I can’t wait to see what that is.
by Andrew Gray (sweetandy@gmail.com) at December 17, 2009 06:24 AM
Retrieving the COM class factory for component with CLSID {BA785E28-3D7B-47AE-A4F9-4784F61B598A} failed due to the following error: 80070005. (Microsoft.SqlServer.ManagedDTS)
Trying to access SSIS (DTS) on Microsoft SQL 2008 with SSMS (SQL Server Management Studio) on Microsoft Windows Server 2008 gave the above error. Trying to create a maintenance plan provided the same error, since it uses SSIS. There were indications online that I should try running SSMS with elevated permissions using the ‘Run as administrator’ option on the context (right-click) menu, however that provided a “The parameter is incorrect” error on startup. Eventually I discovered that the disk that the SQL tools were installed on did not have default (R+X) permissions to the local users group. Once I added this group, I was able to connect to SSIS and create a maintenance plan without issue.
by Andrew Gray (sweetandy@gmail.com) at December 15, 2009 01:51 PM
On Ubuntu 9.04 (jaunty) I had been generating and setting the root password in a bootstrapping script using:
# Generated MD5 encrypted password
/usr/bin/openssl passwd -1
# Set the password
/bin/echo 'root:ENCRYPTED_PASSWORD' | /usr/sbin/chpasswd -e
With shadow 4.1.4, chpasswd now uses PAM, and has dropped the -e option used above, as well as the -c option that I’d used to generate sha512 encrypted passwords. You’ll want to use mkpasswd from the whois package (yeah, weird) for that now, such as:
mkpasswd -m sha-512 -s
The password can be presented to useradd / usermod in encrypted format, such as:
/usr/sbin/useradd -m -p 'ENCRYPTED_PASSWORD' -G admin -s /bin/bash toor
by Andrew Gray (sweetandy@gmail.com) at December 03, 2009 08:14 AM
Theres push to get Chef 0.8 out the door because we’re all anxious for its awesome list of features and fixes, so we’re all hunkering down on fixing bugs. Scott Likens has similar notes and theres some more to be found in Dan Deleo’s 08boot bootstrap recipe. This should help get you going.
On a fresh Ubuntu Karmic install (a VM makes this easy of course):
# Add the Canonical Ubuntu ‘multiverse’ repository for Java.
sudo vi /etc/apt/sources.list # add multiverse to your ‘deb’ lines if it is not there
sudo apt-get update
# Start Chef Gem bootstrap with some notes
# note that I don’t like to install rubygems from source and use the packages instead. this adds a step or two.
sudo apt-get install ruby ruby1.8-dev libopenssl-ruby1.8 rdoc ri irb build-essential wget ssl-cert rubygems git-core -y
sudo gem sources -a http://gems.opscode.com
sudo gem sources -a http://gemcutter.org # for nanite
sudo gem install ohai chef json –no-ri –no-rdoc
We now have enough chef to bootstrap ourselves
# Create ~/chef.json:
{
"bootstrap": {
"chef": {
"url_type": "http",
"init_style": "runit",
"path": "/srv/chef",
"serve_path": "/srv/chef",
"server_fqdn": "localhost"
}
},
"recipes": "bootstrap::server"
}
# End of file
# Create ~/solo.rb:
file_cache_path "/tmp/chef-solo" cookbook_path "/tmp/chef-solo/cookbooks" # End of file
mkdir /tmp/chef-solo
cd /tmp/chef-solo
# Get kallistec’s 08boot bootstrap cookbook
git clone git://github.com/danielsdeleo/cookbooks.git
cd cookbooks
git checkout 08boot
# Bootstrap chef
sudo /var/lib/gems/1.8/bin/chef-solo -j ~/chef.json -c ~/solo.rb
# If the bootstrap hangs for more than a minute after “Installing package[couchdb] version 0.10.0-0ubuntu3″ then hit ctrl+c and run again
Now prepare to install the development versions
# install some development tools
sudo apt-get install rake librspec-ruby -y
sudo gem install cucumber merb-core nanite jeweler uuidtools
# install missing dependencies
sudo apt-get install libxml-ruby thin -y
# get chef from the repository
mkdir ~/src
cd ~/src
git clone git://github.com/opscode/chef.git
cd chef
rake install
# remove the old version of chef
sudo gem uninstall chef -v0.7.14
# patch up some runit paths
sudo sed -i s_chef-_/var/lib/gems/1.8/gems/chef-solr-0.8.0/bin/chef-_ /etc/sv/chef-solr*/run
# allow access to futon for development purposes (http://IPADDRESS:5984/_utils)
sudo sed -i ’s/;bind_address = 127.0.0.1/bind_address = 0.0.0.0/’ /etc/couchdb/local.ini
sudo apt-get install psmisc # for killall
sudo /etc/init.d/couchdb stop
sudo killall -15 couchdb # stubborn
sudo killall -15 beam.smp # yup
# shut it all down
sudo /etc/init.d/chef-solr stop
sudo /etc/init.d/chef-solr-indexer stop
sudo /etc/init.d/chef-solr-client stop
sudo /etc/init.d/chef-client stop
sudo /etc/init.d/chef-server stop
sudo killall -15 chef-server
Build some data and start up Chef
# start up the integration environment
cd ~/src/chef
sudo rake dev:features
# this will create a database
# now hit ctrl+c
sudo mv /var/lib/couchdb/0.10.0/chef_integration.couch /var/lib/couchdb/0.10.0/chef.couch
sudo chown couchdb:couchdb /var/lib/couchdb/0.10.0/chef.couch
# start it all up
sudo /etc/init.d/couchdb start
sudo /etc/init.d/rabbitmq-server start
sudo /etc/init.d/chef-solr start
sudo /etc/init.d/chef-solr-indexer start
sudo /etc/init.d/chef-server start
Start the web server
# the web server is now a separate application and uses the API to reach the server
sudo cp /tmp/chef_integration/webui.pem /etc/chef
cd ~/src/chef/chef-server-webui
sudo /var/lib/gems/1.8/bin/slice -p 4002
Using knife
From the user interface you can create a client keypair to use knife from the web interface. I recommend using ‘view source’ to copy the private key, and remember to save it without any leading whitespace and run knife like so:
OPSCODE_USER=’btm’ OPSCODE_KEY=’/home/btm/btm.key’ /var/lib/gems/1.8/bin/knife
If you can’t get it to work, you can always use the webui’s key:
sudo OPSCODE_USER=’chef-webui’ OPSCODE_KEY=’/etc/chef/webui.pem’ /var/lib/gems/1.8/bin/knife
Hopefully that is enough to get you going. Jump on #chef on irc.freenode.net or join the chef list if you have any problems. Tickets/bugs/features are tracked in JIRA, and all sorts of other useful information is in the wiki.
I recently had to install HP’s System Management Homepage (SMH) on Ubuntu Karmic (9.10) on hardware I had never touched for Hosted Operations to monitor. The hardware wasn’t my choice, but I’m indifferent to it. The operating system is my choice. Apparently they support Debian Lenny (5.0) and Ubuntu Jaunty (9.04), but ours was too new. However, while I commend them for building debs, they’re a little sketchy and broken. Granted, I wasn’t deploying to a supported release, but nonetheless. Here’s a link to download options for the DL360 G6, that may never work because the HP site isn’t meant to be linked to.
Downloading the provided Ubuntu Jaunty iso and mounting it produced a standard debian repository tree for both lenny and jaunty.
sudo mount -o loop HP_ProLiant_Value_Add_Software-8.25-19-12.iso /mnt
I added these packages to our local respository, but you can copy them to every server and install them by hand using ‘dpkg -i DEB’ instead of ‘apt-get install PACKAGE’. You’ll end up installing all of them really. The HP SMH package is mostly an apache fork and a ton of included/vendored libraries.
You’ll log in to HP SMH on port 2381 over HTTPS. As usual, if you get a data stream, you are likely connecting over HTTP by accident. By default a user must be in the ‘root’ group. You can use ‘vigr’ to add another user to the root group as you usually don’t have a root user on Ubuntu. You can edit ‘/opt/hp/hpsmh/conf/smhpd.xml’ and put another group in the ‘admin-group’ element. I put ‘domain-admins’ there because we use likewise to authenticate against a windows domain. I couldn’t figure out how to have groups added via the web interface to save, but that really was a hoop anyway since I wanted to push the configuration files out via configuration management.
I don’t know if HP SMH reads snmpd.conf to figure out how to connect back to snmp locally, but I had to initially run ‘/sbin/hpsnmpconfig’ to generate a few wizardy lines in /etc/snmp/snmpd.conf’. I later pushed this out via configuration management, but if you check that script it does create an “answer file” which looked like a bunch of variables you could export before you run the script non-interactively.
HP SMH gets its information from the HP SNMP agents, so if you log in and don’t see any data, it can not contact the SNMP source. You should see a page like this. Because so many libraries are shipped in the debs rather than being required, libraries are the most common source of issues. I had to restart ‘hp-snmp-agents’ after installation, getting this error on the initial startup in ‘/var/log/hp-snmp-agents/cma.log’:
libcmacommon.so.1: cannot open shared object file: No such file or directory
Another way to say all of this is via my chef recipe:
# # Cookbook Name:: hpsmh # Recipe:: default # # Copyright 2009, Webtrends # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Restart hp-snmp-agents later. it is buggy and has issues with its own libraries when started on package installation service "hp-snmp-agents" do action :nothing end package "hp-health" package "hpacucli" package "cpqacuxe" package "hp-snmp-agents" do notifies :restart, resources(:service => "hp-snmp-agents") end package "hp-smh-templates" package "hpsmh" service "hpsmhd" do action [ :start, :enable ] end service "snmpd" do action [ :start, :enable ] end remote_file "/opt/hp/hpsmh/conf/smhpd.xml" do source "smhpd.xml" owner "root" group "root" mode 0644 notifies :restart, resources(:service => "hpsmhd") end remote_file "/etc/snmp/snmpd.conf" do source "snmpd.conf" owner "root" group "root" mode 0644 notifies :restart, resources(:service => "snmpd") end
As I look back at growing up in a small town, there was a surprising lack of everyone wanting to move to the city as soon as they could. Perhaps that was because there is not a recognizible city anywhere near coastal eastern Maine. Despite, there still was a lingering belief that people were different elsewhere. Granted, they’re different, but in the same ways.
The majority of those I consider my colleagues have not worked for the same companies that I have. While our projects are of importance to our companies, it is usually our passion and not our employment that drive them. Some days I feel certain this is commonly understood, but it only takes a personal blog policy or a social media marketing drive to remind me that I’m actually isolated on an island of like-minded individuals hiding under the radar like stowaways. You can’t escape culture, but you can find different ones.
In Paul Graham’s recent essay about Apple, he markedly warns of mistreating the developers of your platform, lest they form a distaste for your brand altogether. Before I read the essay I was feeling quite sure that it was commonly understood today that developers are your greatest asset. Perhaps more valuable than even your big idea. Likely due to being mentioned by name in the essay, I was reminded of the great Google interview algorithm; commonly known for streamlining their processes at the cost of the interviewee. This seems to only alienate the prospect, unless they happen to enjoy passing tests over creating value. As the strengths of mass-collaboration become more accepted, it strikes me odd that on the whole we’re still missing that it is made up of individual human talent.
The product of our creativity is no longer hidden behind towering walls of corporations. We are global citizens innovating for the sake of it. You won’t see this on a college transcript, in ones knowledge of inodes, or in a six month product road map of release stability. The pieces are not exactly hidden either. I’m tempted to point out how slowly we’re changing by example with the United States’ difficulty transitioning from educating factory workers to innovators now that globalization has helped much of the the rest of the world catch up as industrial nations. However I can’t help but remember that we’ve gotten this far on our own.
Despite reminding us that we are living in a small town, the murmuring you’ve heard from pundits and rabble-rousers but could not make out sounds perfectly clear here. We are not going to wait for you to get it. The catch is that we no longer need to move to the city, because we’re building it every day. Coming?
I have a number of Windows Server 2008 hosts running under KVM in a remote datacenter and using virt-manager to access libvirt+kvm over SSH for a remote console was disappointly slow, so I set out to try libvirt+kvm over SSL/TLS for comparison. In the process of which I had to upgrade virt-manager to 0.8.0 on my workstation to remove a VNC lag issue in the viewer built into virt-manager on karmic. In the end, I’m quite happy with the end result.
The available documentation for configuring TLS authentication for libvirt is a little daunting. My chosen references were these documents for libvirtd and virt-manager.
First create two certificates, one with the hostname for your server (SERVER.EXAMPLE.ORG below) and one for your workstation (CLIENT.EXAMPLE.ORG below), setting the fully qualified domain name (FQDN, the hostname including the domain name) as the Common Name, or CN, when prompted.
openssl genrsa -des3 -out host.example.org.tmp
openssl rsa -in host.example.org.tmp -out host.example.org.key
rm host.example.org.tmp
openssl req -new -key host.example.org.key -out host.example.org.csr
Providing the CSR to your local CA should present you with a signed certificate (host.example.com.cer). Be sure it is in Base64 format (text) and not DER (binary) if interacting with a Microsoft CA. If you are unfamiliar with this process, you’ll want to go and read up a bit first, it’s a useful hoop to learn to jump through.
# Libvirt Server
mkdir -p /etc/pki/CA
mkdir -p /etc/pki/libvirt/private
mkdir -p /etc/pki/libvirt-vnc
cp CA.EXAMPLE.ORG /etc/pki/CA/cacert.pem
cp CA.EXAMPLE.ORG /etc/pki/libvirt-vnc/ca-cert.pem
cp SERVER.EXAMPLE.ORG.key /etc/pki/libvirt/private/serverkey.pem
cp SERVER.EXAMPLE.ORG.key /etc/pki/libvirt-vnc/server-key.pem
cp SERVER.EXAMPLE.ORG.cer /etc/pki/libvirt/servercert.pem
cp SERVER.EXAMPLE.ORG.cer /etc/pki/libvirt-vnc/server-cert.pem
# Virt-manager client
# notice the lack of dashes in clientcert.pem
mkdir -p /etc/pki/CA
mkdir -p /etc/pki/libvirt/private
mkdir -p /etc/pki/libvirt-vnc
cp CA.EXAMPLE.ORG /etc/pki/CA/cacert.pem
cp CA.EXAMPLE.ORG /etc/pki/libvirt-vnc/ca-cert.pem
cp CLIENT.EXAMPLE.ORG.key /etc/pki/libvirt/private/clientkey.pem
cp CLIENT.EXAMPLE.ORG.key /etc/pki/libvirt-vnc/clientkey.pem
cp CLIENT.EXAMPLE.ORG.cer /etc/pki/libvirt/clientcert.pem
cp CLIENT.EXAMPLE.ORG.cer /etc/pki/libvirt-vnc/clientcert.pem
If libvirtd fails to start (which it likely will without these fixes) you’ll see errors in ‘/var/log/user.log’ such as:
Nov 17 17:08:09 lasvirt01 kernel: [69476.008895] type=1503 audit(1258506489.178:77): operation=”open” pid=17104 parent=1 profile=”libvirt-600d5dae-6373-107e-5f1b-5010aff3ffed” requested_mask=”r::” denied_mask=”r::” fsuid=0 ouid=0 name=”/etc/pki/libvirt-vnc/ca-cert.pem”
You’ll need to patch up the apparmor definitions a little:
By default libvirt and VNC don’t allow remote connections and rely on unix sockets for connections. You’ll need to enable
Add a new connection, choosing ‘QEMU/KVM’ as the Hypervisor and ‘Remote SSL/TLS with x509 certificate’ as the Connection, with the full hostname of the remote server and choose Connect. Otherwise operate like you used to.
I had an issue where the display was lagging every view seconds on the VNC session via virt-manager, but when I connected directly using virt-viewer (virt-viewer –connect qemu://HOST.EXAMPLE.ORG/system GUEST_DOMAIN) there was no lag. Interestingly, when you kept both open at the same and ran a command like ‘ls -lR /’ you could easily see the difference. To correct this, I had to backport virt-manager=0.8.0-2 from debian sid, including ‘virtinst’ and ‘libvirt’, the latter requiring a change of ‘libxen3-dev’ to ‘libxen-dev’ in the Build-Depends. This is a somewhat complicated task for those unfamiliar with debian packaging.
I found most of my apparmor related errors by running ‘tail -f /var/log/user.log’. A lot of documentation recommended uncommenting ‘vnc_listen’ in ‘/etc/libvirt/qemu.conf’ but I found that the aforementioned listen xml attribute in the guest configuration overrode that by looking at the ‘-vnc’ options passed to kvm by libvirt by running ‘ps ax’. I’ve had libvirtd on the host segfault a couple times when connecting, perusing the Changelog in the version of libvirt I backported on my desktop noted some fixes that may be relevant to that.
by Andrew Gray (sweetandy@gmail.com) at November 10, 2009 04:54 PM
Savorous, as well as killthe.net, is built on the Sinatra framework and served via Apache and the Passenger extension, with Rack somewhere in the middle.
By default, the application will be loaded with the production environment unless you specify otherwise. I do so in the application’s rackup file, config.ru, like so:
require 'savorous' set :environment, :development run Sinatra::Application
But of course that’s not appropriate for production. So I have a second file, config.ru.production, and in my config/deploy.rb, which the Capfile calls, I have:
namespace :configure do
desc "Putting production rackup file in place"
task :rackup, :roles => :app do
run "rm #{File.join(current_path,'config.ru')}"
run "mv #{File.join(current_path,'config.ru.production')} #{File.join(current_path,'config.ru')}"
end
end
after("deploy:symlink") do
configure.rackup
end
I’m still learning all the ins and outs of Capistrano, so this may not be the best way to do it (though it works). I simply looked for the last task to occur before restart was called and inserted my configure.rackup call there. Good enough!
In trying to expand and solidify my facility with Ruby and related things, as well as scratching my own itch, I’ve started a new project: Savorous.
The problem it’s meant to solve is filling in the missing data in feeds that only deliver bare links without the associated summary or meta-data. It’s made specifically with Delicious feeds in mind, but it could be any feed that is basically just a list of links.
Because the links lack that information, I’m compelled to open each of them with even the tiniest possibility of appreciable content behind it, which can be (and often is) a big waste of time.
The site is ready to go in only the thinnest proof-of-concept sense, and you’re welcome to try it out at Savorous.info.
While my intent is to both cache the fill-in data and to assertively crawl for it, which is a huge undertaking and not something I could tackle up-front, currently everything is done on the fly and no data is stored. This means it’s slow. It hopefully won’t matter much because you won’t be hitting the service on-demand anyway, you’ll be seeing the results of your feed reader’s visits to the service.
Savorous is as simple as I could make it (and no simpler). You can use the form on the front page to generate the URL for your Savorous feed, or simply append your feed URL to “http://savorous.info/?url=” without waiting for the form and its machinery to do their thing.
Subscribe to the resulting URL, and see how it works out for you. I welcome any and all feedback at the email shown here.
Really rough, but easy to drop in and get working with something else, either as an external script or by integrating directly into a larger code base.
The code:
PHP: Gist 214887
Ruby: Gist 222662
(I confirmed embedded gist breakage in Fx3/Lnx & IE8 after @clintziegler tipped me off. Swapped with boring ol’ links for now!)
Last week I spent a few minutes over lunch cleaning up the code behind killthe.net and posting the results to GitHub.
There’s nothing terribly interesting about it if you’re already familiar with the Sinatra micro-framework for Ruby, but I think it’s a decent starter app for someone new to Sinatra or Rack who wants to see how some things fit together.
As described more fully in the README, the app simply checked the Delicious.com “killthenet” tag feed for new entries and adds them to its database. The web interface is just a simple listing of those entries, with minimal search available as well.
It’s a very thin example that ties a lot of technologies together, so I’m sure there’s a helpful example for somebody in there.
by Andrew Gray (sweetandy@gmail.com) at November 04, 2009 07:57 AM
It’s no secret that I think configuration management is epic. I spent yesterday hopping around the office in excitement due to Opscode’s alpha announcement of their new Cookbooks site. This is exactly the next step needed in the CM community. Shortly thereafter, while talking with another developer about how awesome Cookbooks is, he quipped about it being second in awesome to chef itself. I argue that chef exists for cookbooks, not the other way around. It was later said “Chef is the tool, cookbooks are the art”. Cookbooks is all about community.
Chef is a tool, and an open-source one, so it does have a community. A vibrant one at that; a recent ohloh factoid claims “This is one of the largest open-source teams in the world, and is in the top 2% of all project teams on Ohloh.” Cookbooks is the other way around, it is a community first and at tool second. Cookbooks has already been compared to github and launchpad (with PPAs), because like these sites it brings people with a common task together around a tool like git or ubuntu. It has been noted that every configuration management project needs something like Cookbooks, and I agree.
The community that builds around Cookbooks will be a vanguard of agile system administrators looking to reduce their undifferentiated heavy lifting. These include the people who recognized that it is their product that they are selling and no longer their infrastructure, and took up cloud computing as a result. They will soon find that the configuration management community will help them continue to spend less time reinventing the operations wheel, and more time innovating it. Cookbooks will make it easier to share infrastructure-code with like-minded individuals and is the next step beyond github and wikis for this material. These models still required concerted effort to share back with the community when you finished up your recipe, but the Cookbooks site and it’s open API stands to change that. We are now poised to leverage the open source model to increase the rise of infrastructure as code.
So it’s been awhile since I checked in with any HYDRAtweet news. Things are still going on in the background:
After some continued pushback from CodeIgniter, I finally came to the conclusion that what I really wanted and needed, was Rails. So I made my second platform hop, this time completely away from PHP.
The specific considerations that went into this decision centered around my continued velocity as I worked on the application. With CodeIgniter, I felt I gained a bit of time up front, but didn’t really save any going forward. Not that there was a lot of friction with ongoing development, but it wasn’t as fast as I feel it could be with the right framework.
There were some things here and there that I felt Rails gets right and CodeIgniter does not, but the final straw was when I drafted my crawling strategy—the automated collection of Twitter data—and realized there was no provided way to include CLI scripts that were a part of the CI app and had access to all of the code I was building there. And of course Rails has Rake.
My personal history with Rails and even Ruby is spotty. I pick it up for a little bit, as often for personal projects as one-off day job needs, and then because I’m usually employed to work with PHP, six months or a year goes by before I touch it again. But I always enjoy working with Ruby, and Rails is likewise a pleasure to use.
So at this point I’ve written some parts of the application three times, and some pieces have yet to be rewritten as I’m getting back up to speed with Rails and what the current landscape of gems and plugins looks like. But with each successive iteration it’s taken less effort to replicate what was previously built, attesting, I suppose, to the value of prototyping.
Some news about the application itself: as you can see in the teaser above, I’ve decided to launch a free beta, that will likely roll into grandfathered full accounts for free, even though I expect to use freemium pricing once beta testing is complete.
I should note that the above feature list will likely not be complete when the private beta begins, but I hope to have the complete launch feature set complete before entering a brief public beta. Well that’s the plan anyway….
by Andrew Gray (sweetandy@gmail.com) at October 27, 2009 05:22 AM
# /etc/apt/sources.list
deb http://www.backports.org/debian lenny-backports main contrib non-free
# apt-get update
# apt-get -t lenny-backports install wicd
by Andrew Gray (sweetandy@gmail.com) at October 26, 2009 01:54 AM
I feel it might be unfair to announce that I’m no longer using Habari without explaining why. There are several reasons, and I actually didn’t make the move away from Habari with the intent of solving them, either. It wasn’t a matter of ditching Habari for something better, but something different altogether.
Habari is a slick little blogging platform that has a lot of promise. I especially like their emphasis on the quality of Habari’s internals, which I think means it will do very well over the long term.
I am, however, confused by the community of developers surrounding the project. That is, unlike WordPress where people are churning out high quality themes and plugins that provide excellent new functionality, I see a few long-neglected, poorly supported extensions, and a handful of so-so themes.
Perhaps such a healthy community does exist and the project leaders just have a hard time of surfacing their work on the web. I don’t know. But I only found a few plugins that provided functionality I needed, one of them being the Twitter notifier that never, ever stopped telling me there was a newer release available (a notice prominently displayed on the admin dashboard) despite already being that installed release. And this was one of the plugins in the official repository.
Likewise, and again it must be compared against WordPress, which is from a consumer’s perspective the best-of-breed, Habari’s documentation is a tragic shambles. I’m sure in part that’s because this is a volunteer-driven project, but just covering the expected-working scenarios in a wiki doesn’t cover it. Thoughtful, cross-referenced, and well-indexed documentation that covers known problems, common mistakes, and what to do when things go wrong cannot remain embedded in your mailing list history or irc logs.
I think the future of Habari depends heavily on the maintainers’ ability to foster a community of user-contributors around the platform, and to do that you must have good documentation. Without those contributions, it’s not a platform worth switching to or starting with, despite its other merits.
What merits are those? I feel the project leaders have made some very smart, well-reasoned decisions regarding internals and core features (see the FAQ and coding standards for examples). Part of my initial interest in using Habari came from the possibility of contributing to the project, but I’ve found that as most of my day job time is spent on PHP, my after-hours time needs to be more diversified (as you can see elsewhere on this blog). It still looks like a very sharp project for someone hoping to get some FOSS experience to look into. And I may eventually come back to that.
So, what I can say in favor of Habari is that it appears to be a solid development effort, and if I were to summarize my complaints about it, I’d have to say that the secondary development efforts, by the add-on community, are lacking. I’ll leave it at that.
I’ve been running my personal blog on Habari for over a year now and I haven’t liked it for a moment. I’ve liked it little enough that mostly it’s been ignored during that time. Finally I decided I needed to switch, and months later I’ve finally settled on something new, Scanty, “The blog that’s almost nothing”.
I was obviously looking for something on the minimal side, I have a growing number of WordPress installations and I definitely didn’t need that kind of overhead for this blog. It occurred to me that Sinatra would be a great platform for an ultra-minimal blog, and on top of that idea I realized I was probably not the first person to think so.
Sure enough there are several entries on the Sinatra site. I took a look at a few, and initially tried to run with Marley but found it to have too many moving parts. Next up was Scanty, which appears to be perfect for my needs.
Scanty is ultra-tiny and as it’s set up on GitHub it’s meant to be forked so I can maintain my own version of the project. Not only does that sound like a good idea, but it’s working out quite well for me.
I took a low-brow approach and migrated my posts by hand (less than twenty of them) as the alternatives were to either create my own importer, or export from Habari into a fresh WordPress install, and then from WordPress to Scanty. I had a handful of comments as well, and while I wish I could preserve them I don’t even know if I want to have comments on this blog anymore. So the comments are just being dumped and while I’ve enabled Disqus, I’m really not in love with it and may just drop that before long.
We’ll see how it goes.
Because two people made quizzical remarks about using Twitter immediately after I cross-posted to Twitter and Facebook from Tweetdeck:
Twitter isn’t for everyone. I originally scoffed at the service and its name, and then I began using it to track conversations and buzz among tech luminaries. After adoption skyrocketed I found myself engaging in conversations with friends on Twitter as well, and that’s how I use it today.
Twitter has some distinct characteristics that magnetize certain people and repel others. It is a sort of hybrid between email and instant messaging, specifically the style that became popular with AOL instant messenger featuring rapid bursts of short text, versus the longer, more thoughtfully edited style we used in the early days of ICQ.
So, with its limited character-count, it features burst messages bereft of dressing and decoration, making people stick to the point and say what needs to be said. As a programmer, this is the most natural style of communication for me and I enjoy it greatly.
It’s a socially level medium. One person does not facilitate or mediate conversations, and anyone is free to join or leave a thread of conversation at any time.
Twitter does not abuse the term “friend”. You follow other Twitter users, and they can choose to follow you or not. There is no implied friendship and it is not compulsorily bi-directional. I can’t expect Jeff Atwood to accept my assertion of friendship on Facebook, but it’s no bother to anyone if I follow him on Twitter.
Those are a few of the big reasons why I feel Twitter is so agreeable with me. Others may have dissenting views on the same features or find other aspects of the service outweigh those they do find positive, but for me, it’s a fine thing and I’m happy it’s there.
Windows Server 2008 R2 currently requires the beta version of winpcap, and you need to run the installer in compatibility mode to install it.
This is more for my reference than anything else, though I hope anyone who is as stymied as I was when searching for phrases like “force fsck on reboot” and “force disk check next startup” benefit from this.
All you have to do is:
sudo touch /forcefsck
First, and most importantly, I updated the graphic on the placeholder site with something that wasn’t as (unintentionally) TweetDeck-like.
Second, as I mentioned in another post, I abandoned my first prototype of HYDRAtweet just over a week ago and began working it up again in the CodeIgniter PHP framework. There have been a few bumps in the road, having to do specifically with the needs of the application, but overall progress has been steady.
I’ve now got all of the prototype UI (and much more besides) in CodeIgniter, and all of the prototype’s database handling. At this point the only thing that hasn’t been hooked up completely (it’s partially implemented) is adding Twitter accounts to your HYDRAtweet account.
With that and a few other things done (password changer, password reminder, Twitter account revoking, etc.) it’ll be time to work on the post pulling and storage code. From there, well, I have some ideas.
Best I can tell, there is some functionality missing from traditional etherboot images that you can install on Ubuntu via the ‘kvm-pxe’ package that prevents a KVM guest from booting off of a Windows Deployment Services (WDS) server. Perhaps UNDI or such, it isn’t clear. The guest will accept an DHCP offer, but go no further. Occasionally looking for more offers, it obviously isn’t getting the traditional DHCP options it is looking for and I’m willing to bet that WDS doesn’t recognize it as a PXE client enough to speak up for itself.
Fortunately the etherboot project is alive and well under a massive rewrite coined gPXE, and there is much exciting development going on there, like iSCSI boot support. I tried creating different gPXE ROMs from rom-o-matic for the different NICs that KVM supports and replacing the roms placed in /usr/share/kvm by the kvm-pxe package, but didn’t get very far. I wasn’t sure if I had to match up the PCI id’s that KVM produced with the options on rom-o-matic or not. The e1000 ROM hit the “Too many option ROMS” error, apparently common when your PXE ROM exceeds about 60k. You can toggle different KVM NIC models via libvirt by using ‘virsh edit guestName’ and then adding a “<model type=’e1000′/>” line in the interface section of the guest definition. change e1000 to virtio, pcnet, whatever. There’s a list on the Ubuntu wiki KVM page that may provide you with further help choosing a functional gPXE ROM.
I ended up using the gPXE ISO (gPXE:all-drivers from the first pulldown) and booting off that by placing it in /var/lib/libvirt/images and adding it as an IDE CDROM storage device to the guest. Initially it gave up far two quickly when left to its own devices but I found that accessing the command prompt and running the ‘autoboot’ commant worked for me.
I hope to save some other poor saps from suffering the same ill experience I just went through trying to use SQLite3 with CodeIgniter 1.7.2. I’m going to contribute to the wiki page on this topic as well, but it’s a mess and my contribution may become lost in the noise.
So I’ve made the move from laboriously creating a custom codebase for HYDRAtweet (backstory) to implementing it on top of CodeIgniter. I’d already started with SQLite3 via PDO and intended to just carry on using it by defining the models based on the existing schemata.
This didn’t go so well because, as reported in the wiki, SQLite3 in PHP requires the use of PDO, and since CodeIgniter is backwards-compatible with PHP 4.x, in which PDO is not bundled (though it can be added) the provided SQLite support in CodeIgniter makes use of the sqlite_* functions in PHP, which again cannot be used with SQLite3.
The wiki described altering/replacing CodeIgniter’s SQLite driver to use PDO instead of the sqlite_* function calls, but it was unclear if that would break support for SQLite 2, and I found that a bit shabby all-around, so I made a copy of the SQLite driver directory (system/database/drivers/sqlite), naming the new copy sqlite3 and adjusting the names of all the subclasses defined within to include the “3”.
Then I was further confounded, receiving the following message:
The resolution turned out to be quite simple. Never having used SQLite (of any vintage) without PDO, I did not realize the connection string would be different. Therefore my database.php file had to be modified from:
$db['default']['database'] = APPPATH.'db/hydratweet.sqlite3';
to
$db['default']['database'] = 'sqlite:'.APPPATH.'db/hydratweet.sqlite3';
…which was just the inclusion of the “sqlite:” prefix.
And now all is well.
Incidentally, the sqlite_driver.php file contains a method stub called _list_columns() and returns false, with a comment stating that it’s not supported. I immediately said to myself, “that can’t be right!” …but a few minutes later after I’d unsuccessfully tried using “pragma table_info(tablename)” in nested queries and done a little Google-walking, I realized it was true and had to move on. It’s always sad to find your pet technologies have awkward drawbacks.
Regardless, the driver bit works just fine, and if it’ll save anyone some time, here’s the zip. download
Some photos from the Install Fest 2009. It really was fun and we had a number of library patrons drop in to chat and get free software! We also installed Linux on half-a-dozen different systems which were given away at the end of the event.
by mark@foster.cc (delimiter) at September 20, 2009 05:05 PM
Some time after my last post, I completed setting up my technology feed aggregation pipe. Here’s what it looks like:
The feeds (those four blocks in the top-left) are, clockwise from the top & left-most, general programming, PHP, operating systems and general technology, and Ruby. They each feed into a union block, which is then filtered by a set of keywords against each item’s title, and finally they’re checked for duplicate titles or URLs.
The result isn’t pretty.
Two things are immediately obvious when you look at the screenshot above. You can’t tell where anything originated, and, as if to mock me, there are two items with the same exact title (Five Best Virtual-Desktop Managers).
If you read my first post about combining Google Reader & Yahoo Pipes, you’ll know I was concerned about Google’s feed trawling scheduler taking very light notice of my feed because I’m the only one using it. This seems to be the case, as new items come in at a trickle, and I’m almost certainly never going to see some items that should have made it through.
When all is said and done, there’s no way around it. Without Google Reader supporting these features directly, there’s no solution. So as I suggested before, this means I’ll have to do it myself. And that means it’s on again for Sluice.
I’ll probably write up the exact details of implementation later on How I Use Things but I wanted to express my feelings about and behind this move and something so meta as that belongs here instead of there.
I’ve been using Google Reader for several years now, finding it an unparalleled application for managing feeds. However, I think it is seriously flawed in several ways and needs a lot of attention to make it useful in the long term.
The common response to a statement like that, in the open source world, is “OK then make it yourself.” I found that to be the appropriate reaction here and for the first Rails Rumble (original 2007 site is now completely gone, so you get a Wikipedia link instead) along with co-worker and friend, Chris Martin.
I named the new app Sluice and had some great features in mind. One of which was search for past feeds. Another was adding a social aspect to feeds and feed items that would allow you to mark something as publicly liked, among other things. There were some others, but those two things will have any Reader users right now saying “oh but they already do that”, which is true. Sharing just came out, almost two years later. And search was introduced two days before the competition.
On the one hand that’s a big blow, and indeed may have had a lot to do with why the project has been on indefinite hold since the contest ended (we had a very basic, but working feed reader with a pretty decent interface by the time the weekend was up). On the other hand it was very encouraging for a user of Reader. It told me that smart people were actively working on the application to improve it, and they were likely listening to other users who saw the same flaws and unmet needs that I saw.
Unfortunately that evolution has not continued. The sharing features that I had not planned to implement until well after more important needs were addressed are now in play with Reader, but do we have any way to filter feeds based on keywords? Prevent duplicate entries? See links to possibly related entries? It seems to me that they’ve only plucked the low-hanging fruit (they are a search company after all, how hard could adding that have been?) and are not truly putting the effort into evolving the core solution of Reader: consumption and management of feeds.
After once again surpassing the 1000 unread entries mark, I lost my patience with Reader and moved forward with the unholy union that I’ve been dreading since before conceiving of Sluice. I’ve wed Google Reader with Yahoo Pipes.
This is nothing other than an act of desperation. I’m a practitioner of Inbox Zero and having all of my mail processed, organized, and dealt with is an unbelievable relief. The same is true of feeds. The more unprocessed items I have in Reader, the more daunting, heavy, and unworkable it is. Time-sensitive items are lost in the flood of news about Ballmer’s latest clumsy attempts at edging around Linux in the marketplace, or everyone and their grandmother’s opinion of Snow Leopard. Valuable insights into topics relevant to my work and studies are plowed under by countless redundant 5/10/15/30 top/best/most awesome tips/tools/sites/snippets/tutorials for newbies/designers/developers/people with nothing better to do.
So I finally did it. Or I’m in the middle of doing it, anyway. As I process down all of the unread items in each feed, I’m moving it to Yahoo Pipes, where it is being filtered by keyword (“windows”,”erlang”,”nettuts”, etc.), united with the rest, passing through a deduplicator based on title and/or link, and finally being spit out in one big feed.
The downsides to this are so steep that it’s taken me over two years to execute on it:
Feed addition. When I come upon a new feed I want to track, I have to decide whether it’ll require filtering and deduplication. If so, it has to be manually copied into Pipes, rather than the two-click addition to Reader.
OPML Who? I can no longer generate a list of the feeds I track in Reader.
The tinyurl dilemma. Because of the obfuscation layer, I can’t participate in the social features added to Reader.
Gardening. I will essentially have to abandon the idea of dropping old feeds that are no longer positively contributing, unless I do outside research to determine what the feed is injecting into my Pipes rig.
Reader machinery. I’m uncertain how this will turn out, but I fear that because I will be the only one pulling this feed, the Reader-feeding machinery will not poll my Pipe feed often for updates.
Once upon a time it didn’t seem like a big deal to just click any article I want out of my way to mark it read, and move on to the stuff I wanted to read. But as I’ve added and removed feeds over time, I’ve found a core of them that are indispensable, however they take so much effort to maintain that no time/interest is left over for actually reading the items that do have value to me.
Optimally, the Reader team will surprise me with some features that handle the keyword and duplication filtering that will make this manageable for me, causing me to do the whole process over again, in reverse. I’ll be happy to do it when that day comes. Until then, hard compromise.
kvm-img convert small.img small.raw # this is your old image
kvm-img create large.raw 15G # or whatever size
losetup /dev/loop0 small.raw
losetup /dev/loop1 large.raw
dd if=/dev/loop0 of=/dev/loop1
losetup -d /dev/loop0
losetup -d /dev/loop1
kvm-img convert large.raw large.qcow2
Start KVM up again with the new large image. Go to the ‘Computer Management’ mmc applet under ‘Administrative Tools’. Choose ‘Disk Management’ under ‘Storage’. Right click your existing volume and choose extend. Step through the wizard. I got a message that made it appear it didn’t work, but I guess I didn’t read the fine print right and it did work fine enough. There’s always the old small image file if you run into problems. Speaking of which, be careful to not be dyslexic when inputting dd options.
Adam Monsen
Andrew Kvalheim
Bryan McLellan
Dave Cohoe
Eric Butler
Ian Gallagher
Jesse Keating
Mark Foster
Trevor Bramble
Last updated:
September 02, 2010 08:00 PM
All times are UTC.
Powered by:
