Topic: Sysadmin

The other night I was on a panel discussion on Python Deployment at the Django-NYC meetup. The discussion was very good and I think everyone there got a lot out of it (I did). But the format wasn’t really suitable for going into great detail, showing actual code fragments or demos. I’ve written about aspects of our deployment strategy in the past on my personal blog and in comments on other sites, but it’s a continuing work in progress and I’m well overdue for an update. We also have a web-base deployment tool that, while the code has been open and available for a while, we really haven’t officially announced or publicized until now.

There are two angles to Python/Django deployment that I want to discuss. First, there’s deploying Django apps. Then there’s how we use a Django app to deploy Django (and other) apps. They are closely intertwined so I think I can’t really talk about one without talking about the other.

Nothing like having to spend time in a noisy, cold, cramped server room doing upgrades on hardware to make you appreciate the virtues of the cloud.

Here at CCNMTL, we have a pretty diversified setup when it comes to servers and hosting. CUIT provides a lot of resources for us: a very solid network infrastructure, DNS, email, file serving, databases, course management systems, centralized authentication services, backups, and Java and LAMP app servers. CUIT is tasked with serving the entire University though so they tend to be a bit more conservative and in their deployments than an organization like ours, whose mission involves keeping up with the cutting edge, sometimes needs. So we run a few of our own servers, using Debian, Ubuntu, and a Xen Hypervisor setup. This lets us run Django, Plone, TurboGears, PostgreSQL, and experiment freely with anything else that comes along that we see potential value in, or even just make sure our applications run with newer versions of software than what CUIT has deployed.

I subscribe to a lot of programming oriented newsfeeds so I seem to have a steady stream of links coming in that I find interesting and like to share with the rest of the dev team. I usually post the links in our IRC channel as a nice way of informally spreading them around to the other developers who might be interested in them. It's nice and simple. Just copy and paste the URL into my IRC client, maybe followup with a line explaining what the link is, and then the other developers will see it and can check it out and we can discuss it.

For the use case of "share a link with a small community and then discuss", I think it's much easier than sending an email, and really even easier than twitter (plus no obnoxious URL shorteners obfuscating things).

The downside is that not everyone's in the IRC channel all the time. We have a bot that logs the channel, but I don't think many of us rigorously check the IRC logs every time we've been out of the channel for a bit. So sometimes I have a link sitting in a tab in my browser that I want to share with the group but I know that one of the developers who would be particularly interested in it happens to be offline at the moment. So I frequently end up accumulating tabs in my browser until there's sort of a quorum of developers in the channel. That feels like a step back. I might as well be sending a message to a mailing list. What is this, 2003?

This hit me particularly acutely last Friday when it happened that about half the dev team was out that day, and I had some particularly juicy links I wanted to share. I started pondering the problem and came to the conclusion that what I wanted was for the links I (and anyone else) post in the IRC channel to be automatically collected and aggregated into an RSS feed that we could all then subscribe to. So anyone who misses a link in the IRC channel would have a good chance of noticing it come up in their newsreader of choice when they come back.

The most sensible way to accomplish that was probably an IRC bot that would sit in the channel, look for URLs and then re-post them to a Delicious account created specifically for that purpose.

Sky, the reigning IRC bot expert (having set up our ops-keeping and logging bots) was out that day so I couldn't ask him about it. Ethan, one of the few developers in that day, suggested checking out phenny, which is a relatively simple IRC bot written in Python.

Phenny seemed to be just what I needed. I got a phenny-bot running in a couple minutes and started to poke at it to see how it worked. It's built on top of Python's asyncore and async_chat libraries, which are not the most intuitive libraries ever, but are efficient and reliable, and it appeared that Phenny had actually done a good job of making a much more intuitive API accessible. So I could add functionality to Phenny without having to really deal with any of the asyncore madness which surely would've turned my brain into mush on a Friday afternoon.

Even I was impressed when, about ten minutes and an 'import pydelicious' later, I had my own IRC bot that would post URLs up to a delicious account. All I had to write:

from pydelicious import DeliciousAPI
from datetime import datetime

def saveurl(phenny, input):
    """ logs a url so others can check it out"""
    parts = input.split(" ")
    url = parts[1].strip()
    title = " ".join(parts[2:]).strip()
    # the simplest of sanity checks
    if not url.startswith("http://") or not title:
        return

    poster = input.nick
    now = datetime.now()
    comment_url = "http://quimby.ccnmtl.columbia.edu/ircbot/web/" \
    "?y=%04d&m=%02d&d=" \
    "%02d#%04d%02d%02d%02d%02d%02d" % (now.year,now.month,now.day,
                                        now.year,now.month,now.day,
                                        now.hour,now.minute,now.second)

    a = DeliciousAPI('phennyccnmtl','nottherealpassword')
    a.posts_add(url,title,"posted by %s: %s" % (poster,comment_url))

saveurl.commands = ["url"]
saveurl.example = ".url http://www.example.com/ Title For Link"

That gets dropped into phenny's 'modules' directory and it's done. When a user in the channel types ".url http://www.example.com/ Title For Link", Phenny will notice the pattern and post that link up to its Delicious account.

Now I feel like I have a bit more of an idea of why the zombie botnets all use hidden IRC channels as their control mechanism. If you're comfortable learning a couple basic commands, you can easily control a whole army of these bots. IRC's built-in distributed, scalable, relatively reliable architecture is proven and tested and you get it pretty much for free when you create a channel.

Phenny gets the job done, but it's far from perfect. To add a new command to Phenny, you have to put it into the 'modules' directory in Phenny's source directory. There's no real API for registering plugins beyond that. It literally does an opendir('modules') and tries to import each .py file it finds there. This means that to add a custom command, you have to fork Phenny. If there's ever a security update to Phenny or something, we're on our own to merge it in.

I'll live with it though. I have a fondness for simple tools that just get the job done without making things more complicated than they need to be.