A Matrix Sim in JavaScript

Do you ever find yourself working on a large project with a long lead-in time before you have visible results to show? I had been frustrated by this feeling recently. I had been focusing on a personal project which will likely be a long time in the making, so I decided to pick up a short, simple, and quick project to keep things lively and fresh. So our story begins.

During high school, I was wowed by the movie The Matrix. The special effects were mind blowing (for their day) and the story line was something I could really get behind (not so with the sequels, but I digress). I imagine this is how my elders felt when Star Wars was first released.

One of the most memorable visuals in the movie is the code of the matrix zipping by vertically on the screen in a retro green on black. Cypher, makes a statement, which seems to be a favorite among programmers, that he no longer sees the code itself, but what it represents. He has become fluent to the point of transcendence - a geek nirvana experience.

At this point in my life I had written a few moderately complex programs, mostly using Turbo C, and I thought: "Why, I could make my computer look like the matrix." It was one of the more involved programs I had written at the time and I was quite pleased with the result. I brought it in to the computer lab at school, loaded it onto all of the machines, and had the whole lab running it at once. I'm pretty sure that Ben, Yed, and Roman G. remember it. Sadly I doubt the code would run on the DOS emulator that ships with Windows XP and higher. I recall vaguely that Roman ported it to Win32 or something in our high school computer science class. Now that I think about it, I guess you could say that this was my first open source experience.

In thinking of something quick and easy to create with some immediate visual payoff, I decided to recreate my little DOS matrix sim, but this time in the browser. Behold: The matrix in JavaScript.

It turns out that the CSS rules needed to keep columns at a fixed width and height was actually quite complex and took longer than expected to get right. The vertical line height was still elusive in Chrome and Safari, but worked in Firefox and Opera. I don't think that this works yet in Internet Explorer. Though if past traffic is any indication, none of you would have noticed had I not said something :-)

Many thanks to Scudmissile for his mighty CSS kung fu. Oh, and that whirring sound that you hear may be your CPU fan trying to take flight. Apparently rendering a vertical text flow like this using DOM manipulation, CSS, and Math.random may be a bit more processor intensive than my obsolete C code.

A Test Client for App Engine

I created a simple utility library in Python to help anyone debug their App Engine application. Many of the App Engine apps that I've seen use HTTP, HTML form posts, and the Users API and the easiest way to test these features is to fire up the web browser and click through the web pages generated by the app. However this can be a bit slow and repetative and it is difficult to determine exactly what is being sent over the wire (though this is greatly helped by using wireshark, fiddler, tcpdump, or antoher network packet sniffing tool).

Enter my little App Engine HTTP module. It provides a simple interface for making arbitrary HTTP requests and will print the full request and response to the terminal (though you can turn the noisy printing off if you want). Download, copy the http.py file to you working directory and try it out in your Python interpreter.

For our first demonstration, let's try to visit the Google search page.

import http
client = http.Client()
resp = client.request('GET', 'http://www.google.com')

You should see your request and the server's response (with the HTML for the Google Search page) in your terminal window. This should work with just about any website out there.

Other HTTP debugging tools can show you the request and response like this, but I find that this kind of simple Python client can be useful in writing end-to-end or integration tests which contact your App Engine app remotely.

Along those lines, one of the things which standard HTTP debugging tools do not provide, is a way to sign in to an App Engine app with a Google Account so that the App Engine Users API can identify the current user. I wrote an extremely simple app which illustrates the Users API, try it out here:

http://jscudtest.appspot.com/user

After signing in, the page should simply say, "Hello yourusername (yourusername@yourdomain.com)" You'll notice that during the sign in process, you signed in on www.google.com/accounts and were asked to approve access to the app. This kind of interaction works great in a browser, but can be tricky when you are using a command line, browserless, client.

It is possible however, to sign in to an App Engine app without using a browser. You can use the same technique used in appcfg, use ClientLogin and use the authorization token to obtain an app specific cookie which indicates the current user. This simple HTTP library can do this for you and all subsequent requests will use this cookie to tell the App Engine app who the current user is. Try it out by making the request to the simple user app that you visited earlier:

import http
client = http.Client()
client.appengine_login('jscudtest')
resp = client.request('GET',
                      'http://jscudtest.appspot.com/user')
print resp.body

You should see the following text displayed in the terminal:

Hello, yourusername (yourusername@yourdomain.com)

You can use the appengine_login method with your own app, just change the argument to the App ID of the app you want to access.

Along with simplifying access to apps which use Google Accounts, I wanted this library to simplify the process of using another feature used by many web apps: HTML form posts. Now I'm certain you've used HTML forms before, here's a simple example:

http://shoutout.appspot.com/

The above app uses both the Users API and a simple form. As an alternative to visiting this page in the web browser, you can post your shout-out using the following:

import http
client = http.Client()
client.appengine_login('shoutout')
client.request('POST', 'http://shoutout.appspot.com/',
    form_data={'who': raw_input('From: '),
               'message': raw_input('Message: ')})

If you've even wondered what gets sent across the wire to post on a form like this, look back in your terminal to see the request from your computer and the response from the server (this is of course just the HTTP layer, wireshark will show you traffic on the IP and Ethernet layer as well).

That's really all there is to it. I designed this as just a simple script to use on the command line and I wrote it in less time than it's taken me to write this blog post about it (I borrowed atom.http_core from the gdata-python-client as a foundation). With some tweaks to remove the interactive (getpass and raw_input) calls and replace them with parameters, I could see this module as a utility layer in a larger, more complex, App Engine client application. If you're creating on I'd love to hear about it ;-)

For more information on how the appengine_login method works behind the scenes, see this presentation I gave a few months ago:



Many thanks to Trevor Johns and Nick Johnson for helping me to understand how this ClientLogin-to-cookie exchange works.

I'm sure that App Engine's Java runtime users would appreciate a port of this simple library to Java, if you feel so inclined.

New version of my Dirt Simple CMS

I have just uploaded "version 2" of the dirt simple content management system which I developed almost a year ago. For those who don't recall, scud-cms is an extremely thin layer on top of App Engine with a plain text box based editor for creating web pages. The only new feature in this release is the ability to page through all of the content in your app by visiting the /content_lister page. I implemented the pagination system using "key only" queries and order-by-key which are fairly recent features in App Engine. For more potential designs to page through datastore entities, see this article on pagination.

I was inpired to work on this in part by a comment from Jason Worley (swirleydude on twitter) who has been using it and appears to like it quite a bit. At some point I plan to do a version 3 release which will include file uploads (great for images) just as Jason has done in his own modifications. Having someone use your software, enjoy it, tweak it, and build on it is rewarding in a way which is quite unique.

Updating my OLPC XO laptop

For a few months my XO laptop was my primary personal machine. I've written before about some of my favorite features, and after a year and a half with it, my favorites are still:

• A screen that is readable in full sunlight or glare
• Linux
• Long battery life
• Runs absolutely silently, no fans or spinning hard drive

However, there were a few drawbacks. It is a bit short on ram and processing power but most of the time I'm using simple programming tools, so I don't need much. Web browsing on some sites was a bit slow though. The biggest pain point in my book was the lack of copy and paste capability. I often found myself needing to retype a lengthy command in the terminal. On other systems I would have just copied and pasted from the web browser. For a while, I worked around this by using screen, which has a built in copy-paste buffer.

It turns out, in a recent release, the OLPC team has assuaged my number one annoyance. It is now possible to copy text from any of the activities (programs) and paste into another (like the terminal). In order to use this new goodness, I needed to update my XO operating system from the current version (656) to a newer build (802).

The process of updating turned out to be quite simple. If you are undertaking these same steps, please note, this does wipe out some parts of the operating system. New programs which you've installed using yum or rpm's might be erased depending on where they are installed. It looks like everything in the /home/olpc directory is untouched. For complete instructions, see the laptop.org wiki page for olpc-update.

To update, open the terminal and become root by executing

su

After that, make sure that your laptop is plugged in to a source of electricity and connected to the internet as this next step involved downloading the new operating system and overwriting your existing OS. Probably not something you want to lose power during.

To download and install the new OS run

/usr/sbin/olpc-update 802

Once you run this your laptop will spend the next several minutes to hours downloading and installing. A few exciting status messages will appear on your screen. Things like "Deleting old pristine version" - yikes!

I've found 802 to be a significant improvement over 656. The copy paste functionality is great and the UI looks and feels cleaner. Selecting applications and switching between them is easier than it used to be. With copy and paste now firmly in place I plan to use my little green machine quite a bit more.

Partial Function Invocation

My wife tells me that I often jump into an explanation by starting at the beginning of my train of thought without giving any indication of where I'm going. It would be better if I began with the point I'm trying to make, then explain how I reched my conclusion, How am I doing so far? Oh wait, right... Here is my conclusion:

Allowing a function to be partially invoked, to allow some of the arguments to be specified at different times, can allow for code which is more flexible than by just using objects or pure functions.

I've been thinking about this lately as I refactored the gdata-python-client which is a library which can be used with AtomPub services. I'll spare you all the gory details, and offer a simple example of how partial function invocation might come in handy.

When making a request to a remote server, you might need the following information, just for example: username, password. URL, message body, and content type. So we start out by writing a stateless function to take this information and open a connnecion to the server, format our inputs, transmit our request, and parse the response. We'll call it post, and using it looks like this:

serverResponse = post(url, data, contentType, username, 
                      password)

This is all well and good, but suppose the final request, as shown above, is preceeded by a whole series of function calls. Each function would need to dutifully pass along parts of the request. Say for example that the user types in their username and password long before the request is made, so these get passed as parameters to lots of functions which only receive them so they can pass them on. In addition, the username and password are almost always the same from request to request, so the same values are being passed to the post function over and over. In cases like this, we will often use an object to hold common values.

class Requestor {
  username
  password

  method post(url, data, conteentType) {...}
}

Now our request will look like:

client = new Requestor(username, password)
serverResponse = client.post(url, data, contentType)

What we've effectively done here is specified some of the information in advance and left other pieces of information to be specified at the last minute. I would argue that this make the code better (cleaner, less chance of human error in listing lots of parameters, perhaps less data on the call stack, etc.).

Now the question becomes: Did we extract the right pieces of information from the function call into the object? Suppose the code you are writing needs to use a different password for each service you are making a request to, but the content type of the data is always the same. Then it would have made more sense to design our class like this:

class Requestor {
  username
  contentType

  method post(url, data, password) {...}
}

Since we've established that not everyone who is using our code has the same usage patterns, lets design for utimate flexibility. Every parameter can be specified either in the object, or in the function call. Also, if the object has a parameter already, we could override it by passing in that parameter when we call the method. This is not too difficult in in Python, so here is a non-pseudocode example:

class Requestor(object):
  
  def __init__(self, url=None, data=None, 
               content_type=None, username=None, 
               password=None):
    self.url = url
    self.data = data
    self.content_type =  content_type
    self.username = username
    self.password = password
  
  def post(self, url=None, data=None, 
           content_type=None, username=None, 
           password=None):
    url = url or self.url
    data = data or self.data
    content_type = content_type or self.content_type
    username = username or self.username
    password = password or self.password
    # Now we have our inputs, code to make 
    # the request starts here
    ...

If you think this seems a bit excessive, I would agree. I didn't go nearly this far when designing the library that started me thinking about this. There was one request parameter in particular though that does use this pattern. (Five points to the first person to post it in the comments. ;-)

To use the above class, you would do:

requestor = Requestor(username='...')
requestor.password = '...'
...
server_response = requestor.post(url, data, content_type)

It will also handle our alternate usage where we want to give the password to the post method and set the content type at the object level:

requestor = Requestor(username='...', content_type='...')
...
server_response = requestor.post(url, data, password='...')

We can even override parameters which are set in the object:

requestor = Requestor(password='...', content_type='...')
requestor.username = '...'
...
# Override the content_type, just on this request.
server_response = requestor.post(url, data, content_type='...')

With the above example we end up with a lot of code just to let us specify each parameter in either the object or as a function argument. In fact, this can introduce so cases where the user forgets to specify in either, which is possible because all function arguments are now optional. Wouldn't it be better if we could instead specify some of the function parameters, pass the half-specified function call around, and fill in the ramaining values when we finally invoke. For this illustration, I'm using the following syntax to show a partial invocation, < > around arguments instead of ( ).

function post(url, data, contentType, username, password) {...}

started = post<username, password>
...
serverRespense = started(url, data, contentType)

Recall our case from earler, what if the contentType is constant but the password is instead more variable:

started = post<username, contentType>
...
serverRespense = started(url, data, password)

It turns out I'm not the first person to think of this pattern, not by a long shot. Functional programming often makes use of this pattern, referred to as function currying. I found the following example for Scheme which also shows how easy this is in Haskell. The prototype library for JavaScript includes a bind function which can accomplish the same thing. Here's a paper on the topic in C++: (pdf, Google cache HTML). I also found PEP 309 which was a proposal for this in Python. Perhaps I should have called my Python example above: Function Currying using Classes. If you can think of other examples, I'd love to see them.

Recently completed: our half-US tour

With stops in

Auburn CA
Reno NV
Winnemucca NV
Wells NV
Metropolis NV
Bonneville Salt Flats UT
Salt Lake City UT
Green River WY
Cheyenne WY
Denver CO
Littleton CO
Centennial CO
Lincoln NE
Omaha NE
West Des Moines IA
Indianola IA
Sigourney IA
Fairfield IA
Hannibal MO
Fenton MO
Crestwood MO
Sikeston MO
Memphis TN
Little Rock AR
Texarkana AR
Texarkana TX
Plano TX
Dallas TX
Garland TX
Wichita Falls TX
Amarillo TX
Albuquerque NM
Williams AZ
Bakersfield CA
Plesanton CA
Milpitas CA

(Edited) Here's a map of our approximate route:

View Larger Map

Anticipating "it's all crazy..."

One of my favorite bands is releasing a new album tomorrow and I though I'd take a short moment to gush about it. I've been a long time admirer of mewithoutYou's work. Their abstract poetry filled with vivid imagery combines with highly energetic rich sounds and hits a sweet-spot. Certain lines tend to resonate with me. I find "O Porcupine" in particular to be moving.

I like to think of myself as having broad musical tastes, but if I do tend to be critical of a song it is very often overly simplistic lyrics which I fault. I don't know that I've ever sat down and thought through why that is, perhaps another time.

From listening to a few preview release tracks from this album, I don't think mewithoutYou runs any risk of that sort of problem. In fact, while listening to "the Fox, the Crow and the Cookie" I found myself pulling out the dictionary to look up the words canzonette, corvidae, and escadrille! Some of those rhymes are absolutely majestic. I think this album may satsfy even Xirzec's bombastically erudite lexical tastes. Thanks again to Andy for introducing me to their music.

A Two Pun Day

If I were a rock, I wouldn't want to be marble because I wouldn't want people to take me for granite.

I told one of my good friends that I was allergic to milk and he said, "No whey!"

I would estimate that I make at least two puns per week. Something someone says will just strike me at an odd angle and the pun percolates out effortlessly. The other day I thought of the above two. I'm not sure if the first is original, but we were in an elevator debating whether the stone walls were made of marble or granite and I couldn't resist. Later that day, someone mentioned cottage cheese and I thought of the second. It's a certified groaner, five out of six friends agree!

Our New Pi Day Laptop

To commemorate this year's pi day, we've christened our newly purchased laptop PiPuter. It's an Acer Aspire. We were looking for a super cheap full sized laptop with decent spects and found a good deal at the MicroCenter. Vanessa says that our computer has graduated magniu cum laude from its factory and it is currently watching us - watching, watching, always watching.

Septoplasty

I never dreamed that I too would be undergoing septoplasty just a few short months after Joe Greogorio had his operation. It turns out I had a pretty severely deviated septum which likely contributed to the annual sinus infections which I have been experiencing. My allergist recommended I have this operation to straighten the inside of my nose, as my left nasal passage was almost completely blocked near the bridge of my nose. In looking for a possible sinus infection I had a CT scan of my sinuses which revealed just how severe the deviation was. By the way, looking at a slice by slice cross section of your own head is fascinating.

This past Wednesday I had the surgery and I will have the stitches and splints removed next Tuesday. It's been a bit different than I expected. I anticipated some pain but, surprisingly, the part that hurts most is my upper front teeth. I also didn't expect quite so much bleeding. In any case, I'm not feeling too bad and I am looking forward to the long term benefits. Also, this has been my first experience BWOV (blogging while on vicodin).

Two Guys Arguing - Five Questions

I noticed that my friend Ben is participating in a new blog named Two Guys Arguing (in addition to blogs one two and three :-) and I thought I'd piggy back on a recent ice-breaker post.

1. What are you currently hacking on?
2. What are you currently getting better at?
3. What do you do when your computer is asleep?
4. Describe that ‘big fish’ project that’s been stewing in your brain.
5. What are you gonna post about this coming week?

On to the answers:

1. As a small part of my day job I'm currently working on a rewrite of significant portions of the gdata-python-client to support version two of the Google Data API protocol (check out the v2 branch).
2. I've been trying to focus on writing speedier unit tests. Unit testing is great, but sometimes it is necessary to write end to end tests which can take a rather long time to run. Slow tests are run less often, so I've been thinking of some ways to cache some of the more expensive pieces of these end to end tests while still preserving their utility.
3. Playing with Claire is easily one of my favorite non-computer related activities. I have a wonderful wife who keeps me company and there's nothing like relaxing with the family. Aside from the yard work, remodeling, and other house related items, I do very much enjoy playing guitar now and again. I even have a somewhat regular weekly gig.
4. I have a few side projects I like to hack on here and there in my spare time. One of them I've been thinking more about recently is a computer language I've decided to call "Headspace". I wanted to try an experiment and see what the effects would be of applying the rule of seven to programming.
5. I've been planning to write a bit about the surgery I've recently undergone so stay tuned ;-)

Two Guys Arguing seems like it's off to a great start and I look forward to reading along.

Google Tweets

Earlier today, Google posted the following on Twitter

I'm 01100110 01100101 01100101 01101100 01101001 01101110 01100111 00100000 01101100 01110101 01100011 01101011 01111001 00001010

This looks suspiciously like ASCII, so I set out to render it in a more human readable form:

Bin       Hex ASCII
0110 0110 66  f
0110 0101 65  e
0110 0101 65  e
0110 1100 6C  l
0110 1001 69  i
0110 1110 6E  n
0110 0111 67  g
0010 0000 20    (space)
0110 1100 6C  l
0111 0101 75  u
0110 0011 63  c
0110 1011 6B  k
0111 1001 79  y
0000 1010 0A  \n

I doubt I'm the first to post this, but there you have it :-)

Brief Haitus

I'm taking a short break from posting here to study for the GRE. I'm planning to apply to grad school in the not too distant future, and figured I should study. Apologies for the pusillanimously prosaic post.

Guitar Hacking

In highschool, I began learning to play guitar. I took private lessons at a local music store, and although I had read sheet music for over seven years by this point (mostly while playing trombone), my teacher started me out by drawing guitar tabs. Unlike sheet music with staffs, clefs, meters, and measures, guitar tabs read like a simple map of where your fingers belong. In some cases, tabs are a picture of the neck of the guitar. For example, most books of guitar chords contain diagrams that look something like this:

 E A D G B e
0___________
 | | | | | |
 | | | | | |
 | | | | | |
1-----------
 | | | | | |
 | 2 | | | |
 | | | | | |
2-----------
 | | | | | |
 1 | | | | 4
 | | | | | |
3-----------
 | | | | | |
 | | | | | |
 | | | | | |
4-----------

The long vertical lines are the strings, while the horizontal lines are the frets. The lower pitched strings are on the left, while the higher pitched are on the right. Finger placement is indicated by the numbers written in the strings. The index finger is 1 and the pinky is 4. This diagram is of a G major chord and it uses three fingers, two on the third fret, and one on the second.

However, you could also draw a guitar tab like this:

 0       1       2       3       4
e|-------|-------|---4---|-------|
B|-------|-------|-------|-------|
G|-------|-------|-------|-------|
D|-------|-------|-------|-------|
A|-------|---2---|-------|-------|
E|-------|-------|---1---|-------|

The above looks a bit more like the neck of the guitar from the point of view of the person holding it.

One great way to learn how a musician plays an unusual chord, is to look at a video or picture. In that case, you'd see the neck of the guitar reversed, like this:

 
4       3       2       1       0
|-------|---1---|-------|-------|E
|-------|-------|---2---|-------|A
|-------|-------|-------|-------|D
|-------|-------|-------|-------|G
|-------|-------|-------|-------|B
|-------|---4---|-------|-------|e

Unless, of course, you are watching one of the many left-handed guitar players (Hendrix, McCartney, etc.) in which case, you neck would look like this:

 0       1       2       3       4
E|-------|-------|---1---|-------|
A|-------|---2---|-------|-------|
D|-------|-------|-------|-------|
G|-------|-------|-------|-------|
B|-------|-------|-------|-------|
e|-------|-------|---4---|-------|

The interesting thing about learning to play guitar using tabs, is that you actually have less information to go on than with sheet music. Tabs tend to lack information about the rhythm being played, tempo, and volume which are all present in sheet music. The one thing which sheet music lacks, however, is an indication of where your fingers should go when playing a particular chord or riff. Unpacking finger placement information from a cluster of notes on a staff can be difficult enough that guitar tabs make an attractive tradeoff. It is the musical equivalent of a domain specific language.

With the prevalence of recorded music which can be rewound and replayed over and over, a guitar player can often reconstruct the rythms and other necessary information by listening to the song. No longer need all information live on the page, the quick and dirty guitar hacker plays with tabs on the stand and the music in her head.

Mashup Camp 8

I participated in MashupCamp8 and created a simple mashup in a few hours which I have now open-sourced. It is a blog post editor, or article editor if you prefer, which finds key words in your text as you are writing and displays web search results for those key words next to the editor. Here's my elevator pitch:

When writing an article or blog post, I'll often note places where I should add a link and go back later to add them in. Sometimes searching for good resources to link to uncovers new information which I wish I had known while I was writing the article.

As I was walking around here at Mashup Camp, I saw a few web services which would allow me to do some data mining on raw text (like this blog post I'm writing) and search the web for relevant information.

I created a simple Ajax editor, which is hosted on Google App Engine. It takes the article's text, sends it securely (over HTTPS) to Open Calais which extracts key words then disambiguates, dedupes, and scores them for relevancy. Using this semantic web data from Calais, the editor then performs a web search on each of the keywords and requests results from the Yahoo Search BOSS API in the JSON format. The text analysis and web search is done automatically when the editor detects that you have stopped typing. To avoid excessive and distracting search refreshes, the editor will wait between three and eight seconds before performing the search.

You can try it out here:
https://sippycode.appspot.com/article_editor.

Download the source code here.

This simple little mashup was well received, and I think part of the reason is the potential for growth. The fundamental idea, is to have an environment in which relevant data is brought to you as you work. You don't even need to go info hunting, our computers can find relevant information and bring it to you automagically.

For this particular mashup app, I had envisioned a few additional features, the addition of which is left as an exercise to the reader if you should so choose.

• A WSIWYG editor to create rich HTML instead of just plain text. (I looked at tinyMCE but ran out of time.)
• Search on combinations of keywords instead of just individual items identified by Calais.
• Search other information sources, like news results, images, videos.
• Search on multiple search engines. (Google has an easy to use Ajax search API, but I wanted to try something that was totally new to me.)

XML Library with Versioning

I recently created a simple Python library for converting objects to and from XML. Code samples up front, here's how you would define some class to represent some hierarchical XML:

class AtomFeed(XmlElement):
  _qname = '{http://www.w3.org/2005/Atom}feed'
  title = Title
  entries = [Entry]

class Entry(XmlElement):
  _qname = '{http://www.w3.org/2005/Atom}entry'
  links = [Link]
  title = Title
  content = Content

class Link(XmlElement):
  _qname = '{http://www.w3.org/2005/Atom}link'
  rel = 'rel'
  address = 'href'

class Url(XmlElement):
  _qname = '{http://www.w3.org/2005/Atom}url'

class Title(XmlElement):
  _qname = '{http://www.w3.org/2005/Atom}title'
  title_type = 'type'

Now for the whys and hows.

For the past few years I've been working with Web Services and most of them use XML to represent the data (though I hope JSON catches on more widely). There are some great XML libraries out there, and my library is based on one of them (ElementTree). XML parsing is certainly nothing new, so why create a new one?

The Why

There are a few limitations with the XML parsing approaches I've used in Python:

• XML structure isn't documented or available using help()
• No autocompete for finding elements in the XML
• If the XML changes in a new version of the web service, my code needs to be rewritten
• My code interacting with the XML is verbose

Source code can provide a wealth of information, but parsed XML doesn't have the same level of information richness as source code. Between tool tips in IDEs, auto-generated documentation, and autocomplete, having classes loaded for your XML models can bring the tree traversal logic closer to your fingertips. Many software development tools are optimized for working with predefined classes rather than generic XML objects.

However, one of the biggest drawbacks to representing each type of XML element with it's own class is that you end up needing to write lots of class definitions. For this reason I've tried to make the XML class definitions as compact as possible. Specifying a simple XML class only takes two lines of code. For each type of sub-element and each XML attribute, you can add one line of code. You don't need to declare all of the elements or attributes either. The XmlElement will preserve all of the XML which it parses. If there are class members which correspond to a specified sub-element, the element will be placed in that member. Any unspecified elements will be converted to XmlElement instances. You can search over all XML elements (both anticipated members and unanticipated generic objects) using the get_elements method. XML attributes are handled in a similar fashion and can be searched using get_attributes.

I've saved the most unique feature of this library for last: Sometimes web services change the XML definition thereby breaking your code. If it is something small like a change in XML namespace or changing a tag, it seems like such a waste to have to edit lines upon lines of code. To address this kind of problem, this XML library supports versioning. When you parse or generate XML, you can specify the version of the available rules that you'd like to use. You can use the same objects with any version of the web service.

To use versioning, write a class definition with tuples containing the version specific information:

class Control(XmlElement):
  _qname = ('{http://purl.org/atom/app#}control', #v1
            '{http://www.w3.org/2007/app}control') #v2
  draft = Draft
  uri = 'atomURI'
  lang = 'atomLanguageTag'
  tag = ('control_tag', 'tag') # v1, v2

class Draft(XmlElement):
  _qname = ('{http://purl.org/atom/app#}draft', 
            '{http://www.w3.org/2007/app}draft')

If you create an instance of the Control element like this:

c = Control(draft=Draft('yes'), tag='test')

Then you can generate XML for each version like this:

c.to_string(1)

returns

<control xmlns="http://purl.org/atom/app#" 
    control_tag="test">
  <draft>yes</draft>
</control>

while

c.to_string(2)

returns

<control xmlns="http://www.w3.org/2007/app" 
    tag="test">
  <draft>yes</draft>
</control>

Note the difference in XML namespaces in the above. I also added an example of an attribute name which changed between versions, though "tag" doesn't actually belong in AtomPub control (so don't go trying to use it m'kay).

Since this library is open source, you're free to examine how it works and use it however you like. Allow me to highlight a few key points.

The How

Earlier I showed how to define XML element classes which look for specific sub elements and attributes and convert them into member objects. I also mentioned that this XML library handles versioning, meaning that the same object can parse and produce different XML depending on a version parameter. Both of these are accomplished by creating class level rule sets which are built up using introspection the first time an XML conversion is attempted.

In pseudo-code it works like this.

XML --> object
  - find out the desired version
  - is there an entry for this version in _rule_set?
  - if not, look at all XML members of this class 
      in _members
  - create XML matching rules based on each member's type
      (and store in _rule_set so we don't need to generate 
       the rules again)
  - iterate over all sub-elements in the XML tree
  - sub-elements and attributes which are in the rule set 
      are converted into the declared type
  - sub-elements and attributes which don't fit a rule are
      stored in _other_elements or _other_attributes

When generating XML the process is similar but slightly different.

object --> XML
  - create an XML tree with the tag and namespace for this 
      object given the desired version
  - look at all members of this class in _members
  - tell each member to attach itself to the tree using 
      it's rules for the desired version
  - iterate through _other_elements and _other_attributes 
      and tell each to attach to the XML tree

Armed with the above explanation, understanding the source code should be a bit easier.

Vim Line Length

I happen to be a fan of the vi text editor. I also happen to be a fan of code style guidelines. And sometimes it is very handy to know if a line in your source code has gone over eighty characters long. Some style guides still recommend a max line length of eighty characters, a tradition dating all the way back to IBM punch cards, circa 1928. For an even more impressive feat of long term engineering influence look up how the size of Roman war chariots determined the size of NASA space shuttle booster rockets. But I digress.

To mark the characters which are over the eighty character limit, enter the following either within vim or in your .vimrc file.

:match ErrorMsg '\%>80v.\+'

I've noticed that some blogger templates tend to display fifty-five characters in a fixed width format (as in a <pre> block or <code>). To edit code which I plan to post on Blogger, I sometimes turn on

:match ErrorMsg '\%>55v.\+'

Atom-Powered Robots Run Amok

I imagine a grand total of two of my readers will recognize the title of this post. (Hint: it's from RFC 5023.) But it seemed an appropriate title for a recent addition to a local office building where some Android developers tend to congregate.

(Note the tiny t-shirt logo.)

Ai luff mai Android.

Twitter Client

As a proof of concept for using the sippycode HTTP library which I wrote about in my last post, I decided to create a simple text console client for Twitter. Download the Twitter terminal application here.

Twitter's RESTful API is quite simple, and I wrote an open source library for Twitter based on the sippycode HTTP library in a few minutes. Here's an example of posting a new update (tweeting):

import sippycode.http.core as http_core
import sippycode.auth.core as auth_core

class TwitterClient(object):

  def __init__(self, username, password):
    self._credentials = auth_core.BasicAuth(username, 
                                            password)

  def update(self, message):
    request = http_core.HttpRequest(method='POST')
    http_core.parse_uri(
            'http://twitter.com/statuses/update.xml'
        ).modify_request(request)
    request.add_form_inputs({'status': message})
    self._credentials.modify_request(request)
    client = http_core.HttpClient()
    response = client.request(request)
    return response

In the above, the client sends an authenticated POST to the updates URL. Using the TwitterClient in your code looks like this:

client = TwitterClient('my-username', 'my-password')
client.update('Try out this Twitter client: http://oji.me/wP')

To try out this Twitter console app, unpack the download and run sippy_twitter.py. With it, you can update your status on Twitter or read the updates from your friends. When reading, the client displays five updates at a time, since showing more at once would likely cause some to scroll off the top of the screen (assuming the terminal displays twenty-five lines).

This simple application was designed to be a proof of concept, but it's really grown on me. Cycling through all of my friend's updates doesn't require any scrolling, and it feels snappier than the web interface. It seems like others are enjoying this terminal client too.

There are quite a few ways that this client could be improved, so there's plenty of opportunity to pitch in if you are interested. I have received feature requests from friends who previewed this app, such as: support command line arguments which will allow the client to perform updates when being run from another program, show a running countdown from 140 characters as you are typing your update (could probably be done using ncurses), ability to follow users, and read updates from just one user. If you'd like to participate in any of these, let me know in the comments.

Fire up your terminal and give this client a try. Why not post an update to @jscud right now?

An Open Source Python HTTP Client

At Super Happy Dev House 27, I made significant progress on an open source library for making HTTP requests in Python. For the past few years I've been working with web services and APIs (SOAP, REST (wikipedia) - specifically AtomPub, etc.) and I wanted to create an HTTP library which is simple, clean, and precise. Python has a couple of great HTTP libraries already, but one of them is a bit too low level (httplib) and the other is too high level (urllib2).

For example, in httplib you call a method to send data as if you are writing to a file (httplib uses sockets, after all). Required HTTP headers like Content-Length are not calculated for you. You'll need to handle cookies and redirects on your own. On the plus side, you get full control of what is being sent. The higher level library, urllib2, is built on top of httplib. It adds some handy abstractions, like calculating the Content-Length, but it also has some limitations. I haven't yet been able to figure out how to perform a PUT or DELETE with urllib2.

When making HTTP calls to web services, there are often a large number of HTTP headers, URL parameters, and components to the request. Making a request feels like making a function call in most HTTP libraries. In the past, I've wrapped these functions with successive layers containing more and more function parameters. For example, in a request to send a photo and metadata to PicasaWeb, you need to include an Authorization token, Content-Type specifying a MIME-multipart request and the multipart boundary, and a multipart payload consisting of the Atom XML describing the photo and the photo's binary data. If you add in the the ability to specify other headers and URL parameters, your function call might look like this:

def post_photo(url, url_parameters, escape_parameters, 
               photo_mime_type, photo_file_handle, 
               photo_file_size, metadata_xml, 
               metadata_mime_type, auth_token, 
               additional_http_headers)
...

# Sets the request's Host, port, and uri. 
# Makes the request into a MIME multipart request, 
# adjusts the Content-Type and recalculates 
# Content-Length.
# Sets the Authorization header
post_photo('http://picasaweb.google.com/data/'
           'feed/api/user/userID/albumid/albumID', None, 
           False, 'image/jpeg', photo_file, photo_size, 
           atom_xml, 'application/atom-xml', 
           client_login_token, None)

To use the above, you have to gather all of the information in one place, and make the function call. There are cases where you want a design like the above.

However, more and more I think of ways the program could be more cleanly structured if this information could be compartmentalized. This new library relies on an HttpRequest object which various parts of the program modify. Once all of the modifications have been applied, the fully constructed request is passed to an HttpClient which communicates with the server using httplib or urlfetch if you happen to be on Google App Engine. (Support for more HTTP libraries is certainly possible.)

The photo posting example from above could look something like this. Keep in mind that these steps could be carried out in a different order in different segments of code.

photo_post = HttpRequest(method='POST')
# Sets the Authorization header
client_login_token.modify_request(photo_post)
# Adds to the body and calculated Content-Length, 
# sets the Content-Type.
photo_post.add_body_part(atom_xml, 
    'application/atom+xml')
# Makes the request into a MIME multipart request, 
# adjusts the Content-Type and recalculates 
# Content-Length.
photo_post.add_body_part(photo_file, 'image/jpeg', 
    photo_size)
# Sets the request's Host, port, and uri. 
parse_uri('http://picasaweb.google.com/data/'
          'feed/api/user/userID/albumid/albumID'
          ).modify_request(photo_post)

In fact, the above code could make up the body of the post_photo function described in the first code snippet.

I created an open source project for this and other small projects called sippycode (a play on sippy cup). This is a place where code can grow up.