A Test Client for App Engine

I created a simple utility library in Python to help anyone debug their App Engine application. Many of the App Engine apps that I've seen use HTTP, HTML form posts, and the Users API and the easiest way to test these features is to fire up the web browser and click through the web pages generated by the app. However this can be a bit slow and repetative and it is difficult to determine exactly what is being sent over the wire (though this is greatly helped by using wireshark, fiddler, tcpdump, or antoher network packet sniffing tool).

Enter my little App Engine HTTP module. It provides a simple interface for making arbitrary HTTP requests and will print the full request and response to the terminal (though you can turn the noisy printing off if you want). Download, copy the http.py file to you working directory and try it out in your Python interpreter.

For our first demonstration, let's try to visit the Google search page.

import http
client = http.Client()
resp = client.request('GET', 'http://www.google.com')

You should see your request and the server's response (with the HTML for the Google Search page) in your terminal window. This should work with just about any website out there.

Other HTTP debugging tools can show you the request and response like this, but I find that this kind of simple Python client can be useful in writing end-to-end or integration tests which contact your App Engine app remotely.

Along those lines, one of the things which standard HTTP debugging tools do not provide, is a way to sign in to an App Engine app with a Google Account so that the App Engine Users API can identify the current user. I wrote an extremely simple app which illustrates the Users API, try it out here:

http://jscudtest.appspot.com/user

After signing in, the page should simply say, "Hello yourusername (yourusername@yourdomain.com)" You'll notice that during the sign in process, you signed in on www.google.com/accounts and were asked to approve access to the app. This kind of interaction works great in a browser, but can be tricky when you are using a command line, browserless, client.

It is possible however, to sign in to an App Engine app without using a browser. You can use the same technique used in appcfg, use ClientLogin and use the authorization token to obtain an app specific cookie which indicates the current user. This simple HTTP library can do this for you and all subsequent requests will use this cookie to tell the App Engine app who the current user is. Try it out by making the request to the simple user app that you visited earlier:

import http
client = http.Client()
client.appengine_login('jscudtest')
resp = client.request('GET',
                      'http://jscudtest.appspot.com/user')
print resp.body

You should see the following text displayed in the terminal:

Hello, yourusername (yourusername@yourdomain.com)

You can use the appengine_login method with your own app, just change the argument to the App ID of the app you want to access.

Along with simplifying access to apps which use Google Accounts, I wanted this library to simplify the process of using another feature used by many web apps: HTML form posts. Now I'm certain you've used HTML forms before, here's a simple example:

http://shoutout.appspot.com/

The above app uses both the Users API and a simple form. As an alternative to visiting this page in the web browser, you can post your shout-out using the following:

import http
client = http.Client()
client.appengine_login('shoutout')
client.request('POST', 'http://shoutout.appspot.com/',
    form_data={'who': raw_input('From: '),
               'message': raw_input('Message: ')})

If you've even wondered what gets sent across the wire to post on a form like this, look back in your terminal to see the request from your computer and the response from the server (this is of course just the HTTP layer, wireshark will show you traffic on the IP and Ethernet layer as well).

That's really all there is to it. I designed this as just a simple script to use on the command line and I wrote it in less time than it's taken me to write this blog post about it (I borrowed atom.http_core from the gdata-python-client as a foundation). With some tweaks to remove the interactive (getpass and raw_input) calls and replace them with parameters, I could see this module as a utility layer in a larger, more complex, App Engine client application. If you're creating on I'd love to hear about it ;-)

For more information on how the appengine_login method works behind the scenes, see this presentation I gave a few months ago:



Many thanks to Trevor Johns and Nick Johnson for helping me to understand how this ClientLogin-to-cookie exchange works.

I'm sure that App Engine's Java runtime users would appreciate a port of this simple library to Java, if you feel so inclined.

An Open Source Python HTTP Client

At Super Happy Dev House 27, I made significant progress on an open source library for making HTTP requests in Python. For the past few years I've been working with web services and APIs (SOAP, REST (wikipedia) - specifically AtomPub, etc.) and I wanted to create an HTTP library which is simple, clean, and precise. Python has a couple of great HTTP libraries already, but one of them is a bit too low level (httplib) and the other is too high level (urllib2).

For example, in httplib you call a method to send data as if you are writing to a file (httplib uses sockets, after all). Required HTTP headers like Content-Length are not calculated for you. You'll need to handle cookies and redirects on your own. On the plus side, you get full control of what is being sent. The higher level library, urllib2, is built on top of httplib. It adds some handy abstractions, like calculating the Content-Length, but it also has some limitations. I haven't yet been able to figure out how to perform a PUT or DELETE with urllib2.

When making HTTP calls to web services, there are often a large number of HTTP headers, URL parameters, and components to the request. Making a request feels like making a function call in most HTTP libraries. In the past, I've wrapped these functions with successive layers containing more and more function parameters. For example, in a request to send a photo and metadata to PicasaWeb, you need to include an Authorization token, Content-Type specifying a MIME-multipart request and the multipart boundary, and a multipart payload consisting of the Atom XML describing the photo and the photo's binary data. If you add in the the ability to specify other headers and URL parameters, your function call might look like this:

def post_photo(url, url_parameters, escape_parameters, 
               photo_mime_type, photo_file_handle, 
               photo_file_size, metadata_xml, 
               metadata_mime_type, auth_token, 
               additional_http_headers)
...

# Sets the request's Host, port, and uri. 
# Makes the request into a MIME multipart request, 
# adjusts the Content-Type and recalculates 
# Content-Length.
# Sets the Authorization header
post_photo('http://picasaweb.google.com/data/'
           'feed/api/user/userID/albumid/albumID', None, 
           False, 'image/jpeg', photo_file, photo_size, 
           atom_xml, 'application/atom-xml', 
           client_login_token, None)

To use the above, you have to gather all of the information in one place, and make the function call. There are cases where you want a design like the above.

However, more and more I think of ways the program could be more cleanly structured if this information could be compartmentalized. This new library relies on an HttpRequest object which various parts of the program modify. Once all of the modifications have been applied, the fully constructed request is passed to an HttpClient which communicates with the server using httplib or urlfetch if you happen to be on Google App Engine. (Support for more HTTP libraries is certainly possible.)

The photo posting example from above could look something like this. Keep in mind that these steps could be carried out in a different order in different segments of code.

photo_post = HttpRequest(method='POST')
# Sets the Authorization header
client_login_token.modify_request(photo_post)
# Adds to the body and calculated Content-Length, 
# sets the Content-Type.
photo_post.add_body_part(atom_xml, 
    'application/atom+xml')
# Makes the request into a MIME multipart request, 
# adjusts the Content-Type and recalculates 
# Content-Length.
photo_post.add_body_part(photo_file, 'image/jpeg', 
    photo_size)
# Sets the request's Host, port, and uri. 
parse_uri('http://picasaweb.google.com/data/'
          'feed/api/user/userID/albumid/albumID'
          ).modify_request(photo_post)

In fact, the above code could make up the body of the post_photo function described in the first code snippet.

I created an open source project for this and other small projects called sippycode (a play on sippy cup). This is a place where code can grow up.

Hey look, a simple web server

I think I'm done writing my web server. I have gotten it to do what I want, namely this. This server will:

1. Run just about anywhere. I sometimes run it off of a USB pen drive.
2. Send all traffic over an HTTPS connection.
3. Handle user authentication and permissions. You can only read and write where you have permissions.
4. Allow you to store data in a remote location. Just POST to a URL to store something at that location, GET to retrieve it.

There really wasn't much to it. I was able to write this quickly and there wasn't that much code. The main thing it is lacking (in my opinion) is speed. This could be fixed by making it multithreaded and adding some caching since right now it always reads from the disk. Without further ado, here's the code. I also wrote a Python client and a JavaScript client to go with it will be coming soon. If you found this to be useful, please let me know.

I've also been looking at CherryPy as a framework to create the same type of portable, simple, and secure web server. As usual, stay tuned for details.

A simple HTTPS server

Recently, I've been working on an AJAX application in my spare time and there's something I could really use: a simple network data store.

A JavaScript application isn't very useful without some persistent data. However this usually requires running a web server. My original idea was to distribute the application as a file which is loaded from the local disk. At this point you may be saying, "Wait a minute JavaScript running in a browser can't access the local disk." But scripts can read and write to the local disk if they are loaded from disk instead of the Internet. See TiddlyWiki for a great example of a useful application that uses this design. The problem though, is what happens when you want to sync the data from the AJAX application across multiple computers. Well, once again, it looks like I need a web server after all.

So I set out to build a simple server. All it really needs to do is allow applications to store and retrieve data. To make sure that the data remains a secret, the traffic will be sent over an HTTPS connection. Access to certain directories and files on the server will be granted only to select users, so usernames and passwords are required too. Since I've been working with Python recently, I tried to see if it was possible to create a simple HTTPS server which could handle GET and POST requests and perform dynamic behavior. I found a great example on activestate.com which uses an open SSL .pem file. The instructions in the article made setting up this server a breeze. I've been working on a customized version of the above example, but it isn't quite ready. As usual, stay tuned :)