Showing posts with label JavaScript. Show all posts
Showing posts with label JavaScript. Show all posts

Friday, May 11, 2012

JavaScript Caching: appendChild vs document.write

I found myself trying to prefetch some data which would appear in an iframe. I wanted to see if requesting the same JSON in the parent page and then in the iframe would cause the second request, made in the iframe, to hit the browser's cache in all cases. It turns out that for Internet Explorer and Web Kit based browsers it depends on how you request the JSON in the parent page.

If you use appendChild in IE the identical JS request will hit the browser cache.
If you use document.write in IE it will not hit the cache.

However, in Chrome and Safari if you appendChild in the parent, you will miss the cache. Using document.write in the parent page causes a cache hit.

Here are the results I gathered.
Cache Hit? both document.write parent document.write
iframe appendChild
parent appendChild
iframe document.write
both appendChild
Internet Explorer miss miss hit hit
Firefox hit hit hit hit
Chrome hit hit miss miss
Safari hit hit miss miss
Opera hit hit hit hit
Android ? ? ? ?
iOS ? ? ? ?

All versions of IE behaved the same, I didn't check different versions of the other browsers.

To test this I wrote a test web application called json-caching-exploration. It's a simple web app that runs on App Engine.

Server and HTML
JavaScript

I haven't run this against mobile browsers yet. If you give it a try please let me know what you find.

It's a shame that these browsers behave differently, especially given the differences in browser behavior between loading JavaScript using document.write and appendChild.

Friday, September 10, 2010

Greekish

I've long been fascinated with other alphabets. All of these strange and unusual symbols, it's almost like a code. This love of secrets was one of the reasons that I studied Ancient Greek. After reading and writing quite a bit of Greek, reading the alphabet became second nature. I even began taking notes using the Greek alphabet but using English words (since my Greek vocabulary is sadly inadequate). Performing simple character substitution sounded like a perfect one-hour project so I whipped up a simple web page to convert English text into a Greek alphabet equivalent. I call it Greekish. For example, the phrase
So long and thanks for all the fish.
becomes
Σο λονγ ανδ θανκσ φορ αλλ θε φισh.
which would be quite confusing to a Greek speaker but perfectly natural to an English speaker who knows the Greek alphabet.

Note that some English characters do not have direct equivalents in Greek. A c would be a κ, a σ, or a χ for ch. The h is one of the more interesting stories. For a leading h before a vowel Greek uses a breathing mark. When combined with a consonant, special characters are used, like θ for th, χ for ch, and φ for ph. I chose to use φ only for the English f, not ph, since ph does not make the f sound in some English contexts. The word uphill is one example. Also, I didn't bother to handle the special case of s at the end of word, for which ς is used instead of σ.

Now that you know more than you ever wanted to about the Greek alphabet, what is a simple project that you can tackle in an hour? Don't just think of one, go do it!

Friday, June 18, 2010

JavaScript Tricks to Speed up Your Site

One of the techniques which makes the web so powerful is the ability to load code, images, and other resources from all over the Internet. So often though, the process of loading these resources and ensuring that all of the required pieces are in place leads to a slow experience for visitors. With the ability to include so much code from across the web, visiting a site could potentially be like installing a new program when it comes to the amount of stuff that needs to be downloaded.

With this in mind, there are a couple of nifty tricks that can help make your app more responsive and I've written up an example site and testing server that shows some ideas for speeding up the user experience when you need to wait for the DOM to load or for additional JavaScript to be fetched and run. We'll begin with document operations.

Often the JavaScript running on a page manipulates the DOM, using document.getElementById here and document.createElement there. In order to ensure that all the pieces of the page are in place, web programmers often take advantage of the onload callback. It might be used like this
<body onload="runMyCodeNow()">
Using this technique ensures that all of the things your code might want to read and write from the page are in place. All images have been downloaded, CSS rules have been applied, the layout is all there. However, all of this comes with a cost, your code doesn't run until every last resource has been fetched and rendered. Even the little footer at the bottom of the page, for example, that your code doesn't care about.

There is a another way, we could request that resources be loaded in parallel and start executing our code before the page is fully loaded. Chances are, your code doesn't need the complete page to be loaded before it starts running, and running before onload will reduce the delay for your users. Before I dive into how this can be accomplished, lets look at an example which uses the old fashioned way.

Lets say you have a web page, a little HTML which includes five JavaScript files. One may be a library used to do animation, another one for loading the users data. In any case, all of these files need to be loaded and some of them depend on others.

The biggest bottleneck for your users is almost certainly having all of these resources load. Network latency is a killer, and something that is often overlooked during development. To create a simulated network environment which can give a more realistic (or even pessimistic) view of the cost of loading these resources, I wrote a "slow server" which can introduce a delay to the file requested. Here is the code for my testing server (designed to run on App Engine):
def FilePath(path):
"""The requested path into a local file path."""

return os.path.join(os.path.dirname(__file__), 'files', path[1:])


class SleepyRenderer(webapp.RequestHandler):
"""Serves the requested page with a client configured delay.

Delay is given as a URL parameter in hundredths of a second to delay.
For example, 200 means wait 2 seconds before responding.

Example request:
http://localhost:8080/hi.html?delay=300&contenttype=text/html
"""

def get(self):
path = self.request.path
delay = self.request.get('delay')
content_type = self.request.get('contenttype') or 'text/html'
if delay:
time.sleep(int(delay)/100)
http_status = 200
requested_file = None

try:
requested_file = open(FilePath(path))
self.response.out.write(requested_file.read())
requested_file.close()
except IOError:
http_status = 404

self.response.set_status(http_status)
self.response.headers['Content-Type'] = content_type


def main():
application = webapp.WSGIApplication([('/.*', SleepyRenderer)],
debug=True)
util.run_wsgi_app(application)


if __name__ == '__main__':
main()
With the above code we can introduce a delay on each individual file. To see this in action with our example, here is some HTML which shows a traditional approach, include script includes and an onload callback when everything has loaded.
<html>
<head>
<script src="/testa.js?delay=500&contenttype=text/javascript"></script>
<script src="/testb.js?delay=400&contenttype=text/javascript"></script>
<script>
function init() {
document.getElementById('output');
output.innerHTML = [
'a is ' + a,
'b is ' + b,
'c is ' + c,
'd is ' + d,
'e is ' + e
].join('<br>');
}
</script>
<script src="/testc.js?delay=300&contenttype=text/javascript"></script>
</head>
<body onload="init()">
<script src="/testd.js?delay=200&contenttype=text/javascript"></script>
<div id="output"></div>
<script src="/teste.js?delay=100&contenttype=text/javascript"></script>
<script src="/testa.js?delay=500&contenttype=text/javascript"></script>
</body>
</html>
With the above, the page takes several seconds to load and when the very last script has loaded, the 'output' div gets its contents. In many cases, the code really doesn't need to wait for all resources to load, only the ones that are necessary for the code to run. In this case, since the information is added to the output div, we need the output div to exist in the DOM, but we may not need the entire page to load.

If you look at this loading process in a profiler you might see something like this:Now for our first nifty trick. One way to check to see if the necessary prerequisites are present, is by polling the DOM or the JavaScript environment, to see if conditions are right for the code to run. Here is an example of how this code might be rewritten when using some polling helper functions:
    <script>
loader.whenNodePresent('output',
function() {
var output = document.getElementById('output');
loader.whenReady(function() {return window['a'];},
function() {
output.innerHTML += 'a is ' + a + '<br>';
});
loader.whenReady(function() {return window['b'];},
function() {
output.innerHTML += 'b is ' + b + '<br>';
});
loader.whenReady(function() {return window['c'];},
function() {
output.innerHTML += 'c is ' + c + '<br>';
});
loader.whenReady(function() {return window['d'];},
function() {
output.innerHTML += 'd is ' + d + '<br>';
});
loader.whenReady(function() {return window['e'];},
function() {
output.innerHTML += 'e is ' + e + '<br>';
});
})
</script>
The code to track the prerequisites and poll is quite simple:
loader.waiting = [];


loader.whenReady = function(testFunction, callback) {
if (testFunction()) {
callback();
} else {
loader.waiting.push([testFunction, callback]);
window.setTimeout(loader.checkWaiting, 200);
}
};


loader.checkWaiting = function() {
var oldWaiting = loader.waiting;
var numWaiting = oldWaiting.length;
loader.waiting = [];
for (var i = 0; i < numWaiting; i++) {
if (oldWaiting[i][0]()) {
oldWaiting[i][1]();
} else {
loader.waiting.push(oldWaiting[i]);
}
}

if (loader.waiting.length > 0) {
window.setTimeout(loader.checkWaiting, 200);
}
};


loader.whenNodePresent = function(nodeId, callback) {
loader.whenReady(function () {
return document.getElementById(nodeId);
}, callback);
};
In the above we use the whenReady function which takes a couple of functions, one to return a truthy or a falsey value, and one to call back when the first function evaluates to true. If the condition function isn't true when this first call is made, we check back every so often to see if it is ready.

With these changes, we shave several seconds off of the user perceived loading time. Specifically we no longer need to wait for the duplicate load (of the testa script) at the end of the body. The page also appears to be more responsive because the later script's messages appear just after they load but before the page is complete.

Now that we've seen a way to work around the need for an onload callback, lets look at another place we can tweak the browser's behavior to make a web page more responsive: dynamic script loading.

The most straightforward way to include new code in your page is to use a script tag, something like:
<html>
<head>
<script src="some_great_sites_javascript">
...
When the browser's JavaScript interpreter encounters this script src, it stops whatever it's doing and fetches that resource. It doesn't do any more rendering or executing of code until it's finished. This behavior varies a bit in different browsers and is likely an artifact of an old design in which this kind of single threaded behavior was the only option. Since some sites might depend on this linear behavior to get a script's dependencies all in order, this quirk might be with us for a long time. Most of this time, waiting like this is a really silly idea. How often do the scripts that you include depend on one another?

There are a few parts to this trick. The first is to not put all of script includes in the HTML, you could have JavaScript add new script elements to the page which will cause new code to be loaded as needed. In this way, you could load only the resources that are needed at the moment, perhaps some resources would not end up being requested at all. Including a new script could be done in two ways:
document.write('<script src="somefile.js"></script>');
or
var newScript = document.createElement('script');
newScript.src = 'somefile.js';
document.body.appendChild(newScript);
Each of the above is appropriate in different situations. Document write adds HTML directly into the page at the point where the page is being loaded, it should only be used for script tag inclusion if the page is not yet loaded. If that page is loaded, using document.write to add the script tag will wipe out the existing body entirely. I've seen this issue in the wild, if you assume document.write is always safe, you'll be bitten when using it after the page has loaded.

Instead you can perform a check to see if document.body exists, if it does then use document.body.appendChild. If it does not yet exist, use document.write. The code for this loader logic might look something like this:
loader.loadScript = function(url) {
if (document.body) {
var newScript = document.createElement('script');
newScript.type = 'text/javascript';
newScript.src = url;
document.body.appendChild(newScript);
} else {
document.write('<scr' + 'ipt type="text\/javascript" src="' +
url + '"><\/scr' + 'ipt>');
}
};
Now we can request that new JavaScript code be loaded on the fly and it works when the page has not yet finished loaded as well as after it has.

There is one more trick we can add to this loader. Some browsers will interpret the JavaScript in the order in which the scripts were requested, not the order in which they finished loading. That means that a fast loading script further down the list won't be run until a slower script, which appears above it, is loaded. One way we could defeat this delay, is to break the script includes out of linear execution in the JavaScript. If you use setTimeout to introduce a delay in adding the script include to the page, then the code which sets up the script requests can finish quickly and the browser will get back to the script requests later without the same linear constraints. In our code, we wrap the section of loader.loadScript in a short timeout as follows:
loader.loadScript = function(url) {
window.setTimeout(function() {
if (document.body) {
var newScript = document.createElement('script');
newScript.type = 'text/javascript';
newScript.src = url;
document.body.appendChild(newScript);
} else {
document.write('<scr' + 'ipt type="text/javascript" src="' +
url + '"><\/scr' + 'ipt>');
}
}, 1);
};
With the above changes in place, our example page from before now loads like this when profiled (note that the messages appear in the order that the scripts were loaded, we don't have to wait for everything before we edit the page):Through the course of this post, I've written a small library for using these tricks when loading JavaScript dynamically in the page as well as a server for trying it out. These are available here as open source code. There are some improvements that could be made here. Off the top of my head, the checkWaiting function could eventually time out if a condition continues to not be met. Also the loader could do more to check to see if a requested script has already been loaded. Any more ideas?

Wednesday, January 13, 2010

DOM Manipulation in JavaScript, a Utility

In my latest addition to the q12 JavaScript library, I have added a simple function to construct a nested document structure. I grew tired of writing repetitive and difficult to follow code like this
var div = document.createElement('div');
div['id'] = 'outer-div';
var link = document.createElement('a');
link.href = 'http://blog.jeffscudder.com';
link.appendChild(document.createTextNode('my blog'));
div.appendChild(link);
Using the new tree function this would become:
var div = tree([
'div', {
id: 'outer-div'}, [
'a', {
href: 'http://blog.jeffscudder.com'}, [
'my blog']]]);
In the process of writing this utility function, I came across another small oddity in editing HTML using JavaScript related to the style attribute. Most HTML attributes are simple text properties, however when editing the CSS style of a node, you cannot set the full content of the style attribute.
// You cannot do this:
div.style = "color:red; font-weight:bold";
Instead, you set each sub item in the style attribute:
// This works:
div.style.color = 'red';
div.style['font-weight'] = 'bold';
The tree function that I've written handles this nested style structure as well. Here's an example:
['div', {
'id': 'foo',
'style': {'color': 'red', 'font-weight': 'bold'}},
'This works too!']
Now that you have an idea of how to use this utility, here's the code. Minus the comments it's only about twenty-one lines.
/**
* Creates a DOM tree from a simple list.
* The structure of the tree passed in is as follows:
* ['elementTag',
* {attribute1: value,
* attribute2: value,
* style: {property1: value,
* property2: value}},
* 'child text node',
* ['elementTag',
* {property: value},
* 'grandchild text node'],
* 'third node']
* The above will result in a DOM node which has three child nodes, the
* first and third will be text nodes because the values were strings.
* The second child node will be a DOM node as well.
*
* @param {Array} t The tree's structure as a collection of strings, lists,
* and simple objects. The structure is as follows
* ['elementTag', {attributes}, child, child, child, ...]
* @return {Element} Returns a new DOM element.
*/
function tree(t) {
// Create the node using the tag which is first in the list.
var domNode = document.createElement(t[0]);
// Add all HTML attributes to the node.
for (var key in t[1]) {
// The style attributes get special treatment.
if (key == 'style') {
for (var styleAttribute in t[1].style) {
domNode.style[styleAttribute] = t[1].style[styleAttribute];
}
} else {
domNode[key] = t[1][key];
}
}
// Iterate over all child nodes, converting them to either text or HTML
// nodes.
for (var index = 2, child; child = t[index]; index++) {
if (typeof(child) == 'string') {
domNode.appendChild(document.createTextNode(child));
} else {
// Buid recursively.
domNode.appendChild(tree(child));
}
}
return domNode;
}
What do you think, is there anything you would add?

On a realted note, if you're interested in a more powerful templating system, you might want to try Closure Templates which were recently open sourced.

Thursday, October 22, 2009

The joys of onKeyDown

I recently found myself in need of a way to detect when a user presses enter when typing in a text input box and happened upon one of the quirkiest of browser features, key press detection in JavaScript. Searching the Internet provided a few ideas but nothing that quite fit what I was looking for, so I thought maybe I should get the word out about what I've learned.

Lets assume that we have a form with an input box and we want to perform some special action when the user presses the enter key while focus is on the text input box. We begin by adding an onKeyDown event to the input box (you might also consider onKeyUp, but onKeyPress seems like it might be somewhat quirkier).
<form>
<input type="text" id="my-input">
</form>
<div id="output"></div>
<script>
document.getElementById('my-input').onkeydown = function(e) {};
<script>
Now the first hurdle is that in IE, Safari, and Chrome you can access the ASCII character code for the key which was pressed in window.event.keyCode but in FireFox you'll need to get the event object and look at the which member.
<script>
document.getElementById('my-input').onkeydown = function(e) {
var keyCode = 0;
if (window && window.event && window.event.keyCode) {
keyCode = window.event.keyCode;
} else if (e && e.which) {
keyCode = e.which;
} else {
alert('fail');
}
document.getElementById('output').innerHTML = 'You pressed ' + keyCode;
};
<script>
Now there is one gotcha with what we've written so far. Each key that we press when focus is on the desired input causes our code to be run, but the browser also performs the default behavior for that key. If our input is a simple text box, the characters pressed will show up in the text box. If the input is part of a form with a submit button, pressing enter will cause the form to be submitted. If the form is submitted then the browser is sent to a different page which in this case is not what we want.

There are a few ways to prevent the key presses from also triggering the default behavior and do only our behavior. One of the simplest ways is to return false from the onKeyDown function. If we change our handler to the below, characters will not show up in a text input box because our handler consumes them.
 document.getElementById('my-input').onkeydown = function(e) {
var keyCode = 0;
if (window && window.event && window.event.keyCode) {
keyCode = window.event.keyCode;
} else if (e && e.which) {
keyCode = e.which;
} else {
alert('fail');
}
document.getElementById('output').innerHTML = 'You pressed ' + keyCode;
return false;
};
With the above, the last key press is displayed in the output, but in most cases we probably do want the user to see the chracters they've typed showing up in the input box. We just want to prevent the form submit when the user presses the enter key. To accomplish this behavior, we can return false when we don't want the key press to propoage and return true when the default behavior should also be performed.
 document.getElementById('my-input').onkeydown = function(e) {
var keyCode = 0;
if (window && window.event && window.event.keyCode) {
keyCode = window.event.keyCode;
} else if (e && e.which) {
keyCode = e.which;
} else {
alert('fail');
}
document.getElementById('output').innerHTML = 'You pressed ' + keyCode;
if (keyCode == 13) { // 13 is the key code for the enter key.
return false;
} else {
return true;
}
};
There you have it, a way to detect when the enter key is pressed in our input box while also preventing the form from being submitted.

Wednesday, August 26, 2009

A Matrix Sim in JavaScript

Do you ever find yourself working on a large project with a long lead-in time before you have visible results to show? I had been frustrated by this feeling recently. I had been focusing on a personal project which will likely be a long time in the making, so I decided to pick up a short, simple, and quick project to keep things lively and fresh. So our story begins.

During high school, I was wowed by the movie The Matrix. The special effects were mind blowing (for their day) and the story line was something I could really get behind (not so with the sequels, but I digress). I imagine this is how my elders felt when Star Wars was first released.

One of the most memorable visuals in the movie is the code of the matrix zipping by vertically on the screen in a retro green on black. Cypher, makes a statement, which seems to be a favorite among programmers, that he no longer sees the code itself, but what it represents. He has become fluent to the point of transcendence - a geek nirvana experience.

At this point in my life I had written a few moderately complex programs, mostly using Turbo C, and I thought: "Why, I could make my computer look like the matrix." It was one of the more involved programs I had written at the time and I was quite pleased with the result. I brought it in to the computer lab at school, loaded it onto all of the machines, and had the whole lab running it at once. I'm pretty sure that Ben, Yed, and Roman G. remember it. Sadly I doubt the code would run on the DOS emulator that ships with Windows XP and higher. I recall vaguely that Roman ported it to Win32 or something in our high school computer science class. Now that I think about it, I guess you could say that this was my first open source experience.

In thinking of something quick and easy to create with some immediate visual payoff, I decided to recreate my little DOS matrix sim, but this time in the browser. Behold: The matrix in JavaScript.

It turns out that the CSS rules needed to keep columns at a fixed width and height was actually quite complex and took longer than expected to get right. The vertical line height was still elusive in Chrome and Safari, but worked in Firefox and Opera. I don't think that this works yet in Internet Explorer. Though if past traffic is any indication, none of you would have noticed had I not said something :-)

Many thanks to Scudmissile for his mighty CSS kung fu. Oh, and that whirring sound that you hear may be your CPU fan trying to take flight. Apparently rendering a vertical text flow like this using DOM manipulation, CSS, and Math.random may be a bit more processor intensive than my obsolete C code.

Sunday, September 14, 2008

The Ascention of JavaScript

I added a couple of new things to my little JavaScript library (q12). It turns out there was a flaw in the library when making HTTP requests which included custom HTTP headers in Firefox3 (and maybe others). If you ever run into (NS_ERROR_FAILURE) [nsIXMLHttpRequest.setRequestHeader], try opening the HTTP request first (as I recently learned). I've always said that writing this library was for education more than any anticipated serious use.

I also added a function for ARC4 encryption, which doesn't provide strong security, but it is still used in some places.

This has been quite a big month for JavaScript. In just a couple of weeks V8, Tracemonkey, and SquirrelFish Extreme have leapforgged one another and improved the state of JavaScript execution speed. It's gotten me thinking.

The more I use JavaScript, the more I like it. The language itself is extremely simple, I would argue even simpler than C - it's the APIs in browsers which are sometimes complex and inconsistent ;-) With all of these optimizations, JavaScript is looking better and better as a language in which to implement a new dynamic language.

Between the Just In Time compiler (JIT) and the polymorphic inline cache, some state of the art tools are coming to bear on one of the most widely available programming languages today. JavaScript is making strides to overcome a very difficult problem: maintaining complete runtime flexibility while providing speedy execution. There are significant investments being made in JavaScript as a language, and it seems that some of these open source components should be reusable. I'm not sure how much of these libraries would stand alone, but it would be a very interesting experiment to see if a new language could be implemented in JavaScript.

Sunday, September 07, 2008

A Revived Project

q12 is back!

I'm slowly starting back up again on my note taking wiki application. I created my own version of TiddlyWiki several months ago, but then started working on other projects. I'm planning to rewrite my note taking Ajax application to run on Google App Engine, and as I was getting started I realized that there were a few things missing from the Ajax library that I had written as part of this project.

I had created my own simple unit test framework in JavaScript, and I finally got around to uploading the unit tests for the library to the open source project. I've also been learning about manipulating browser cookies from within JavaScript. Aside: Cookie's in JavaScript are weird! When you say document.cookie = something, reading document.cookie doesn't give you the same thing back (the expiration, domain, and path information are squirreled away somewhere else).

I've also added a minified version of the q12 library, it weighs in at a mere 10k. Download the library today!

Wednesday, July 23, 2008

Password Generator part 2

In a previous post, I mentioned that I had designed a simple password generator for use with the myriad of websites on which I have accounts. Rather that store passwords and carry them around with me, I've decided to carry around the code for a password generator in my head, so that I can generate difficult-to-guess passwords using a few easy to remember pieces of information. Namely these are a reusable master password, my username on the site, and the domain name of the site (like twitter.com for example).

The generator algorithm takes these three pieces of information as strings of text (ASCII characters to be exact), and uses them to populate three pseudo-random data streams (I used ARC4 for the pseudo-random algorithm because it is easy to memorize). These three streams are combined to create the characters in the password. For more details, see the list of steps in the first post about the password generator, or better yet, take a look at the source code.

I have uploaded the password generator here, so if you'd like to use it, feel free.

In order to account for some websites which do not allow special characters or passwords that are thirty characters long, I created a settings file which has special rules for some websites. If there are any websites that you would like me to add to the settings, please let me know in the comments.

Now for a disclaimer: I'm not entirely sure that these passwords would stand up to cryptanalysis. It might be possible to figure out the inputs (the three secrets). So I recommend just using it for websites which are not too sensitive. I'm just using it on social news websites at the moment.

Someday this might all be made unnecessary through the use of OpenID or some other authentication solution. I'm looking forward to it.

Tuesday, June 10, 2008

Simple Password Generator

My most recent little side project is a web page for generating passwords. It's a very simple page, which takes a few inputs, generates a pseudo-random stream from them, and produces a password using printable characters. Now now, hold it down. I can hear you from way over here. "But Scudder, aren't there like 40,000 password generators out there? Won't a web page be much slower at complex calculations than an installed desktop application? Wouldn't a truly random password stored in an encrypted keystore be a more secure way to handle passwords?" Well yes that all tends to be true. The idea here though, is to be able to regenrate passwords that I use non-critical accounts (you know, for sites like digg.com and reddit.com) on whichever computer I happen to be using. In fact, even if this password generator web app is for some reason unavailable, I made sure that it is simple enough that it all fits in my head. In a pinch, I could rewrite it from memory.

So here's how this little app generates passwords:
  1. Ask for a username, password, and domain (like example.com).
  2. Generate an RC4 stream for each of the three inputs above.
  3. Throw away the first 1,000 bytes from the three streams.
  4. Get three sets of the desired number of pseudo random bytes, one for each key stream. And xor each group of three bytes together to get single pseudo-random values. For example, if you want a 15 character password, get 15 bytes from each of the three keystreams.
  5. Translate the bytes into printable characters, or alphanumeric characters, by taking the modulus of the pseudo-random byte and using it as the index to looking up a character in a table.
A couple of the items above need some clarification. First is RC4. RC4 is a well known algorithm for generating a pseudo-random stream of bytes. It's not really up to snuff in terms of high security these days, but it is really simple. So simple in fact, that it is easy to remember the steps, and therefore, I chose it for this all-in-my-head password generator.

The second detail you would need to know in order to reproduce this generator, are the tables that I use to translate pseudo-random bytes into characters. I created this table, by typing in characters beginning on the first row of a US keyboard then I repeated the first row while holding the shift key. For example, the table begins with: `1234567890-=~!@#$%^&*()_+qwertyuiop[]QWERTYUIOP{}...

It turns out that some websites don't allow special characters like % and & in passwords. Why they don't is beyond me. So I created a table using the same method as above, but it only contains alphanumeric characters. Like this 1234567890qwertyuiopQWERTYUIOP... For websites which don't allow special characters, you can choose to use the alphanumeric table to translate the pseudo-random bytes into a password which is usable on the website.

I must sign off for now, so stay tuned for part two.

Tuesday, April 22, 2008

Early vs Late Binding

I've been thinking recently about programming languages (surprised?), specifically about the things that make them different. One of the really nice things about C, is that it compiles into machine code which tends to run lean and mean. By that I mean it is blazing fast and doesn't take up much memory. On the other hand, programming in Python and JavaScript has really been growing on me. There is so much flexibility to create elegant solutions quickly and without rewriting lots of existing code. In fact, I'd say greater ability to reuse existing code is a natural outgrowth of programming language flexibility.

So where does this flexibility come from? One place I tend to notice it most, is in the ability to give an existing function a new body, in other words, you can plug in different behavior in place of the default.

Here's a simple example to illustrate the idea. Let's say that we created a simple checkout register which takes a receipt, adds the sales tax, and spits out the grand total. Here's our code foundation in both Python and JavaScript (these two examples do essentially the same thing):

Python:
def CalculateTax(amount):
return amount * 0.18

class Receipt(object):

def __init__(self, items=None):
self.items = items or []

def CalculateTotal(self):
return sum([item + CalculateTax(item) for item in self.items])
JavaScript:
function calculateTax(amount) {
return amount * 0.18;
}

function Receipt(items) {
if (items) {
this.items = items;
} else {
this.items = new Array();
}
}

Receipt.prototype.calculateTotal = function() {
var total = 0;
for (var i = 0; i < this.items.length; i++) {
total += this.items[i] + calculateTax(this.items[i]);
}
return total;
}
To use the above code, you might write something like this:

Python:
my_order = Receipt([5.50, 10, 7.89])
print my_order.CalculateTotal()
JavaScript:
var myOrder = new Receipt([5.50, 10, 7.89]);
alert(myOrder.calculateTotal());
Now let's say someone asks you to change the tax rate which is used when calculating the total. Here's the catch, you're not allowed to change the existing code. It turns out this is actually really easy. You can define a new function, then make an existing function name point to the new function. Here's an example of how to inject our new code:

Python:
def CalculateHigherTax(amount):
return amount * 0.25

CalculateTax = CalculateHigherTax

print my_order.CalculateTotal()
JavaScript:
function calculateHigherTax(amount) {
return amount * 0.25;
}

calculateTax = calculateHigherTax;

alert(myOrder.calculateTotal());
After adding the above code to the foundation we started with, you will notice that the calculate total method now uses calculate-higher-tax instead of the original function, even though you are calling the same method on the same object as before. Congratulations, you have just witnessed late binding in action.

So what is late binding? The idea is that the computer decides which code should be executed while the program is running. This seems normal in scripting languages, but compiled languages often use this too (I'm looking at you Java and C++). For example, overloaded methods and polymorphism take advantage of late binding. With late binding you can change the meaning of an identifier (for example, change the behavior when you call a specific function) at just about any time.

Now lets take a look at a language which uses early binding. C is a great example. With early binding, the meaning of things like function names are locked in when the code is compiled. There is no dynamic lookup while the program is running to see which code should be executed, instead the address of the desired code is embedded directly into the binary machine code.

Here is how the same calculate-total example might look in C:
#include<stdio.h>

float CalculateTax(float amount) {
return amount * 0.18;
}

typedef struct {
float* items;
int num_items;
} Receipt;

float CalculateTotal(Receipt this_order) {
int i;
float total = 0;
for(i = 0; i < this_order.num_items; i++) {
total += this_order.items[i] + CalculateTax(this_order.items[i]);
}
return total;
}

int main(void) {
Receipt my_order;
float my_items[3] = {5.50, 10, 7.89};
my_order.items = my_items;
my_order.num_items = 3;
printf("%f\n", CalculateTotal(my_order));
}
If you try to set CalculateTax to a new function definition, you will get an error at compile time because a function cannot be changed once it is bound. Early binding tends to produce more efficient programs. However, if you want to, you can still use the flexiblity available in late binding in C.

Using function pointers, you can store the address of the code that you want to be executed, and change the address while the program is running. We can achieve the same late binding effects that I've illustrated in Python and JavaScript by making some small changes to the C code (marked in bold below). Declare a function pointer named TaxCalculator which will store the address of the desired calculate-tax function, then change CalculateTotal so that it uses the TaxCalculator instead of directly calling a calculate-tax function.
#include<stdio.h>

float CalculateTax(float amount) {
return amount * 0.18;
}

float CalculateHigherTax(float amount) {
return amount * 0.25;
}


typedef struct {
float* items;
int num_items;
} Receipt;

float (*TaxCalculator)(float) = &CalculateTax;

float CalculateTotal(Receipt this_order) {
int i;
float total = 0;
for(i = 0; i < this_order.num_items; i++) {
total += this_order.items[i] + (*TaxCalculator)(this_order.items[i]);
}
return total;
}

int main(void) {
Receipt my_order;
float my_items[3] = {5.50, 10, 7.89};
my_order.items = my_items;
my_order.num_items = 3;
printf("%f\n", CalculateTotal(my_order));
TaxCalculator = &CalculateHigherTax;
printf("%f\n", CalculateTotal(my_order));

}
There you have it!

Here's another way to think about this comparison. In high level languages which don't expose pointers, functions, variables, and other identifiers actually act like pointers.

Tuesday, March 11, 2008

BusyList

Andy and I started work on a simple little open source project for tracking tasks; it's called busylist. We wanted to experiment with Ajax, Python, and web service APIs, so we whipped up a basic system in a few hours. There is still quite a bit of work to be done, but it has been a great learning experience so far. An extremely alpha test version is available in subversion along with some instructions on the project's wiki pages. If you're interested, feel free to check it out (pun intended) and contribute if you like. It is an open source project after all.

Friday, December 21, 2007

An open source JavaScript library: q12

I've written some JavaScript utility functions as part of my ongoing wiki/note taking application. It is growing into a full fledged Ajax application with a web server and now a minimal Ajax library which I've decided to release as a separate open source project.

From the project's front page:

I found myself needing a few common utilities as I was writing an Ajax application. Rather than use a heavyweight or verbose library, I wanted something compact that minimized the amount of typing I needed to do. This is where the name q12 comes from, just three little keys up there in the upper left corner of the keyboard.

This library provides functions for the following:

  • Basic DOM manipulation
  • Asynchronous HTTP requests with callbacks
  • Class methods and inheritance
  • Base64 encoding and other forms of data escaping
  • AES encryption
Writing your own Ajax library is also a great way to learn JavaScript (IMHO). I'll be making little tweaks as I work further on my project, it's getting quite close. I think I've probably said that before but rewriting from scratch tends to set one back a bit. Third iteration's the charm?

Thursday, November 08, 2007

IE, XMLHttpRequest, and caching

I recently noticed an oft cited irritation with using XMLHttpRequest in Internet Explorer. Turns out, the browser intercepts HTTP GET requests made from within JavaScript and gives your Ajax application a cached response instead of fresh data.

I was running some tests on scorpion_server, using the JavaScript client in the example that I've packaged with the server, and I noticed that I could change the data stored in a resource when using IE6, but I couldn't get the value I had just set. This means my client-server pair is pretty much useless in IE6 (probably 7 too). But I thought of a solution. IE is just performing a test for exact match when checking the cache, so if any part of the URL is different, IE will actually perform the query like I want it to. So I added a timestamp URL parameter to all GET requests made by the client if the browser is IE, and I configured the server to ignore all URL parameters on GET requests. I wasn't using them anyway, perhaps someday I will, but I don't foresee a need because all I want is a super simple remote data store. (Who knows, YAGNI might just be my new mantra.)

Note that IE was the only browser (of those I tested) that exhibited this annoying behavior. Mozilla based browsers (Firefox, SeaMonkey, Flock), Opera, and Safari all behaved correctly and don't require the timestamp URL parameter hack.

Sunday, November 04, 2007

JavaScript client for my Scorpion Server

I've added a JavaScript client to the simple web server project: scorpion server. The client is able to set the username and password and make authenticated GET and POST requests to retrieve and modify resources on the server. Now that this portion is complete, I'm ready to rebuild my Ajax Wiki application to run on top of this simple web server.