jcla1.com

The Baby Emulator

Joseph Adams (whitegolem@gmail.com) — Thu, 07 Aug 2014 00:00:00 +0000

The Manchester Small-Scale Experimental Machine (SSEM) aka: "the Baby" lay the foundations for all modern computing, being the first stored-program computer, and built to demonstrate the viability of the Williams-Kilburn tube.

Despite being a breakthrough computer during its lifetime, it had a very limited instruction set and store. Presenting only 32 words of memory (that's 1024 bits) to store both program & data and only 7 instructions to program in, it was still Turing complete and ran its first program on June 21, 1948.

Seeing it as a valuable history lesson (and because the other emulator, a Java applet, is very cumbersome to use), I decided to write a nice (and fast) emulator that can be used from the command line. And here it is:

# It's on Github, of course: https://github.com/jcla1/gobaby
$ gobaby -t -l 27 -p=f examples/factor.asm
Execution took: 14.160445ms
Value at location #27: 131072

This trivial program (finding the largest factor of 2^18, by going through each number down, starting with 2^18-1) was the first to be run on the Baby and it took ~52 minutes to get the answer back then.

The emulator also supports printing out the memory contents as their equivalent assembly mnemonics (which is the default behaviour after execution), which makes it particularly useful when having to keep re-running programs, like this sucessive prime generator:

$ gobaby examples/primegen.asm | gobaby | gobaby -l 21 -p=f
Value at location #21: 5

Running this will give you the 3rd prime, and for all the ones following that, you've just got to keep on adding successive calls to gobaby.

So that's all there is to say about the emulator, to find out more about the Baby you should read David Sharp's Technical Introduction To Programming The Baby and of course its Wikipedia article.

Share and enjoy!

Experiencing Haskell

Joseph Adams (whitegolem@gmail.com) — Mon, 07 Jul 2014 00:00:00 +0000

There seems to be a growing interest in (purely) functional languages these days. I've looked at lots of different languages in the past (such as Go, Python, Ruby and others) but never a (purely) functional one. So recently I decided to dig into Haskell, with a view to discovering some of the fundamental differences. I wanted to share some of my findings with you in this post.

The Type System

This is one of the things Haskell is exceptionally good at: types! It has a very strong static type system which is based on Hindley-Milner type inference, that allows the compiler to infer most if not all of the types at compile time. So only in edge cases, or for optimization purposes is the programmer forced to give type annotations. Another great feature of Haskell's type system is its support for type variables. It's Haskell's equivalent of C++ templates, and a feature Go is still missing. They allow for expressive type signatures and contraint based generic typing, i.e.

sort :: Ord a => [a] -> [a]

This says that there's a function, called sort, that takes a list with values of any type, contrained by the typeclass Ord, and then returns a value of the same type. Here the Ord typeclass signifies that the values in the list have to be orderable. Now the a could have also been any other string. But the point is, that just from the type signature it's immediately obvious what transformation the function applies to its arguments.

Because of this expressiveness, there's a separate search engine that will search for functions with a given type signature (regardless of the variables used).

Algebraic Data Types

Another one of Haskell's interesting properties is its support for algebraic data types (ADTs). Think of an ADT as a type-safe version of a union in C. They let you describe data structures in terms of the possible values a variable of this type can have.

data Tree a = Empty
          | Leaf a
          | Node (Tree a) (Tree a)

In this example we describe what a variable of type Tree can be. A Tree is either Empty, or it's a Leaf, or it's a Node (where each node again, contains a tree itself). But that wasn't all, the a is a type variable for the ADT Tree, so this is the implementation of a generic tree-structure.

These ADTs combined with Haskell's powerful pattern matching, allows for articulate function definitions. For example, the following function takes another function and returns a new tree with the same structure as the original one, but with all the values replaced with the output of the function, when passed the original value:

In a real implementation you would consider the tree to be a functor, implement the Functor typeclass and use fmap instead, but that's a subject for another post.

mapTree :: (a -> b) -> Tree a -> Tree b
mapTree _ Empty = Empty
mapTree f (Leaf x) = Leaf (f x)
mapTree f (Node l r) = Node (mapTree f l) (mapTree f r)

As you see, we define the function in terms of the "shape" of its arguments. Mapping an empty tree, is an empty tree, mapping a Leaf, is also a Leaf, but with the function applied, etc.

Be careful though! Algebraic Data Types are not the solution to all problems. Sometimes they make implementing even the simplest data structures, like a doubly linked-list, near impossible.

Code Length & Execution Time

I was quite surprised about this, but from what I've found my Haskell programs are about the same length or even longer as equivalent versions in Python and I take a lot longer to write them, though that is probably due to my inexperience with Haskell's style of programming. But this chart from the Computer Language Benchmarks Game shows that I'm not alone:

Regarding execution time, my results don't match up with the above image. I found (when running a test with a simple log processing script) that the Haskell version performs significantly slower (>200x slower) than the Python version. From what I've heard though, Haskell's built-in list, which is used extensively in the example, is notoriously slow. I should've instead used a faster list data structure like: Data.Sequence.Seq.

Wrapping Up

So you can see, Haskell has quite a few interesting features, that when using them, it feels like you're learning to program for the first time again. Nevertheless you should absolutely try it out and have your mind reprogrammed!

Optimizing page timings for Google Analytics

Joseph Adams (whitegolem@gmail.com) — Sat, 07 Jun 2014 00:00:00 +0000

Many of you will have one of those recreational text pads where you jot down some notes you wanted to share with others. Like a blog. Sometimes, or in fact very often, you might want to track just how successful these quick notes are. Well you'll probably already know about Google Analytics.

It's one of those tools that everyone uses, it's free, it's easy, but it doesn't solve the problem completely. What I'm talking about is that Google Analytics' page timings are just plain bad when it comes to blogs.

Let's consider the situation : Someone visits your blog from a link aggregator, via an RSS reader or from lumpa-land. What will happen is, the visitor reads your article, hopefully likes it, and then leaves your site. Now this is the ideal situation for showcasing where Google Analytics is lacking somewhat.

This legitimite visitor, that may have even liked your blog post! will be recorded. Of course, you've got Google Analytics installed, that's why, but he will be saved as a bounce off of your site with zero pageview time. So the Google Analytics dashboard will tell you something like this:

Now this surely can't be right, it's impossible to read my blog posts in just 44 seconds, you may think. Well, often you're correct, it's Google Analytics' fault that the average session duration is so low. But instead of complaining about this, let's fix it!

1st attempt

Now let's consider why Google Analytics' page timings are incorrect. Well the root of the cause is that the way Google Analytics measures the time spent on a page, they calculate it by taking the difference in time between visiting different pages.

Ah, see here's the problem! On a blog, like in our scenario, a visitor will only be visiting the one blog post he navigated to in the first place. So there's no opportunity for Google Analytics to calculate a proper page timing! We can easily fix that.

Because Google Analytics will also consider events as pageviews when it comes to calculating the session duration. All we need to do is send off an event at the appropriate time... but when is the appropriate time?

This is a tough question and it's very hard to answer, so let's send one every ten seconds, just to make sure. Here's the adjusted Google Analytics snippet:

ga('create', 'UA-XXXXXXXX-X', 'example.com');
ga('send', 'pageview');

setTimeout((function(timing) {
    ga('send', 'event', 'time', 'log', timing);

    // Bind another timeout to run in 10s
    // to fire off the next event
    setTimeout(arguments.callee.bind(null, timing+10), 10000);
}).bind(null, 10), 10000);

Now what does that give us? Let's see:

Ah, that's what I like to see! We've got a 3,331.25% increase in the average session duration and the bounce rate dropped by 91.54%. So this solution seems to work, or does it? Because I think 26 mins is a bit too long for reading my relatively short blog posts.

Maybe we can find an explanation by adjusting our situation: Now, the visitor visits your page from a link aggregator and if I think about the way I use link aggregators, I'll exit them with 20 new tabs open.

See, there might be the cause! The visitor opens your blog post, but he may not actually view it until he's finished dealing with the other 19 tabs. This wasn't a problem when Google Analytics had to figure the average time out by itself, because most recorded timings were zero.

But now this has changed! Once a user opens the tab with your page loaded in it, we're telling Google Analytics that the visitor is on your site! So that's the next problem to fix!

2nd attempt

As I said, we need to find out, when the user actually starts viewing your blog post. But how are we going to do that? Well, this is the ideal situation to show off the relatively new Page Visibility API. It can tell us if the browser's frontmost tab contains our site, or not.

This should deliver a more accurate measure of when the page is actually viewed and read. But we can't just wait until a user views our page, i.e. the visibility event fires, because the user might just flit briefly over the tab, when navigating between them, for example.

ga('create', 'UA-XXXXXXXX-X', 'example.com');
track_ga();

function track_ga() {
    // We can be quite sure that the user _is_ on our page,
    // once he's there for more than 10s
    var waiting_time = 10000;

    // Just send the pageview if page visibility is not supported
    if (!get_hidden_prop()) {
        ga('send', 'pageview');
        return;
    }

    if (!is_hidden()) {
        // If the page is visible, setup a timeout to send
        // the pageview, that gets cancelled when the user
        // navigates away from the page too soon.
        var timeout = setTimeout(function() {
            ga('send', 'pageview');
            setTimeout(event_poller.bind(null, 0, waiting_time), waiting_time);
        }, waiting_time);

        document.addEventListener('visibilitychange', function() {
            document.removeEventListener('visibilitychange', arguments.callee);
            if (is_hidden()) clearTimeout(timeout);
        });
    } else {
        // Otherwise, wait until the page becomes visible again
        // and do everything over again.
        document.addEventListener('visibilitychange', function() {
            document.removeEventListener('visibilitychange', arguments.callee);
            track_ga();
        });
    }
}

// Same as our previous periodic GA event sender.
function event_poller(timing, wait) {
    ga('send', 'event', 'time', 'log', String(timing));
    setTimeout(event_poller.bind(null, timing + wait / 1000, wait), wait);
}

The functions get_hidden_prop() & is_hidden() are from, and can be found on HTML5Rocks.com get_hidden_prop() checks to see if the Page Visibility API is supported and for any vendor specific prefixes. is_hidden(), obviously, returns true/false depending on if the page is visible.

As you see, this version is a lot more complicated than our initial snippet, but will it improve our measurements?

Hmm, no not really. So it seems, that's the best we can get with this technique.

Final thoughts

Although our second attempt didn't yield the expected results, I can still confidently say, that the second snippet is better than the first. For one, it saves resources (network), due to the fact that we only start sending events once the user is actually active on your site, which is especially important if you've got a lot of mobile vistors reading your blog. And the page timings are definitely more accurate than with the default page timing mechanism. But with the second snippet we also get insight into when our pages are just prerendered.

So as you see, there doesn't seem to be a perfect solution, yet.

Improving Go cookie handling

Joseph Adams (whitegolem@gmail.com) — Mon, 19 May 2014 00:00:00 +0000

Many people argue, a lot of Go's popularity comes from developers moving to Go from Ruby & Python. And in a way, that may certainly be the case. Because Go, like Python famously, also has Batteries Included ®, one of the many reasons I love the language.

But like everything in the world, Go has its little idiosyncrasies. It's desperately trying to be RFC conformant with its cookie mechanism. Which is the reason, why certain cookies are just ignored by Go's cookiejar. In particular, Go just ignores cookies that contain characters like: \ (backslash) or " (double quotes).

Now there might have been good reasons to exclude these from the set of valid characters, i.e. it's obviously necessary to exclude a semi-colon, the cookie delimiter. But nevertheless, there may be situations in which Go will have to put up with servers sending cookie headers which are, acording to the RFC, just plain wrong and invalid.

And this is exactly the situation I encountered! For a Quizduell API library I was working on, I also had to talk to their 3rd-party server. And, you guessed it, they sent invalid cookies!

So after about 3 hours, the time it took me to debug why all API calls weren't authenticated, I came across an issue on the Go issue tracker, where a similar problem to mine was explained (basically, someone wanted to be able to have commas in cookies). And though the issue was resolved and a patch to allow commas in cookies was merged, the comments clearly indicated that the Go maintainers weren't happy with the idea of loosening the cookie character restrictions (although most browsers allow practically any character in a cookie).

Knowing that my problem was an acknowleged issue, I was still left with my problem, that I needed to fix. Looking for a solution, that was not copying the whole cookiejar implementation to a new package and just modifying this tiny function that checked wether a character is valid or not, I came up with the following solution:

cookie := response.Header.Get("Set-Cookie")
if cookie != "" {
    cookie = strings.Replace(cookie, "\\", "_", -1)
    response.Header.Set("Set-Cookie", cookie)
    c.Jar.SetCookies(request.URL, response.Cookies())
}

In the code example, c is the API client that has his own cookiejar, since the http client's jar is private.

Simply put, this snippet just takes the raw cookie string and replaces, in this case, the backslash with an underscore (a character known not to occur in cookies issued by the Quizduell servers), puts the cookie back into the http response and manually tells the API client's cookiejar to store the cookies of this particular response. Notice, you need your own cookiejar (c.Jar), since the one included in http clients is private.

The following snippet, that loads the cookies stored in our cookiejar and attaches them to a new http request, is a bit more complicated, because it needs to handle the case of there already being other cookies that were set previously on the request.

cookies := c.Jar.Cookies(request.URL)
if len(cookies) > 0 {
    for _, cookie := range cookies {
        s := cookie.Name + "=\"" + cookie.Value + "\""
        s = strings.Replace(s, "_", "\\", -1)

        if c := request.Header.Get("Cookie"); c != "" {
            request.Header.Set("Cookie", c+"; "+s)
        } else {
            request.Header.Set("Cookie", s)
        }
    }
}

But the idea is the same, load the cookie as a raw string, replace the underscores with the forbidden backslashes and manually attach the cookie as a header on the request.

And it works! Perfectly fine in fact. Though as you can guess, this is not the ideal solution for all possible situations. We quickly run into a problem if there's no character where we know that it won't occur in the cookie (in our case that was the underscore, incase you haven't noticed yet).

A possible workaround would be to base64 encode the relevant cookies and store the encoded version in the cookiejar.

100 Days of Github

Joseph Adams (whitegolem@gmail.com) — Tue, 08 Apr 2014 00:00:00 +0000

In recent times there's been a trend coming up, of having long commit streaks on Github. Be it for fun, motivational purposes or just out of boredom. In any case, I decided to also hop onto the train and try it for myself. In the following, I'd like to share my experience with you as to what helped me get through my first 100 day commit streak on Github.

An open source project

Github is a platform for open source projects, so to build up your commit streak it will obviously be a great help to either contribute regularly to or even maintain an open source project, or multiple projects. This has saved me a few times during my streak, because as Ryan Seys already noticed in his blog post about commit streaks:

Pull requests count for 1 contribution when you make them and another contribution if they are merged. If they don’t get merged, sorry, no second contribution.

So no need to directly send a pull request (PR) when you push, you can take your time knowing that your PR will also count as a contribution. Plus if you get promoted to contributor status in a repository, your merge commits (of other PRs) also count as contributions.

In my case, I contributed to a repository of algorithms, which you should go check out (pun definitely intended) and contribute to. This repository also has the nice effect that there's a near infinite number of algorithms to implement (combined with an equally large number of different languages), so you won't run out of things to implement and might even learn a new language.

A blog

Blogs are the medium of developers — it's through blogs that developer communicate and share their thoughts & ideas. So there is no reason why blogs (specifically your blog) should be excluded from your achievements as a developer, your commit streak.

Be it writing posts or just drafting them, adding microdata markup (which is what I did) or completely redesigning them, you will definitely profit from maintaining a blog as a developer and so will your commit streak. As you see a blog is key, not just for your commit streak, but for your existence as a developer.

Programmer's challenges

This is a topic for another post, but for now what I recommend you do is: Keep a list of interesting challenges to solve, they needn't be programming related, but most of them will be (hopefully). To get you started on your list I can highly recommend Jason Rudolph's Programming Achievements, Matt Might's course work assignments (especially the scripting language assignments) and of course the exercises from The Structure and Interpretation of Computer Programs (SICP). Of course, for this to be beneficial to your commit streak it needs to be in a Github repository, so no solving Project Euler problems ;-) !

Final thoughts

Beyond all this, the most important thing is you. Well not really, it's your ideas. They are what make your streaks possible in the first place, so they'd better be good ones. But I digress.

Just make sure you have fun and get over the first two weeks, those are the hardest. So with this in mind, onwards and upwards! 200 days of Github, hopefully see you then.

FOSDEM 2014 lightning talk video

Joseph Adams (whitegolem@gmail.com) — Sat, 01 Mar 2014 00:00:00 +0000

Last month I was in Brussels with @qmacro for FOSDEM 2014. At the time I was working on a little side project: Gisp, a Lisp to Go compiler. On the Sunday I decided it was time to present Gisp to the world, so I gave a lightning talk in the Go room.

In the talk I showcase the generated Go code Gisp produces for some simple Lisp programs, as well as the Go AST generating REPL that's included in Gisp. Also great thanks to Andrew Gerrand and Brad Fitzpatrick for organizing the Go dev-room at this year's FOSDEM!

I'm up first in the video, but the other talks are definitely worth while too.

Original video source

Personal Annual Report 2013

Joseph Adams (whitegolem@gmail.com) — Thu, 02 Jan 2014 00:00:00 +0000

For those of you just looking for the actual report, you can find it right here.

Numbers are great! I think we can all agree on that. But I think they have even greater value if they give you insight into something.

Astounded by the endless reports from Nicolas Feltron, the packed article from Stephen Wolfram and especially the eye-opening reports of Jehiah Czebotar, I decided to publish my own Personal Annual Report.

Even though my first tracking attempts were as late as September, I'm quite pleased with the results. Following Jehiah's example I've been mostly tracking computer activity (especially web-usage).

The things I'm tracking include: currently active application, open applications, URL of the frontmost tab, number of open tabs, etc. But beyond that I'm hoping that for this year's (2014) report I can include more things.

Technical Details

Although the actual report looks fine, I'm not happy with the data collection mechanism. Basically every 15 seconds an activity logging script is run, via cron, which collects data and then writes it to a file. Though there are a number of problems with this implementation. For example, when trying to aggregate the data (in Python) there are numerous occasions where a log file is corrupted and I have no idea why this is. Also running these script (currently written in a mix of Shell and Applescript) is quite slow.

So to solve these problems I've set myself the task to rewrite (and extend) the data collection/aggregation mechanisms.

Feedback and ideas appreciated

Share and enjoy

Crawling HackerNews

Joseph Adams (whitegolem@gmail.com) — Mon, 13 May 2013 00:00:00 +0000

With the new HN Search there's also a great REST API provided. You should go check it out!

It's been a while now since I started developing HN2JSON and I am actually still quite happy with that achievement. If you've not heard of HN2JSON yet, it's basically a simple Ruby interface to HackerNews.

But my dream was always to collect a complete copy of the HackerNews database! This wasn't possible with HN2JSON, because it is too slow and I didn't want to hammer HN with millions of requests. What I was left with was the HNSearch API

This was going to be a hard battle to win. And it was I can tell you. I won't trouble you with the details, but I had coded and thought for a whole week until I finally figured it out.

The battle begins

To understand why I thought this was such a hard problem to solve, you must know about the most complained about "feature" of the HN API. It restricts your search and you can't retrieve HN posts by their natural id. Instead the db holding the data assigns a signed_id to every db entry. You can get a post by signed_id, but they are near-impossible to guess.

So confronted with this problem I had thought about writing a distributed system (that would guess entries based on pre-crawled data) and also just brute-force guessing all possible signed_ids. Looking back, these approaches are ludicrous compared to the simple solution that I found in the end.

For you to understand this simple solution, you must know that, when calling the API URL you can pass in filters based on various available parameters. Let me show you an example:

http://api.thriftdb.com/api.hnsearch.com/items/_search? \
  filter[fields][username]=pg

This URL will give you access to 10 posts and comments on HN where the submitter's username is "pg". Of course 10 entries isn't very helpful, apart from the fact that you can up that limit to 100, but what I realized was that there is also a filter on the submission date. This filter takes a date range:

http://api.thriftdb.com/api.hnsearch.com/items/_search? \
  filter[fields][create_ts]=[2013-01-01T00:00:00Z + TO + *]

This URL will return you 10 posts and comments from start of 2013 upto anytime. Might not look helpful at first, but when we combine this with a sorting filter we are able to collect all items in the database.

Now let me explain that last bit again.

Let's say, we sort all entries there are by date in ascending order. When we call the corresponding URL, we are presented with the first 10(0) entries ever on HN. And now let's say that we take the last entry we get from the API and set a date-range filter going from that last entry to anytime. Now we get the next 10(0) entries from HN. If you keep updating the URL, you are able to collect the complete HN database. An example URL for that would be something like:

http://api.thriftdb.com/api.hnsearch.com/items/_search? \
  sortby=create_ts asc
  &limit=100
  &filter[fields][create_ts]=[<PUT LAST ENTRY'S DATE HERE> + TO + *]

Conclusion

So there you are, simple after all. Coding this mechanism up and making it fault-tolerant didn't even take me 2 hours. I plan to collect the whole db in the very near future and then to open-source it so that you and everyone else can benefit from it.

Javascript and MapReduce

Joseph Adams (whitegolem@gmail.com) — Sat, 11 May 2013 00:00:00 +0000

Don't we all love feeling like big data scientists, data miners or <insert data buzzword here>?
Well here's a way to get that feeling in your everyday life:

Recently I discovered the map, reduce, filter functions in Javascript. They are absolutely awesome! To demonstrate their abilities we'll be building a wordcount program, using MapReduce, in Javascript, all chained.

Digging Data

As a datasource we'll use HackerNews' big RSS feed and since we can treat RSS as plain HTML, it's very simple to get started:

var titles = document.getElementsByTagName('title');

This is simply getting all the titles from the RSS feed. But we have a problem, getElementsByTagName returns a NodeList and not an Array, which hasn't got the cool MapReduce functionallity we want.

So we have to convert it to an array, and whilst doing so, we can also grab all the titles as strings, instead of HTML Nodes

var titles = Array.prototype.slice.call(document.getElementsByTagName('title'))
  .map(function(node) {
    return node.innerText;
  });

Simple isn't it? And now we have a list of all the titles on HackerNews (HN).

Introducing .map(callback [, thisArg])

map is a nice small function that takes 2 parameters, but usually only the first one is specified.

As you can tell from the above code, the first argument is a callback function that is to be called over every element in the array, the second one is just to specify a value for this during the function execution. More important are the parameters passed to the callback function. They are: the array element itself, its index and the whole array/context. So the callback function's signature would look something like this:

var callback = function(element, index, context) { /* omitted */ }

Unwanted punctuation & Individual words

As you may have noticed, the HN titles don't just contain letters, but they also carry numbers and punctuation. These are unwanted in our final wordcount, so we remove them. This can be done in the same map call. Isn't Javascript a great language:

var words = Array.prototype.slice.call(document.getElementsByTagName('title'))
  .map(function(node) {
    return node.innerText.toLowerCase().match(/([a-z]+)/g);
  });

If you're coding along, you'll have noticed this had a side-effect: It automatically gave us all the individual words, thanks to the RegEx, which returns all matches in an array.

Even with this helpful side-effect, we are presented with a further problem. The words array is 2d, which means we need to flatten it. Good job we've got reduce:

var words = Array.prototype.slice.call(document.getElementsByTagName('title'))
  .map(function(node) {
    return node.innerText.toLowerCase().match(/([a-z]+)/g);
  })
  .reduce(function(last, now) {
    return last.concat(now);
  }, []);

Introducing .reduce(callback [, initialValue])

As map, reduce takes 2 arguments. The first one is a callback function again, which is to be called on every element in the array. The initialValue parameter will be easier to understand once you see the callback's signature:

var callback = function(previousValue, currentValue, index, context) { /* omitted */ }

Note: We wouldn't actually end up with a ginormous string, because strings don't have a push method. It would just error out.

As you can see, the first parameter passed to the callback is the previous value. If you wanted to sum up an array of numbers, this would be not problem. But in our case we want an array returned, so we specify an inital value for the previousValue. If we didn't do this we would end up with a ginormous string.

Counting words

For this task, we can again use reduce, but before we start let me show you what the result will look like:

[['the', 'on', 'news', 'hacker', ...], [50, 66, 20, 19, ...]]

It'll be a 2d array again. The index of a word will correspond to the number of its occurrences. As an example the word news has the index 2, in the first array, so its score is at index 2 in the second array, in this case 20.

To code this, we'll use reduce again and as an inital value we'll specify: [[], []]

var scores = Array.prototype.slice.call(document.getElementsByTagName('title'))
  .map(function(node) {
    return node.innerText.toLowerCase().match(/([a-z]+)/g);
  })
  .reduce(function(last, now) {
    now.forEach(function(word) {
      last.push(word);
    });

    return last;
  }, [])
  .reduce(function(last, now) {
    var index = last[0].indexOf(now);

    if (index === -1) {
      last[0].push(now);
      last[1].push(1);
    } else {
      last[1][index] += 1;
    }

    return last;
  }, [[], []]);

Zipping up the arrays

We're nearly done with collecting the data. All we need to do now is to combine the 2 arrays into one. The format this final array will be in is as follows:

[['the', 50], ['on', 66], ['news', 20], ['hacker', 19], ...]

For this we'll use reduce again and utilize the 4th parameter passed to the callback:

var scores = Array.prototype.slice.call(document.getElementsByTagName('title'))
  .map(function(node) {
    return node.innerText.toLowerCase().match(/([a-z]+)/g);
  })
  .reduce(function(last, now) {
    now.forEach(function(word) {
      last.push(word);
    });

    return last;
  }, [])
  .reduce(function(last, now) {
    var index = last[0].indexOf(now);

    if (index === -1) {
      last[0].push(now);
      last[1].push(1);
    } else {
      last[1][index] += 1;
    }

    return last;
  }, [[], []])
  .reduce(function(last, now, index, context) {
    var zip = [];
    last.forEach(function(word, i) {
      zip.push([word, context[1][i]])
    });
    return zip;
  });

This works, because the callback is only called once, so we can zip up the words with their scores.

Now we have all our data collected. We could stop here, but humans love visualisations!

Visualisation

We're going to use D3.js to visualize our data. And to be specific we'll create a force-directed chart.

I won't explain how the visualisation works, because that isn't the subject of this post, but to find out you can always view source!

Browser ABC

Joseph Adams (whitegolem@gmail.com) — Thu, 09 May 2013 00:00:00 +0000

Deeply inspired by Tim Bray's "Letter Sweep" and my dad's "My Browser A-Z". Notice, there are 3 Google subdomains, but no google.com. Is the omnibox killing the Google homepage?

A: api.thriftdb.com/api.hnsearch.com/users/
Even though it's an API I visit it regularly. Probably also got an extra push due to my blog post about it.

B: bost.ocks.org/mike/
For the latest and probably greatest visualisations on the net.

C: closure-compiler.appspot.com
I can't even remember the last time I visited this, I'd grade it questionable.

D: datamapper.org/docs/
My ORM of choice when working with Sinatra.

E: en.wikipedia.org/wiki/Category:Fundamental_categories
I visit this page regularly to explore Wikipedia, it's very helpful for discovering new interesting topics.

F: file://localhost/Users/joseph/Desktop/...
All tinkering starts here!

G: github.com
Great! Nothing else.

H: hnsearch.com/api
Not sure if it's really useful or completely useless.

I: isup.me
Slow internet connection makes me think sites are down :-(

J: jcla1.com
My personal blog.

K: kuler.adobe.com
I know it's flash, but still good for finding nice color schemes.

L: localhost:8000
You can't imagine with how many different ports I found this.

M: mail.google.com
Apparently not dead and still useful.

N: news.ycombinator.com
Makes >30% of my time on the web. GO HN!

O: ozone3d.net/tutorials/mandelbrot_set_p3.php
Really great introduction to fractals. A must-read!

P: plus.google.com
Is not being mainstream, too mainstream?

Q: qvc.de
Not recommended for buying tech, definitely.

R: repl.it
Refrence implementation of web and esoteric languages. Helped a lot when building a Brainfuck interpreter.

S: stuenings.de
Resource for a paper I have written recently.

T: translate.google.com
What was that word again?

U: underscorejs.org
New second best friend? (d3 is top!)

V: vivid.chengyichao.info
Useful for coding along with The Little Schemer book that I'm reading at the moment.

W: wolframalpha.com
To answer all the questions Google can't.

X: xkcd.com
Well, of course. to quote Tim Bray.

Y: youtube.com
For my weekly dose of physics and interesting factoids.

Z: zachholman.com
Great presentations and more Github!

On Ulam spirals and matrix generation

Joseph Adams (whitegolem@gmail.com) — Fri, 28 Dec 2012 00:00:00 +0000

One of my Christmas presents was the fabulous book: Getting Started with D3 by Mike Dewar.

I recommend reading it, if you haven't already! But anyway, after I had read it, I was looking for some data I could visualize. The book suggested the New York transit dataset, but I didn't want to have to clean it just to play with d3. So after a bit of thinking and doing distractive things, I remembered having read about Ulam spirals quite a while back.

The Idea

The decision was made, what was missing was the data. I wanted to refresh my knowledge about Ulam spirals, so I loaded up Wikipedia and copied the spirals onto a whiteboard.

I gave myself the task of finding an algorithm that would generate just number spirals, given a dimension and [x, y] coordinates to return a value. I also gave this task to my father @qmacro, so we would compete on finding an algorithm.

The source of both finished algorithms is here:

Mine: Github
My father's: Github

and I ran them through JSPerf here.

Finding an algorithm

My approach was, I would say, a bit clumsy. It took quite a while, until I actually got anywhere, but when I finally did something it was an analytical approach.

My idea was to just stare at the spiral for a long time until something magically flew to my mind. And, as opposed to the expectations you may have to the outcome of this, I suddenly had an idea. During the staring time I made some interesting observations, as can be seen in the next section.

Observations

Here is a sample 4x4 & 5x5 matrix so you can follow along:


                    ___________________   ________________________
                    |16 | 15 | 14 | 13|   |17 | 16 | 15 | 14 | 13|
                    |-----------------|   |----------------------|
                    |5  |  4 |  3 | 12|   |18 |  5 |  4 |  3 | 12|
                    |-----------------|   |----------------------|
                    |6  |  1 |  2 | 11|   |19 |  6 |  1 |  2 | 11|
                    |-----------------|   |----------------------|
                    |7  |  8 |  9 | 10|   |20 |  7 |  8 |  9 | 10|
                    -------------------   |----------------------|
                                          |21 | 22 | 23 | 24 | 25|
                                          ------------------------

Number spirals are just some matrices that follow a certain pattern.
The dimension squared is the highest value of the matrix.
The location of the dimension squared is dependent on parity: If odd, in the bottom right, if even, in the top left, corner of the matrix.
You can work out the numbers on the outer rim of the matrix with simple formulas.
If you compare the two matrices, how are they different? The 5x5 matrix has one "frame" more than the 4x4 matrix.
You could imagine it just being stuck on one of the sides, if you need higher dimensions. Again the side on which the last frame was "stuck" is determined by the parity of the dimension.
(In this case onto the left-bottom side)

So from these observations you could draw some conclusions, which I did!

Conclusions

Based on the previous observations I had the thought that one could just "drop a frame" if the desired number doesn't lie within the area of that frame.

So I wrote a function that worked out if "dropping a frame" was possible. It takes coords of the location of the number you would like to have, and the dimension of the matrix.

function check_if_drop_frame(coords, d){
  if (d % 2 === 0){
    if (coords[0] != d && coords[1] != 1){
      return true;
    } else {
      return false;
    }
  } else if (d % 2 === 1) {
    if (coords[0] != 1 && coords[1] != d){
      return true;
    } else {
      return false;
    }
  }
}

In simple terms, the function checks the parity of the dimension and then checks if the coords
[x, y] lie within the last frame.

The idea is just to make the matrix smaller so, the original coords lie on the outermost frames, from which we know from the observations that they are easy values to work out.

But in order to do so, we can't simply decrement the number of dimensions and leave the coords the same, we have to translate them.

function translate(coords, d_old){
  if (d_old % 2 === 0){
    coords[1] -= 1;
  } else if (d_old % 2 === 1) {
    coords[0] -= 1;
  }

  return coords;
}

This function again checks the parity and from that is knows which coord it needs to decrement, either x or y. So putting those two thing together we just drop frames/dimension as long as that is possible:

while (check_if_drop_frame(coords, d)){
    coords = translate(coords, d);
    d -= 1;
}

Once that loop has finished, it is guaranteed that the coords describe a number on one of the outermost frames.

From there one only needs to find formulas to work out the value. Luckily I have already done that for you, they are a bit messy but they work. Wrapped in a function they look roughly like this:

function work_out_value(coords, d){
  var val = -1;

  var x = coords[0],
      y = coords[1];

  if (d % 2 === 0){
    if (coords[1] === 1){
      val = Math.pow(d, 2) - (coords[0] - 1)
    } else if (coords[1] === d){
      val = Math.pow(d, 2) - 2 * (d - 1) - (d - 2) + (x - 1) - 1
    } else if (coords[0] === 1){
      val = Math.pow(d, 2) - 3 * (d - 1) - (d - 2) + (y - 2)
    } else if (coords[0] === d){
      val = Math.pow(d, 2) - d - (y - 2)
    }
  } else if (d % 2 === 1){
    if (coords[1] === 1){
      val = Math.pow(d, 2) - 2 * (d - 1) - (x - 1)
    } else if (coords[1] === d){
      val = Math.pow(d, 2) - (d - 1) + (x - 1)
    } else if (coords[0] === 1){
      val = Math.pow(d, 2) - 2 * (d - 1) + (y - 1)
    } else if (coords[0] === d){
      val = Math.pow(d, 2) - 3 * (d - 1) - (y - 1)
    }
  }
  return val;
}

Basically, the function works out which is the outermost frame and then takes the dimension squared, whose position we can foresee, and works backward towards to desired coords.

With a bit of boundary checking you can see the complete source here.

Hooking it up with d3!

After I had figured out all of this it was night time, and I wasn't really bothered with good code style at that time of day, but since I wasn't satisfied yet I hooked it up with d3, and made my own ulam spiral.

This demo generates a 2d array and then loops though each element and checks if it is a prime. If it is it sets the fill-color to red, else just leaves it gray. Then it just gets d3 to draw the matrix.

Getting to work with CodeMirror

Joseph Adams (whitegolem@gmail.com) — Fri, 15 Jun 2012 00:00:00 +0000

Recently on Hackernews I found an interesting link. It pointed to RubyFiddle. The site is really nice, but my favourite bit was the editor.

By then I had already guessed they hadn't done it themselves so I investigated (looked in the source..) and found they're using CodeMirror. I had heared about CodeMirror before but I never really liked online IDEs. But this editor really caught my eye so I decided to try the most basic setup.

Turns out, it's really simple! It's just 1, in words one, line of Javascript! (Plus basic HTML)

CodeMirror(document.getElementById('container'));

That is all that is needed. Thanks to Marijn Haverbeke for such a great online editor.

Below is a demo of it:

Web Audio API overview (Part 2 of 2)

Joseph Adams (whitegolem@gmail.com) — Fri, 06 Apr 2012 00:00:00 +0000

In the first part, we looked at the basics of the Web Audio API and using it to visualize music. In this second part I'll be showing you how to implement a LowPass Filter and gain control in our example from the first part. We'll add some colour to it too.

Let's start with the easiest bit, the color. To change the color of the bars, all you need to do is change ctx.fillStyle = "white"; to whatever color. If you're like me, you won't like single color bars. I think those bars deserve a nice rainbow gradient. This is done very easily:

gradient = ctx.createLinearGradient(0, 0, 0, canvas.height);

gradient.addColorStop(0, "rgba(255, 0, 0, 1)");
gradient.addColorStop(0.15, "rgba(255, 255, 0, 1)");
gradient.addColorStop(0.3, "rgba(0, 255, 0, 1)");
gradient.addColorStop(0.5, "rgba(0, 255, 255, 1)");
gradient.addColorStop(0.65, "rgba(0, 0, 255, 1)");
gradient.addColorStop(0.8, "rgba(255, 0, 255, 1)");
gradient.addColorStop(1, "rgba(255, 0, 0, 1)");

ctx.fillStyle = gradient;

And that's it now we've a nice gradient.

Next up, gain. You can create one just like an analyser:

source = audioContext.createMediaElementSource(audioElement);
gain = audioContext.createGainNode();
analyser = audioContext.createAnalyser();

source.connect(gain);
gain.connect(analyser);
analyser.connect(audioContext.destination);

Now we have a GainNode, but no way for the user to interact with it.

To give the user some control over the gain, we'll have a well-known input.


<p>Gain: <input id="gain" type="range" value="1" min="0" max="1" step="0.01"></p>

The reason it has a max value of 1 is that the GainNode by default sets this value (It is possible to change this). Of course an input alone doesn't do anything. We'll add an event listener to it:

document.getElementById("gain").addEventListener('change',
  function(e){
  gain.gain.value = this.value
});

This snippet sets gain.gain.value to the value on the input. And that is all there is to it.

Adding a LowPass filter is the same procedure.

Add an input box.
Create the LowPass filter in Javascript.
Add a binding between input and LowPass filter

Here is the source code:


<p>LowPass: <input id="lowpass" type="range" value="5000" min="0" max="5000" step="10"></p>

<script>
function init() {
  source = audioContext.createMediaElementSource(audioElement);
  gain = audioContext.createGainNode();
  filter = audioContext.createLowPass2Filter();
  filter.cutoff.value = 22050;
  analyser = audioContext.createAnalyser();
  source.connect(gain);
  gain.connect(filter);
  filter.connect(analyser);
  analyser.connect(audioContext.destination);
  draw();
}


document.getElementById("lowpass").addEventListener('change',
  function(e){
    filter.cutoff.value = this.value
});


</script>

In this short series I have demonstrated how easy it is to work with the Web Audio API. You now know how to create Audio nodes and change their effect on audio.

Web Audio API overview (Part 1 of 2)

Joseph Adams (whitegolem@gmail.com) — Sun, 11 Mar 2012 00:00:00 +0000

In the next 2 blog posts I'll be showing you some essential features of the Web Audio API. You can find its specification here. The Web Audio API provides nearly all the functionality of a normal synthesizer, one of the reasons why it is so powerful.

Anyway let's get going. I'm going to talk you through the source of a little audio visualizer in this post and the next one. Here is a demo of the final product. In this first post I'll concentrate on a simplified version.

Warning: The code shown in this blog post is outdated and will only work in older version of Chrome.

Now let's look at the HTML body structure. In this case it is very simple, it has a main #container, that (as the name says) contains a canvas and an audio element. After that there are a couple of script tags.


<body>
  <div id="container">
    <canvas height="200" width="500" id="fft"></canvas>
    <audio id="audio" src="IO2010.mp3" preload controls></audio>
  </div>
  <script>
  // requestAnim shim layer by Paul Irish
    window.requestAnimFrame = (function(){
      return  window.requestAnimationFrame       ||
              window.webkitRequestAnimationFrame ||
              window.mozRequestAnimationFrame    ||
              window.oRequestAnimationFrame      ||
              window.msRequestAnimationFrame     ||
              function(callback, element){
                window.setTimeout(callback, 1000 / 60);
              };
    })();
  </script>
  <script>
    // Some Javascript
  </script>
</body>

You may already know what the contents of the first script tag are for. It is a shim by Paul Irish that makes it easier to use the requestAnimationFrame() (To get more info read his blog post). I'm not going to go further into how it works, but all it does really is make user friendlier animations.

The important parts for this blog post are the contents of the 2nd script tag:

// The audio element
audioElement = document.getElementById('audio');

// The canvas, its context and fillstyle
canvas = document.getElementById('fft');
ctx = canvas.getContext('2d');
ctx.fillStyle = "white";

// Create new Audio Context and an audio analyzer
audioContext = new webkitAudioContext();
analyser = audioContext.createAnalyser();

// Canvas' height and width
CANVAS_HEIGHT = canvas.height;
CANVAS_WIDTH = canvas.width;
// We'll need the offset later
OFFSET = 100;
// Spacing between the individual bars
SPACING = 10;
// Initialize and start drawing
// when the audio starts playing
window.onload = init;
audioElement.addEventListener('play', draw);

function init() {
  // Take input from audioElement
  source = audioContext.createMediaElementSource(audioElement);
  // Connect the stream to an analyzer
  source.connect(analyser);
  // Connect the analyzer to the speakers
  analyser.connect(audioContext.destination);
  // Start the animation
  draw();
}

function draw() {
  // See http://paulirish.com/2011/requestanimationframe-for-smart-animating/
  requestAnimFrame(draw, canvas);
  // New typed array for the raw frequency data
  freqData = new Uint8Array(analyser.frequencyBinCount);
  // Put the raw frequency into the newly created array
  analyser.getByteFrequencyData(freqData);
  // Clear the canvas
  ctx.clearRect(0, 0, CANVAS_WIDTH, CANVAS_HEIGHT);
  // This loop draws all the bars
  for (var i = 0; i < freqData.length - OFFSET; i++) {
    // Work out the hight of the current bar
    // by getting the current frequency
    var magnitude = freqData[i + OFFSET];
    // Draw a bar from the bottom up (cause for the "-magnitude")
    ctx.fillRect(i * SPACING, CANVAS_HEIGHT, SPACING / 2, -magnitude);
  };
}

The webkitAudioContext() created, is a context in which the audio interactions take place. Similar to the ones mentioned in my previous post.

You could divide it into 2 main parts:

The setup
The animation's draw function

The setup

sets up all the variables (except "freqData") and provides information on how each bar drawn should look. The init function connects the source of the audio (the audio element) with the analyzer and connects the analyzer with the destination. i.e. the speakers

You can imagine connecting up source, analyzer and destination as taking a few plugs and plugging them in some hardware. The only difference is that this is virtual!

The animation's draw() function

takes care of drawing the bars.

But what does that mean exactly?

Call the requestAnimFrame function to restart the animation at just the right time.
Create a typed array (called freqData) for holding the individual frequencies. The parameter passed at creation is the size of the array (In this case 1024 items).
Call a function on the analyzer to put the frequencies in freqData (it doesn't return anything).
Simply clear the canvas.
Loop through all the frequency data (except that in the offset) and each time:
- Get the current frequency
- Draw a bar that is as high as the frequency. The magnitude has to be negative here so that the bar is drawn in the correct direction.
Lather. Rinse. Repeat.

Nearly everything that has to do with the Web Audio API inherits from an object called AudioNode, which contains some basic structure for working with audio. For example the analyzer we are using here is also inherited from AudioNode. Other examples of Audio Nodes are BiquadFilter, LowPassFilter, AudioGainNode and many more. I will be covering some of them in part 2 of this mini series.

Exploring the V8 JS engine (Part 2 of 2)

Joseph Adams (whitegolem@gmail.com) — Sun, 08 Jan 2012 00:00:00 +0000

This is the second part of a 2 part series giving a simple technical overview of the V8 Javascript engine.

In the first part of this series I showed you how simple it is to get started with the V8 Javascript engine.

In this second part we'll look into sharing objects and variables in a V8 program, as well as digging into the source of the popular (on V8 based) Node.js, to see how variable/object sharing is implemented in a real world example.

In V8 there are 2 types shared variables:

Variables where you can change the value
Variables with an unchangeable value

Warning: You are able to change the value of and unchangeable variable, but it will only be reflected in the Javascript (not the C++).

For variables with an unchangeable value you only have to set the value.

This is a bit trickier with shared variables that are changeable, for those you need "Getter" & "Setter" functions, that get called when the variable is accessed.

I will cover both of those types below and then show them implemented in the Node.js source.

Variables with an unchangeable value

Based off our previous example (part 1 of this series), sharing an unchangeable variable is pretty simple. All you have to do is use the Set(...) function to set the variable's value.

This program will create the variable pid in the Javascript's context:

#include <v8.h>
#include <unistd.h>
// Include the header that contains the "getpid()" function

using namespace v8;

int main(int argc, char* argv[]) {

  // Create a stack-allocated handle scope.
  HandleScope handle_scope;

  Handle<ObjectTemplate> global_templ = ObjectTemplate::New();
  global_templ->Set(String::New("pid"), Integer::New(getpid()));

  // Create a new context.
  Persistent<Context> context = Context::New(NULL, global_templ);

  // Enter the created context for compiling and
  // running the hello world script.
  context->Enter();

  // Create a string containing the JavaScript source code.
  Handle<String> source = String::New("pid;");

  // Compile the source code.
  Handle<Script> script = Script::Compile(source);

  // Run the script to get the result.
  Handle<Value> result = script->Run();

  // Dispose the persistent context.
  context.Dispose();

  // Convert the result to an ASCII string and print it.
  String::AsciiValue ascii(result);
  printf("%s\n", *ascii);
  return 0;
  }

If you save it in a file called variable_share_unchangeable.cc (in ~/dev/v8/) you can compile and run it with:

$ g++ -m32 -Iinclude libv8.a -lpthread variable_share_unchangeable.cc \
  -o variable_share_unchangeable

For more information on running and compiling see the first post

When this program is run it will print out its own pid. The interesting thing about that is now though, that the pid is accessed by the Javascript not the C++.

Variables with a changeable value

Defining variables that a C++ and Javascript program share, where you are able to change the value are a bit trickier.

As I said before, you need to define a "Getter" and "Setter" for each variable you want to share. Here is a program that demonstrates the use of shared changeable variables:

#include <v8.h>

using namespace v8;

// This is the variable we are going to share
int x = 15;

// Gets called when the value of x is requested
Handle<Value> XGetter(Local<String> property, const AccessorInfo& info) {
  // Create a new Javascript int from the
  // current value of "x"
  return Integer::New(x);
}

// Gets called when "x" is set to a new value
void XSetter(Local<String> property, Local<Value> value, const AccessorInfo& info) {
  // Change the value of the "x" in the C++
  // to a 32-Bit representation of the value passed
  x = value->Int32Value();
}

void CompileAndPrint(const Handle<String> source) {
  // Compile the source code.
  Handle<Script> script = Script::Compile(source);

  // Run the script to get the result.
  Handle<Value> result = script->Run();

  // Convert the result to an ASCII string and print it.
  String::AsciiValue ascii(result);
  printf("%s\n", *ascii);
}

int main(int argc, char* argv[]) {

  // Create a stack-allocated handle scope.
  HandleScope handle_scope;

  // Create a new ObjectTemplate
  Handle<ObjectTemplate> global_templ = ObjectTemplate::New();

  // Set the XGetter and XSetter function
  // to be called when the value of "x" is requested
  // or "x" is set to a different value.
  global_templ->SetAccessor(String::New("x"), XGetter, XSetter);

  // Create a new context.
  Persistent<Context> context = Context::New(NULL, global_templ);

  // Enter the created context for compiling and
  // running the hello world script.
  context->Enter();

  // x is still equal to 15 here
  Handle<String> s1 = String::New("x;");
  CompileAndPrint(s1);

  // Here we change x to 250
  Handle<String>>s2 = String::New("x = 250;");
  CompileAndPrint(s2);

  // We print out x in C++ to see if the value has changed
  printf("%d\n", x);

  // Dispose the persistent context.
  context.Dispose();

  return 0;
}

It seems a lot of code for just one variable, though as you will see in the Node.js source, once you use multiple variables you will hardly notice the surroundings.

Node.js source

Here we'll dig into where and how Node.js uses these shared variables.

Specifically I will use the process object as an example.

It is not necessary that you fully understand how the internals of Node.js work, all you need to know is that there is a Start function right at the end of this file, that calls another function, which then calls another function that sets up the process object. The name of this function is SetupProcessObject.

This function sets up, among others, the pid variable that we had as an example. In the case of Node.js, this variable is unchangeable.

An example of a changeable variable in Node.js is the process.title, it is the name or the title of the current program.

Try it out by doing:

$ node
> process.title = "this_is_a_test";
'this_is_a_test'

You won't notice a difference in node but if you look up the process name of node in Activity Monitor or Task Manager, etc. it will be "this_is_a_test" (without quotes):

End

That was a short overview of shared variables in the V8 Javascript engine.

I didn't cover shared functions, because the post was that long already, but if you would like me to write a blog post about them too, just email me at whitegolem@gmail.com.

Exploring the V8 JS engine (Part 1 of 2)

Joseph Adams (whitegolem@gmail.com) — Sat, 07 Jan 2012 00:00:00 +0000

This is the first part of a 2 part series giving a simple technical overview of the V8 Javascript engine.

First of all some basic facts:

Developed and maintained by Google
Javascript engine behind Google Chrome
Also powers Node.js?
Really Fast!

V8 is written in C++ so you should have basic understanding of OOP and some C/C++ knowledge wouldn't do bad too.

The engine (V8) executes Javascript in so called contexts, these are sandboxed and you can create multiple contexts in one V8 virtual machine (engine).

This has the advantage that if you run 2 (or more) Javascript programs you don't have to worry about namespacing. The creation of these contexts is not as memory hungry as you may think, so don't worry about that.

One of the great features of V8 is sharing C++ functions, objects and variables with Javascript, which we will cover in the second part of this series. To use V8 you write a C++ program that uses the V8 libs (to set up the contexts, scopes, templates) and then executes a string which is your Javascript program. You will understand how this works later in this post.

I'll take you through setting up the V8 lib and a simple Hello World! program.

Downloading and Building

I'm using a Mac to build the library, but Windows is supported too.

To build V8 you will need:

Subversion 1.4 or higher
Python 2.4 or higher
SCons 1.0.0 or higher
GNU Compiler (GCC) 4.x.x

To download the source type this into you terminal:

$ mkdir ~/dev/
$ cd ~/dev/
$ svn checkout http://v8.googlecode.com/svn/trunk/ v8-read-only
$ mv v8-read-only v8

Now you have the source you can build the V8 library and header file:

$ scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
...

Simple isn't it? Now once the build has finished you should be able to see the libv8.a file. This file is important for compiling the C++ program that uses the V8 library.

Your first V8 program

Now that you have all the parts needed for compiling a C++ program that uses the V8 classes, let's get to the interesting part.

Here is a simple C++ program that runs the Javascript: "Hello World!". Obviously, this is not a very spectacular Javascript program, but it should do to our needs. The comments explain what a each part does:

#include 

using namespace v8;

int main(int argc, char* argv[]) {

  // Create a new context.
  Persistent<Context> context = Context::New();

  // Enter the created context for compiling and
  // running the hello world program.
  context->Enter();

  // Create a stack-allocated handle scope.
  HandleScope handle_scope;

  // Create a string containing the JavaScript code
  // to execute (notice the quotation).
  Handle<String> source = String::New(" 'Hello World!'; ");

  // Compile the Javascript code.
  Handle<Script> script = Script::Compile(source);

  // Run the script to get the result.
  Handle<Value> result = script->Run();

  // Get rid of the persistent context.
  context.Dispose();

  // Convert the result to an ASCII string and print it.
  String::AsciiValue ascii(result);
  printf("%s\n", *ascii);

  return 0;
}

Save that in a file called hello_world.cc in the ~/dev/v8/ directory.

Next you need to compile the hello_world.cc file by doing:

$ g++ -m32 -Iinclude libv8.a -lpthread hello_world.cc -o hello_world

This compiles the hello world program to a 32-bit executable that includes the libv8.a library.

If the compilation was successful you should have a new file called hello_world in your V8 directory. Then you execute the file and it should print out "Hello World!" (without the quotes):

$ ./hello_world
Hello World!

Congratulations you have just run your first program that uses V8.

In the next part of this series we'll look into sharing variables and objects and dig into the source of Node.js.

Using content providers in Android

Joseph Adams (whitegolem@gmail.com) — Tue, 03 Jan 2012 00:00:00 +0000

In this blog post I'll show you how to get data stored on an Android phone using content providers. We'll write an app that fetches all the bookmarks and history from the phone's browser.

Most Data in Android is exposed through so called "Content Providers". The person who wrote a specific part of Android (i.e. the browser, but surely it wasn't just one person), could decide how to implement the storage. If he used SQLite or just a basic file or json, etc. It didn't matter because Google decided to let their developers implement these content providers so that there would be a standard way for developers like us, to access the data.

In Android all the content providers are stored in the package: android.provider.*

They are queried using their URIs and Cursors.

Specifically the Browser's Bookmark and History content provider URI is in: android.provider.Browser.BOOKMARKS_URI
(The name is a bit misleading, since it is the URI for BOTH Bookmarks and History.)

To retrieve data you need to find the names of the columns you want to retrieve and put them in an Array. You can find the package that contains the column definition for the bookmarks and history here.

So let's get started!

Let's first set up all the variables:

import android.widget.Button;
import android.provider.Browser;
import android.net.Uri;

import java.util.List;

Button getBookmarksButton = (Button)this.findViewById(R.id.getBookmarksButton);
// I have set up a simple button in my layout, for which we will write a onClick listener.

Uri bookmarks =  Browser.BOOKMARKS_URI;
// The URI to query for bookmarks and history

List bookmarksList = new ArrayList();
// Here we'll store the retrieved data.

String[] columns = new String[] {
    BookmarkColumns.URL
};
// Array of all the columns you want to get. If you wanted any more like when it was created,
// just add another entry in the Array.

Now that we have the basic variables we can write the cursor. The function that creates it takes a lot of arguments. As you can see not many of these are important for us:

import android.database.Cursor;

Cursor managedCursor = managedQuery(
    bookmarks, // URI of the resource
    columns,   // Which columns to return
    null,      // Which rows to return (all rows)
    null,      // Selection arguments (none)
    null);     // Order the results (in the order they come)

Next we'll make a new onClick listener for the button. This listener basically loops through all the data rows and does something with them:

import android.view.View.OnClickListener;

getBookmarksButton.setOnClickListener(new OnClickListener() {
    @Override
    public void onClick(View v) {
        if (managedCursor.moveToFirst()) {
            String url;
            // Variable for holding the retrieved URL
            int urlColumn = cur.getColumnIndex(BookmarkColumns.URL);
            // Reference to the the column containing the URL

            do {
                url = cur.getString(urlColumn);
                // Get the field values
                bookmarksList.add(url);
                // Do something with the values.
            } while (managedCursor.moveToNext());

        }
    }
});

Now you have a list containing the bookmarks and history of a phones browser and you could for example send the urls to a server.

You can download the whole Activity here

RSA public/private key encryption explained

Joseph Adams (whitegolem@gmail.com) — Sat, 10 Dec 2011 00:00:00 +0000

In this blog post I'll show you how to calculate a simple RSA private-/public-key pair.

First of all you need to know that each key (the public-key and the private-key) consists of 2 parts. The first part is different in each key, the second is equal in both. So let's start calculating!

Warning: The encryption in this tutorial has a very low security level, because of the low values of p and q! To improve the security level choose higher primes.

Think of 2 prime numbers.

In real openssl, these prime numbers are very large and have a similar bit-size. So that we have no problem doing the calculations I'll choose 2 numbers that are quite small:
```
p = 3, q = 11
      
```
Calculate the modulus for the public/private key.

This is the number that is the same in both keys, let's call it n. It is used as the modulus in en- and decryption. You calculate it by doing:
```
n = 3*11
```
Calculate the totient of n.

Calculating the totient is easy. You could use a table and look up the value, but the calculation is just as easy and faster. Just do:
```
totient(n) = (p - 1) * (q - 1)
```
In our case we do:
```
totient(33) = (3 - 1) * (11 - 1)
```
This equals 20.
So far we have:
```
p = 3
q = 11
n = 33
totient(n) = 20
```
Choose a number for e

This number is is a bit harder than the others. It has to be between 1 and n, also coprime to n.

This basically means that the greatest common divisor of both numbers is 1. If you choose a prime number for e all you need to do now is check that e isn't a divisor of n. I'll choose the number:
```
e = 17
```
Calculating the modular multiplicative inverse of e * (mod totient(n))

Now at first this sounds a bit overwhelming, I struggled a bit to find out what it means. Expressed in an easy way you could say: What is the solution to the equation:
```
(e * x - 1) mod (totient(n)) = 0
```
It would take quite long to work it out by hand so I wrote a small Javascript function that does the work for me:
```
function doLoop(e, totient) {
  var i = 1, x;
  while (true) {
    x = (e * i - 1) % totient;
    if (x === 0) {
      console.log(i);
      break;
    }
    i++;
  }
}
```
The function takes 2 arguments, one is e the other is the totient(n). Depending on your processor and the size of the numbers you choose it can take longer or shorter to run.

In our case the function will log the value 13, which I'll call d. Now you have all the values needed for public-/private-key encryption and decryption.
Putting it all together

All the values up to now:
```
p = 3
q = 11
n = 33
totient(n) = 20
e = 17
d = 13
```
Your public-key is now: e = 17, n = 33

Your private-key is now: d = 13, n = 33

With this private and public-key you can now encrypt data by doing this:

We'll encrypt the value:
```
m = 9
```
To encrypt with the public key, you take m to the power of e (in the public-key) mod n
```
m ^ e mod n
```
```
9 ^ 17 mod 33 = 15
```
Our encrypted value is:
```
c = 15
```
This can only be decrypted with the private-key.

To decrypt it, you take c to the power of d (in the private-key) mod n
```
c ^ d mod n
```
```
15 ^ 13 mod 33 = 9
```
Now we have our original value!