´óÏó´«Ã½

a bits about the bits recipe...

So, here's the basic recipe I've been using to create a bare bones, proof-of-concept, semi-automated text categoriser -- which I call "conText":

1. (careful, that's a !)
2. into individual text files, one per Wikipedia article (and this makes things much bigger, to the tune of 20GB+, so make sure you've got the hard drive space cleared)
3. Use a indexer to create a searchable collection (inc. ) of your new local Wikipedia text files, one Lucene document per Wikipedia article
4. Use Lucene's to of a chunk of your own text content to the Wikipedia documents in your new collection
5. Treat the ranked Wikipedia articles returned as suggested for your text

So, that's the very high level recipe I've been working with. You can have a look at the type of results this approach gives at my .

(If you do want further brutal details, let me know in the comments area below -- but for now, I think I'll leave it to another day to delve into the scary, hacky details of the XML and Lucene search collection document structures, the fields and weightings, the tokenisers used and discarded, the messy Perl, Rails, and Java code -- yes, nastily, all three are involved -- the confidence rankings, the result filtering heuristics, etc.)

the results of the recipe...

The main point of all this is to explore whether the results this recipe coughs up are useful for content categorisation, and whether they might help audiences navigate to and around content. So, let me give you a sense of what this Wikipedia + Lucene approach spits out. Take, for instance, the following descriptive text from the episode 'A Riot' of the Radio 4 programme Making of Music:

A Riot. James Naughtie presents a series chronicling the historical influences that affected the course of classical music. 10/30. A Riot Igor Stravinsky's The Rite of Spring caused a riot in Paris on its premiere in May 1913. Why did it create such an uproar and how did it become an iconic piece? Read by Simon Russell Beale.

Paste that text into the form on the and hit "Get Wikipedia URIs".

The results are pretty interesting:

The top six results are:

• etc...

Those first 3 results seem like very useful categorisations for this episode of Making of Music, don't you agree? The result , in particular, thrills me and shows that the conText categorisation recipe offers something that, say, and don't (at least not yet ;-) ). "Classical music riot" is a pretty darn accurate and (dare I say?) insightful category for this content, but that phrase wasn't present in our submitted text, per se. Turns out that our content shared enough with the text of the article to come up as a match. Language and words and computers and people are interesting like that.

could these bits about the bits be useful?

How can these results help with the "bits about the bits" challenge, though? Well, for instance, we could use this list of results to create category tags on the episode's page, which, when clicked, take users to a page pulling together all ´óÏó´«Ã½ programmes sharing that category (a little like ) -- and which offer an onward journey to a ´óÏó´«Ã½ /topics page featuring links to a complete collection of content on that subject. There's plenty of other relevant ´óÏó´«Ã½ and Web-wide content available on, say, The Rite of Spring. Let audiences know!

Here's a picture showing the kind of "related Topics badge" that could appear with content, based on categorisation techniques like conText:

The idea is that a horizontal navigation badge like this can help audiences find more of what they are interested in.

getting even better bits about the bits...

Of course, the results above bring up some pretty obvious questions and potential limitations, too. Why hasn't the system picked out, say, "James Naughtie" and "Simon Russell Beale"? What could we do to improve the result ranking and filtering heuristics, so that we minimise (some might say) "noisy" results like "Concerto_for_Piano_and_Wind_Instruments_(Stravinsky)"?

And so on -- this is the kind of proof-of-concept that keeps asking questions even while it answers a few. If you have suggestions about all this, let me know in the comments below.

Now, I do have a possible tweak in mind to help address the "proper names" question above, i.e. how to get the system to send back as a category even when there's not much supporting text specifically about him.

Imagine running entity extraction on the content text first, then using the entities extracted as "narrow-the-scope guides" for the conText approach. For instance, pop the episode description text we used before into the . Then, go back to the conText submit page, use the text as before, but this time, in the box that says "Use the text above to disambiguate a specific concept:", put in "James Naughtie" (imagine the entity extractor supplying it automatically) and submit the form.

This time, the top 2 results are:

And, the top result's "confidence rank" of 1.6976633 makes it likely that is the best page for the specific "James Naughtie" our text refers to.

So, what's the trick/heuristic here? Just throw the extracted text (in this case "James Naughtie") against a (I should actually use Google or Yahoo!'s search APIs, but haven't bothered yet). This will give you the page-ranked set of Wikipedia pages the Web is using to link to "James Naughtie"-ness. Use this sub-set of Wikipedia to constrain the conText MoreLikeThis similarity comparison, so that only a page from this set can be a result. More times than not, you can disambiguate very accurately with this approach.

better the bits you know...

And the best part? This disambiguation process gives you back a URL for the category, not just a text label. This URL can be used as a for the category -- anywhere, on any system. You can use this URL to -- other people (and software) can simply use the URL to explaining it, to find out what it means and why it's relevant as a category. And this URL-as-category will help the overall of the Web become more and more , while still allowing the Web to play to its strengths as the , , we all know and love and which probably none of us could live without anymore. But .

finally, the bit at the end...

Mr Negroponte's insight into the importance of "the bits about the bits" -- i.e the context around a TV or radio programme: who made it, who's in it, what it's about, who else has listened to, commented on, blogged about, or emailed it to a friend -- well, reading that today, in my line of work, sends shivers of recognition up and down my geeky spine. And he could see it so clearly, way back in the halcyon days of '95 -- back when I was still trying to decide whether email was a good way to send Beat poems to a potential love interest or just a passing fad (email, that is, not Beat poetry).

The insights are many years old by now, and while the name's the same, do recent phenomenon like Wikipedia change the game? Go ahead and have a play with the : do you agree with me that Wikipedia + Lucene's MoreLikeThis = useful bits about the bits? Let me know what you make of all this -- do leave a comment on your way home. And if you decide to give this recipe a try in your own app, please get in touch and let me know how it goes. I'd love to hear if this thing's got legs.