Developing Search At The ´óÏó´«Ã½ - Pt 1
Part 2 is here. This post is part of the tenth birthday celebrations of bbc.co.uk
Until a couple of years ago, I was a Senior Development Producer at the ´óÏó´«Ã½'s New Media department. Whilst I was there I used to , and the team at the ´óÏó´«Ã½ Internet blog has asked me to contribute some articles here about the history of the ´óÏó´«Ã½'s web site.
I first started to work at the ´óÏó´«Ã½ in 2000, as a junior member of a small team looking after the ´óÏó´«Ã½'s search engine. Back then, searching the ´óÏó´«Ã½ site was a bewildering and perplexing experience, as there was no global search across all of the content.
Instead, on the Today site, you could find a small box in the top right-hand corner that only searched the Today site. Or, if you were on the EastEnders site, there was a long search box at the bottom of the homepage, that only searched the EastEnders site, and so on.
As well as being somewhat randomly placed, the search boxes weren't even all using the same technology. ´óÏó´«Ã½ News used a product from , whilst other bits of the ´óÏó´«Ã½ were indexed using software called . The results could be pretty appalling. One of my first jobs involved artificially putting the right URLs at the top of search engine results.
This wasn't a hi-tech solution. We had a spreadsheet that listed search terms, and the URL that should be displayed if a user employed them. We used to improve it based on the frustrated emails we got from the public. A mail would come in saying "I searched for 'Jeremy Paxman' and I never found the Newsnight site", and the team would dutifully add that 'jeremy paxman', 'paxman' and 'rottweiler' should produce bbc.co.uk/newsnight as the number one result.
The Muscat search engine was also unable to distinguish between different languages, so if you typed in 'Tony Blair' you were just as likely to get a news story mentioning his name from the ´óÏó´«Ã½'s Portuguese news site as from the English language site.
It was obvious it needed to be improved, and as part of the re-branding of ´óÏó´«Ã½ Online to ´óÏó´«Ã½i in 2001, a new global search was introduced. The grey 'toolbar' was added to the top of (nearly) every ´óÏó´«Ã½ web page, placing a search box on every page of the site.
Users still didn't get the same results from everywhere. If they were on the Radio 1 site, they only saw results from Radio 1 web pages, unless they chose to do otherwise. This was usually OK for about 85% of searches, which would generally be 'in scope', but it would give users bad results some 15% of the time. Plus of course, if you searched for "The White Stripes", there was no reason why content on the Radio 2 or 6 Music sites about the band wouldn't be of interest to you.
To get around this, the scope of the "Best links" that used to be hand-coded into that spreadsheet was increased. It became a large behind-the-scenes taxonomy mapping relevant ´óÏó´«Ã½ content against thousands of keywords and concepts. If you typed in something like "Test The Nation" as your search anywhere on the site, the top results would include a ´óÏó´«Ã½ "Best Link" taking you to the national IQ quiz homepage. , but it was a vast improvement, and the indexing of the ´óÏó´«Ã½'s content was improving as well.
In my next post, I'll be looking at how the ´óÏó´«Ã½ introduced web search to the site in 2002.
Martin Belam is a former Senior Development Producer, New Media
Comments
Ah memories. I spent many months coding the HTML that delightful 2001 version of the Search site, and took great pride in persuading our designer to cut down the number of images we used! I'm afraid I am the reason why those tabs look so boxy!
The worst bit was that I had to code something like 60 different looks and feels for different search sites - most of which were based on the same design, but had different colour choices, different graphics - the lot. (Of course this was before we could use CSS in the ´óÏó´«Ã½) Oh and then there was the other 10 which all had completely different designs... And of course, each design was a completely different set of five template files...
In the end, I hacked together my first ever Perl script in an attempt to try and save my sanity. I'm not sure it worked...