New ´óÏó´«Ã½ Site Search
Ìý
This week, we launched our new site search product. This launchÌýis a big step on our journey to deliver aÌýsingle, unified and consistent discovery experience for the ´óÏó´«Ã½.
Try it outÌýwith the search boxÌýin the top right-handÌýcorner of most pages (or, if you can't think of something to search for, tryÌýLiz Hurley,ÌýMiranda,ÌýUnemployment,ÌýSpain). Let us know your thoughts using the form at the bottom of this page.
The new search is based on the Search+ platform, which I blogged about earlier in the year, and includes many refinements and enhancements.
This new search supersedes both the standard search product and the three-column Search+ product to offer a single site search experience for ´óÏó´«Ã½ users.
Before I go through the new features in detail, I'd like first to give you an insight into our product vision and discuss some of the issues that have shaped it.
To scope or not to scope
When I started in this job, there were hundreds of distinct sites on bbc.co.uk and most of those websites would have their own search "scoped" to their content. Not only was this inefficient and wasteful but also it led to an inconsistent experience in which search boxes in exactly the same position on, say, the Schools website and the History website would produce different results. People were often confused; indeed, I remember analysing the logs for the Gardening website search and discovering that the top result was for [eastenders] and returned no results.
We decided to reduce the number of scoped searches and make the search box in the top right-hand corner of every page return the same results, wherever you were on the site. However, this was not universally popular. People tend to think of bbc.co.uk as only that part of the website that they like to use. Many people only use the News website, or GCSE Bitesize or iPlayer and, not unnaturally, are bemused when they search for something and get results from another area. We get many emails asking for the search box on News to return results from News only.
So, on the one hand, people are finding no matching results in scoped searches, even though relevant content exists elsewhere on the website. On the other hand, people like the convenience of scoped searches, particularly when they only use one of the ´óÏó´«Ã½'s services.
The table below further illustrates this problem. It is a log of the top queries coming from different ´óÏó´«Ã½ websites on one day in June 2010. The three columns show where the searches originated (on the iPlayer, News and Sport websites). Those queries in black are the ones for which the best results will come from that website (so, for a search in iPlayer for [luther], we should expect the best content to be from iPlayer). The red ones are the queries for which the best results will come from other parts of the ´óÏó´«Ã½ website (so, for a search in iPlayer for [revision science gcse] we would expect the best results to come from Learning rather than iPlayer).
Ìý
So, as you can see, the picture is pretty mixed, with a closer correlation visible the higher up the search logs one looks between the originating domain of the search and the provenance of the optimal result set.
Clearly, we need to have a search product which recognises that many users want to scope their searches to the domain they are currently in while simultaneously revealing great, relevant content that happens to be in another part of the website.
Product Vision
With this in mind, we created the following Product Vision.
Our aimÌýfor search is:
1. To create a scoped search for each ´óÏó´«Ã½ productÌý(News, iPlayer,ÌýC´óÏó´«Ã½Ìýand so on)Ìýthat "understands" the domain knowledge behind each product and uses that information to create the best possible sorting rules, display and filtering options within search. In this way,Ìýwe will provide the richest search experience available anywhere on the web for each of our products.
2. To allow users to search across product scopes to see everything that the ´óÏó´«Ã½ has on a subject.
3. To ensure that, regardless of which scoped search someone uses, they are always exposed to the best relevant content the ´óÏó´«Ã½ has to offer. This will ensure that, for example, a heavy News user gets to know about iPlayer content relating to their query.
4. To offer related results from non-´óÏó´«Ã½ sources.
5. To make it possible for people to discover trending topics on the ´óÏó´«Ã½ site and make feeds of results available to be embedded in other products (on and off the site). To allow search result pages to be recommended from within other ´óÏó´«Ã½ products.
We think Search v2 starts to fulfil a number of these ambitions. Let me run through some of the new features and demonstrate how they relate back to the Product Vision.
1: A scoped search for each ´óÏó´«Ã½ product
This example is a search for [graeme swann]. The results have been scoped to Sport. This shows the main elements of the page design, which I have highlighted with red numerals.
Ìý
Ìý
Ìý
1. The central column will contain the main list of results. The large heading (in this case, Sport) shows that you are searching within the Sport scope.
2. The left-hand column will contain navigation to the other scopes and filtering options such as media type, date range and sorting choices.
3. The right-hand column will contain search results from other parts of the ´óÏó´«Ã½ (Elsewhere on the ´óÏó´«Ã½) or other websites (Elsewhere on the web).
The styling of the new search results uses the ´óÏó´«Ã½'s Global Experience Language, which brings it in line with the rest of the website and which Bronwyn comprehensively explains here.
Category-specific filters
In the image above, you can see four sets of filters in the left-hand column. The first is the category filter. It shows you how many results there are for your query within each category. Below this are "scope-specific" filters. For the Sport scope they are:
Ìý
Ìý
Ìý
Ìý
Ìý
Ìý
Ìý
Ìý
Ìý
Ìý
Ìý
Ìý
Ìý
Ìý
Ìý
Ìý
1. Allows you to filter your results to show only video, audio or text & pictures. The default setting is All Media.
2. Allows you to set the earliest and latest dates from which you want to find results.
3. Allows you to change the ordering of your results. As I said above, the default order is Newest First, but you can also choose to order by Most Relevant First or Oldest First.
Ìý
Ìý
Over time, all scopes will be given filters best suited to that particular type of content, so Learning, for example, will have filters such as Subject, Syllabus and so on.
Context-driven search results
As discussed above, there is a strong correlation between the part of the website that a user is exploring and the type of content they are searching for.
So, we have set the system to return results automatically from the category that relates to the section of the website you are using - if you are on the News site when you enter a search term, you will get results from the News category in the main, central column. Likewise, if you search from Sport, you'll get sports results.
Over time, we will extend this functionality to other sections of the site - iPlayer, Learning, Food and so on. The eventual aim is that all products on the site will scope to their own search category while staying within a single, consistent design. As this happens, we will remove the old "scoped searches" .
From today, however, if you search from any part of the site other than /news or /sport, you will get combined results on our All Results page. You will know which area you are about to search in from the text inside the search box. On News pages the box looks like this:
Ìý
Ìý
Ìý
On pages where you are searching All Results, the box looks like this:
Ìý
Ìý
Ìý
Ìý
Ìý
2. Allow views across ´óÏó´«Ã½ products
Unscoped searches
When you hit the All Results page, the central column will contain the top results from all the matching categories. The categories are ordered by their relevance to the search query, using an adaptation of the Smart Zones process that I discussed in an earlier post. Here is the [graeme swann] query from above, run on the All Results scope.
Ìý
Note that the page layout is the same as in the earlier example. However, because I haven't scoped my search, I will see results from all areas of the site in the central column. In this case, Sport is rightly deemed most relevant to the cricketer, followed by Blogs, News and then TV & Radio sites. A maximum of three results are shown from each category.
Likewise, the query [christmas cake] returns Food as the top category. (Ignore the Editor's Choice result in each of the next three examples - I deal with this feature in detail later on.)
Ìý
Whereas a search for [iceland] returns News as the top category.
Ìý
And a search for [trigonometry] will put Learning content at the top.
Ìý
You can get to all results for a particular category by clicking the links in the category title (or in the navigation in the left-hand column).
Editor's Choice
Many search result pages will also include a selection of editorially chosen results that we believe are most likely to be what you are looking for. If you search for a programme title, say [miranda], you will see a link to the programme website.
Ìý
If you search for a country, say [burma], we will highlight the ´óÏó´«Ã½ Country Profile (as well as other editorially selected features) and if you search for a British place, say [blackpool], you'll get a link to the nearest ´óÏó´«Ã½ Local website.
Ìý
Ìý
3: Ensure that related content is never hidden
Elsewhere on the ´óÏó´«Ã½
As I said in the Product Vision, our aim is never to hide great, relevant content from users, regardless of the scope within which they are searching. So, whenever you search inside a scope, you will always see the Elsewhere on the ´óÏó´«Ã½ module showing results from other parts of the website.
Ìý
4: Offer related results from non-´óÏó´«Ã½ sources
Elsewhere on the web
Below the Elsewhere on the ´óÏó´«Ã½ module, you will see the Elsewhere on the web module showing related results from around the web. These are a mixture of editorially selected websites and algorithmically driven results.
Ìý
5: Linking to search results
Linking to search results from within the ´óÏó´«Ã½
In the example below, you can see that iPlayer now links to ´óÏó´«Ã½ search. In this case, a programme about Operation Mincemeat links to search result pages for World War II, Adolf Hitler and the show's presenter, Ben Macintyre. This is possible because search produces a controlled vocabulary of around 40,000 topics that are available as links and which I wrote about previously here.
Ìý
Search feeds
These 40,000 topics are also available as RSS feeds. Click on the orange symbol in your browser's address bar to get the feed.
Ìý
How we put this together
For those with a technical inclination, our senior engineer on this project, John Muth, will blog on the technical approach next week and I will add a link from this blog then.
We're hiring!
Finally, a bit of news about the ´óÏó´«Ã½'s search team. We are moving to Salford in the summer as part of a ´óÏó´«Ã½-wide relocation. As you can imagine, such a big change has resulted in a fair amount of turnover within the team and so we are now actively hiring software engineers (predominantly Java) and other roles, including my job - Technical Product Manager, Search. You can read about the recruitment process here.
Comment number 1.
At 17th Dec 2010, Matthew McDonnell wrote:*
Complain about this comment (Comment number 1)
Comment number 2.
At 17th Dec 2010, dagi wrote:I gave up using the search on www.bbc.co.uk some years ago, instead I've found that using Google and entering bbc.co.uk in the site box on their advanced search page (, always returns good results.
I'll give the new solution from the Beeb a try and I'll be impressed if it beats searching via Google.
dagi
Complain about this comment (Comment number 2)
Comment number 3.
At 21st Dec 2010, Jamie Tetlow wrote:It's clearly a vast improvement from a visually aesthetic point of view and the filtering has certainly cracked the skinned-scoped search problem (although /iplayer doesn't seem to be playing game?) so congrats... BUT... the results themselves still seem to be quite poor for programmes that aren't currently available to watch/listen. The first example I chose to test was the wonderful ´óÏó´«Ã½ Four programme The Joy of Stats which despite expiring it's availability yesterday has, on it's page, a wealth of rich information, gorgeous links (Search included!) and even available video clips... So, looking at the Search results for The Joy of Stats i'm very disappointed to get only one result of an audio clip from the Today programme (that doesn't, in itself, link to the programme page).
I don't underestimate the magnitude of the task; i do congratulate you on the improvement but to me (and my biases) there's a significant oversight.
Complain about this comment (Comment number 3)
Comment number 4.
At 21st Dec 2010, Russ wrote:I tried it with a dozen or so recent programme titles. It failed to find any of them.
Nul points.
Russ
Complain about this comment (Comment number 4)
Comment number 5.
At 22nd Dec 2010, Matthew McDonnell wrote:Jamie & Russ - thanks for your posts. We are working on improving the results for programmes which are outside their iPlayer on-demand window (typically older than 7 days). It is understandably something that many people in the ´óÏó´«Ã½ are very concerned about. We should see every programme being returned in search sometime in January (we had planned to release this functionality with this launch but some problems meant we had to de-couple it from the main release). I will post again once it is out.
Russ - could you give me a few examples of the programme titles that you searched for? So I can check the reason for them not returning.
Complain about this comment (Comment number 5)
Comment number 6.
At 22nd Dec 2010, dotconnect wrote:Just tested out the new search, and I'm finding it a huge improvement, both in terms of results and the experience of the search itself. More functional, and better designed (design of course being about so much more than just "visual aesthetics" - it's now easier to identify links and filters and other information). Well done Matthew.
Complain about this comment (Comment number 6)
Comment number 7.
At 23rd Dec 2010, Matthew McDonnell wrote:Dotconnect - Thankyou for your comments, it is great to know you are seeing improvements. We feel that the platform gives us a great base upon which to develop in the future. There is still a lot to do: comprehensive programme results which I mentioned above; fleshing out the filters for scopes other than News and Sport; better support for misspellings; changes to the auto-suggest system and new advanced features.
Complain about this comment (Comment number 7)
Comment number 8.
At 23rd Dec 2010, Jamie Tetlow wrote:Thanks Matthew, I look forward to the update... and keep up the great work, these improvements to Search certainly help bbc.co.uk feel more like one site - those links to search results from content pages are acting like a glue!
Complain about this comment (Comment number 8)
Comment number 9.
At 4th Jan 2011, Russ wrote:Matthew, in response to your request in #5, here are a few programme titles taken from the R4 drama and readings area that don't give any results:
What Do You Know?
Faith Healer
Gilead
On the Ceiling
The Caretaker
Achting Tan
Troubles
Severed Threads
God's Revolution
Tokyo Vice
Only the ´óÏó´«Ã½ could launch a new search function knowing it was incapable of finding ´óÏó´«Ã½ programme titles.
Nul points again for a total failure in strategic vision.
Russ
Complain about this comment (Comment number 9)
Comment number 10.
At 10th Jan 2011, seated wrote:This comment was removed because the moderators found it broke the house rules. Explain.
Complain about this comment (Comment number 10)
Comment number 11.
At 14th Jan 2011, Jacques wrote:This comment was removed because the moderators found it broke the house rules. Explain.
Complain about this comment (Comment number 11)
Comment number 12.
At 24th Jan 2011, Charlie Hull wrote:I couldn't find John Muth's article about the technical approach - would be interested to know what technology you base the search on - is it open source?
Complain about this comment (Comment number 12)
Comment number 13.
At 19th Feb 2011, jacmak wrote:Since the design elements are so consistently overbearing in their eschewing of aesthetic appeal so as it would seem to purposefully diminish customer experience of the product, then insofar as ´óÏó´«Ã½ online is funded by the viewers without any option of exempting themselves from any part of it, then ´óÏó´«Ã½ online has placed itself in a rebartative stance vis a vis those who fund it. Which of course, is an inequity on more than one modish 'angry', but nevertheless, decorous battlefront.
Complain about this comment (Comment number 13)
Comment number 14.
At 27th Feb 2011, Anna Sempe wrote:@dagi (#2) - I've been doing the same for the last couple of years - using Google and including bbc.co.uk together with my search term generally results in getting the results much faster. Also, and google's indexing technology is pretty sophisticated in working out what's news related and what's product relates so I've rarely bothered using the ´óÏó´«Ã½ search facility.
I have to say though, from what's described above in terms of scoped and unscoped searches I'd probably give the new ´óÏó´«Ã½ search a go - think I'd get a more useful group of results now than from google
(UK blogger)
Complain about this comment (Comment number 14)
Comment number 15.
At 28th Feb 2011, godolphin2010 wrote:I think that there should be a collaboration between old and new teams on MI High
Complain about this comment (Comment number 15)
Comment number 16.
At 8th Mar 2011, Martin wrote:Within 24 hours of "Wonders of the Universe" I tried to find information about the first broadcast using the ´óÏó´«Ã½ search engine, despite long ago having taken such searches elsewhere.
Today, if you search for "Brian Cox" you get 680 results, including relevant ones.
If you search for "Brian Cox arrow time" you get none - not one - 0 results. In Google the top two results point to ´óÏó´«Ã½ pages.
If you search for "arrow of time" on the ´óÏó´«Ã½ search engine, the only relevant result for this user is a link to a Daily Telegraph review of the programme.
The simple conclusion is that specification for the search engine is so sophisticated and feature rich that it overlooks the basic fact that it is supposed to do something.
It looks like a decision that is somewhere strategically wrong - wrong platform, or something.
This an issue that people have raised for a long time, and for lack of any evident response, they give up mentioning it as a problem. In general, I expect the public just give up on the search engine, ignore it.
Is there any information at all about the use of the search engine indicating whether or not it satisfies most searches, or about the extent to which links come from Google searches, for example?
This just has to be an issue that ´óÏó´«Ã½ management need to be aware of and address, and tell us that they are addressing.
What is said here has been said by many other people - just look at the comments above - and there is no evident response. Even if those responsible for it despair of improvement, it is not good enough. Where do you send a question to actually get a reply?
Complain about this comment (Comment number 16)