´óÏó´«Ã½

Selecting the content

The editorial requirement was to display the "top headline from the News, Sport, Business and Entertainment menus" plus "something from Weather, if possible" and "the existing promotional text".

The existing promotional text already comes from a simple tool made by the TV Platforms team, and this would be added in by the homepage publisher.

Selecting the top story from the four sections wasn't going to be hard, but we wanted to make sure it matched exactly what was actually in those different sections. We decided that we'd build the list of headlines exactly as it was just another menu in the main part of the service - but a menu that would copy the top item in the four other menus. We already do something similar in the Business section on the Sky service where the top two stories from Business News appear on the top level Business index (page 200).

There was a complication to this - occasionally, the same story appears at the top of the News Main Headlines and at the top of one of the other three indexes, and we didn't want the same headline to be duplicated in that case. We decided to drop such duplicates, reducing the number of headlines that would be displayed.

We also had another problem. Occasionally, the journalists "overtype" the headline for certain index pages if they think, for example, that an Entertainment story headline needs more clarity when it appears in the core News menu. Our system can't tell when the two different headlines point to the same story which means we may occasionally have the same story in the headlines twice (though the actual headline itself would be different.)

The "something from Weather" also posed a bit of a challenge. Weather content is fed to Red Button through a separate system and there was insufficient time to make any modifications to that feed. As it stood, there was nothing in that feed that was the right length and there simply wasn't room to put a useful summary or forecast information or weather warning as these can run to multiple pages.

What we could do, though, was count the number of weather (or flood) warnings - and if the number was one or more, add some text to state that "Weather warnings are currently in force". Counting is something we can do fairly accurately without needing any modifications to the feed!

Feeding the downstream system

As I mentioned, to us in Journalism, the list of headlines from the bridge is simply yet another menu in the Red Button service, albeit one that is formatted in a slightly different way. However, it needs to be delivered to a different publishing system to the rest of the service though luckily it can be published using the same protocol. What we do then is build a complete version of the whole service - the main text service and this extra menu - and deliver it to both downstream systems, filtering out the inappropriate content for each destination.

The core functionality was already there, since we already filter out certain test data and "glue" scenes from the feed. All we needed to do was make this filtering process a little more flexible and configurable. The bridge publishing system, then, gets a service delivered to it consisting of exactly one menu - the Bridge Headlines menu. The main text publisher gets everything except the Bridge Headlines menu (and the test and glue content).

Displaying page numbers

We wanted to display page numbers alongside the headlines to allow viewers whose interest was piqued by a headline to find out more about it. It would also help to draw people into the text service, perhaps including people who had never explored it before.

The first problem was a rather simple one - the top story in each of the indexes didn't have a page number of its own. Rather than using the page number of the menu, we decided to add a new page number to the top story of each of these menus - so now page 104 will always take you directly to the top story in the News Main Headlines, for example.

Secondly, we had to decide how we were going to figure out the correct page number to add in each case. There were three solutions proposed. In the first case, we could hard code the page numbers to appear in each menu position - so the first menu item would always link to page 104. This would work if the first menu item always pointed to the top story of the News Main headlines, but would have limited our flexibility if, for example, we wanted to point to a different destination (at one time, we considered pointing to the market data index page if the Business headline was a duplicate of the top News headline).

The second option was for us to keep a hard-coded list of the page numbers for all the pages we might want to link to, and look up the right one. We didn't like this since it leaves us with maintaining the list - our colleagues in the TV Platforms team deal with all of that). It also left us with the problem that our system can deliver content to multiple publishing chains, and those publishing chains might not all have exactly the same page numbers defined on them.

The third option was for the TV Platforms team to insert the correct page numbers that were currently defined in the central page number database so we could be sure we were using the latest, correct information. Ideal from a technical viewpoint, but unfortunately too much work to complete in the time available. We settled on a compromise between the second and third choices - the TV platforms team now let Journalism look up the quick codes on the fly, so we can insert them in the text, but they are currently hard-coded, rather than being looked up dynamically in their database. Hopefully one day the code will be added to make this look-up dynamic.

Finally, we had to decide what format the page number would be displayed in. Various alternatives were mooted, but there was no real substitute for trying a few out. We love making stuff configurable (far better to spend a few seconds updating a bit of text in a database at 2am, rather than having to go through all the pain of a full blown code deployment), so we extended our existing "Title Override" capability to let us modify the Headline text on-the-fly.

Our title override for the top Main Headline now looks like this:

%%{TITLE}%%{QC(FORMAT=" - p%s%")}

%%{TITLE} means "put the real page title here", and %%{QC(FORMAT="something")} means "insert the 'QuickCode' (the internal working title for the project that delivered page numbers to the service) here, using the specified FORMAT". After numerous tests, we opted for space, en-dash, space, letter p, page number (%s%) if we could find the right page number, skipping the text if we couldn't.

It is also possible for us to define a different format if we can't find the page number somewhere, giving us the ability to hard code a link to the Sport Headlines menu if we can't find a page number for the 2nd sport headline, should we ever decide to display this in the future.

The %%{TITLE} is substituted pretty early in our system, since we always want the same headline to appear, no matter how many publishing systems we are copying the feed to. The page number, though, is substituted much later in our chain - pretty much at the last minute - so we can choose the right page number for the specific publishing system that we are delivering to at the moment. Just before we publish, we ask the downstream system for a list of all available page numbers, and insert the correct one (if it exists).

We did find a little bug in the way we implemented this code.

Any developer who has had to deal with XML will either already have come across encoding problems, or else will soon come across them. We use XML a lot internally, and XML gives a special meaning to the '&' character (as well as a small handful of other characters, including the double quote).

Any text that includes the "&" character needs to be encoded properly, otherwise it will become corrupted when it goes through an XML processor. You find out pretty quickly if you forget to encode the text - your system collapses in a heap when the first unencoded "&" character passes through, and it prints out some pretty specific error messages.

The problem comes when you respond to this situation by encoding the text - but find that under certain circumstances, the text has already been encoded once. Each "&" gets translated to "&" (which is correct) but then gets encoded again to "&" which is decidedly wrong - but which doesn't trigger the errors that would cause you to notice the problem in your automated tests.

In this case, we needed to encode the title text (which could contain & or " characters), but not the FORMAT section (which needed to keep the " character unencoded) - and we overlooked this. Luckily, this showed up in our QA tests, and we were able to release a fix to the code before we went live. Nevertheless, it was a firm reminder to be very aware of text encoding issues when dealing with XML.

Incidentally, we found a different problem, also relating to encoding, after we went live. The "£" character is not part of the US-ASCII character set, and has lots of different representations depending on which encoding system you choose to use.

Unfortunately, the live satellite publishing system was configured to use a different encoding by default than the one that was used on our QA system. On the day we went live, the Business news headine had a £ in it - which displayed incorrectly as a result. Luckily, we were able to get this fixed pretty quickly by changing the default character encoding used on the live systems... Encodings need thinking about!

You Ain't Gonna Need It

In the Software Development world, there is a saying - "You Ain't Gonna Need It", usually abbreviated to YAGNI. Essentially, it means that you shouldn't over-design your systems to allow this kind of flexibility unless the specification says that this flexibility is required, and the formal written specification we had didn't require it.

To this, I have three responses. First, much of this functionality was implemented with small additions to existing code, which already had this flexibility built into it. Second, the flexibility allowed us to iterate through potential designs rather rapidly, trying out half a dozen things in an afternoon rather than taking a couple of weeks to get there - so the specification may not have required flexibility, but the specification itself was flexible.

Third, the first time we showed this to the working journalists, they started talking about all the things they might be able to do with the new functionality - so it looks like we will need it after all...

Bill Michell is Software Engineering Team Leader for Future Media and Technology Journalism.