18 Months Of Blogs (Part 1)
Part One: History & Technical Challenges
The ´óÏó´«Ã½ Blogs Network has been up and running for 18 months. This milestone provides a good opportunity to give you some insight into what we've been doing, how we think our efforts measure up, and where we might be headed in the future, editorially and technically.
That's a lot to cover, so we're splitting the discussion in two. Today's post deals with the technical aspects of our Blogs Network and later this week, I'll return with a second post looking at editorial challenges.
So, first: some history. In early 2006, following several forays into blogging by the ´óÏó´«Ã½, most notably ´óÏó´«Ã½ Scotland's Island Blogging, Ouch! and Nick Robinson's , the decision was made to customise and install an off-the-shelf blogging solution and create the ´óÏó´«Ã½ Blogs Network, which launched in April.
The graph below provides four snapshots, taken at six month intervals, of the unique visitors and visitor sessions for the ´óÏó´«Ã½ Blogs Network, starting with that first month and ending with October 2007.
The graph illustrates that the ´óÏó´«Ã½ blogs have, in aggregate, found a large and steadily growing audience. Additionally, it's worth noting that the average time of each visit has remained fairly static at just under four minutes.
The downside of this success has been our technical infrastructure becoming increasingly unstable. It was built quickly and involved a number of customisations to the software which effectively ended our ability to easily install software patches and bug fixes provided by the vendor in order to deal with some of the technical issues we've faced.
Our technical woes will have been invisible to most people visiting a ´óÏó´«Ã½ blog, but this will be of little comfort to those who have encountered them first-hand, including Newsnight editor Peter Barron who recently posted on The Editors: "Often I try to respond to a comment or complaint about the programme and end up gnawing my knuckles in frustration as the response either doesn't appear for many hours or fails to materialise at all. Hardly the best way to have a free flowing dialogue with our viewers."
It's not just those trying to comment who have ended up frustrated. An increasing number of live programmes, such as the recently launched iPM and World Have Your Say have begun to use their blogs to engage with their audiences directly. Inevitably, this leads to upset programme makers too.
Eddie Mair wrote a post at 7am on Saturday, the launch day of his new programme iPM: "I do want to say what a bang-up job the team have done. As you'll see from the iPM Blog, they have done a ton of great work - I find it really rewarding to have a rummage round the blog." Later that morning, he updated the post to say: "By the way we are aware that the 502 error messages are back. Great day for it to happen, no?"
We are aware of the problems and of how frustrating they are for anyone who takes the time and effort to compose a thoughtful comment only to see that effort result in a 502 error or an automatically generated message from the server saying, wrongly, that they've been banned. Here's where we're at in our efforts to diagnose and resolve them.
The early results of a full technical review we've commissioned from an outside agency - as well as our own investigations - indicate that these technical problems are almost certainly down to the bottleneck that exists between our three front-end web servers and the database server. The database is what ingests, stores, and allows the management (including moderation) of comments and then outputs those comments and the other data that make up the web pages you look at.
The graph below [click for large] shows the percentage of CPU usage on our blogs database server over a week. Clearly visible are the spikes, sometimes reaching as high as 100%. It's during these spikes that you are most likely to encounter problems commenting.
The primary cause of these spikes is spam attack. Worryingly, the amount of spam comments submitted is on the increase and, across the network, we can now get as many as 50,000 spam comments per week. In one particularly bad day last week, the comment cgi was hit well over 30,000 times.
Each time a spam comment is submitted, usually by a bot, the database CPU crunches the data. We're aware that the version of the application we use is less efficient in dealing with spam than newer versions but, as I mentioned above, customisations we made to the application have effectively ended our upgrade path, so we've been unable to take advantage of upgrades since made available by the vendor.
That's the bad news. The good news is that our new(ish) applications engineer, David, has been busy firefighting and implementing various anti-spam measures. We've also commissioned to conduct a full technical review of our blogging platform, the results of which will be delivered on November 19th. I don't want to pre-empt that report but I can say that, in addition to reviewing our existing platform, the report will also present us with a number of suggestions for improving our technical infrastructure and these are likely to include a mix of software and hardware upgrades or replacements.
My apologies if you've encountered problems when using the ´óÏó´«Ã½'s blogs. When the review is delivered next week, we'll begin the process of planning and implementing a robust solution over the following weeks and months. In the meantime, we'll carry on with our daily fire-fighting and will continue to provide as good a service as we can - although there are bound to be a few problems from time to time.
I'd be glad for any thoughts you might have and hope to see you on Wednesday for Part Two.
Robin Hamman is a Senior Community Producer.
Comments
Will you be posting a breakdown of the report on the blog when you get it?
It's nice to have an in-depth from you, I hope to see more in the coming months :)
I know the pain of installing too many plugins and the chaos it causes at upgrade time, and that's just running a little system for a non-profit site which has five users (and certainly no comments!)
All it takes is one plugin author not to upgrade, and it all goes screwy! It's made me a lot more cautious about customising stuff, I can tell you.
Interesting that you name-check Headshift, but not Six Apart, the 'software vendor', or Movable Type, the product.
Does this indicate some sort of dissatisfaction with the vendor/product, and might you be looking to switch to a different product, I wonder...
today's the 19th!
xx
ed
And another 502 to add to my considerable collection!
Mon Nov 19 13:07:58 GMT 2007
and another
Mon Nov 19 13:20:06 GMT 2007
and...
and
and why do I bother?
AND TODAY'S THE 24th
Might the word "customisations" have got stuck across a pipe somewhere?
With Christmas so close - A merry 502 to one and all.
Thanks for the responses. A presentation was made of Headshift's key findings on the 19th. I'm hoping to post more about that when I have the chance to read through and pull out the most interesting bits from the final report.
Since then, we've been kept busy having various discussions about what resources will be required and where they might come from to create a development plan and being to put it into action.
As to the 502 problem, we do have a software engineer dedicated to firefighting problems with our existing platform. Over the past month we've made a lot of headway but the spammers are just as adept as we are at changing their tactics and causing our database to be overwhelmed. We're doing what we can and have several more tricks up our sleeves so please do try to bear with us. Believe me, I know it's painful - I've hit the 502 myself a couple of times.
Returning to Frankie's question, the ´óÏó´«Ã½ does often mention the name of the person or organisation commissioned to review one of our services. We don't, however, usually mention the brand names of different equipment or systems we use.
Our current blogging platform, Movable Type 3.2, has some known issues that have been resolved by Six Apart in subsequent releases. I personally have met with various people from Six Apart and they were all happy to provide as much help and advice as we could ask for. Unfortunately, the customisations we made during the installation of our blogging platform has made it difficult, if not impossible, to make use of this assistance and various upgrades that others have benefited from.
The affect of customising complex applications such as Movable Type upon the later availability of vendor supplied, or third party, upgrades and plug-ins is something that will weigh heavily in the decisions we make when devising and implementing our new blogging platform.
I'll come back and post again when I have more news, or questions to respond to. Thanks for your interest and, in some cases, patience.
Robin
´óÏó´«Ã½ Blogs Network
mm.. thank you man
As Martin Belam has pointed out on his Currybet weblog, Scotblog was the first blog at the ´óÏó´«Ã½, not Ouch! or Island Blogging.