Prototyping Weeknotes #41 (26/11/2010)
After the success and frenzy of the Autumnwatch trial last week (blogged about here and here), there's some washing up to do on Monday. George is on leave all this week. Meanwhile, I spend the day at an internal FM&T management conference at Television Centre, soaking up strategy. Tris meets a chap from Worldwide to talk about music links.
On Tuesday, Tris starts research and planning on a new Storytelling project and has meetings with some interested people throughout the week. Chris G, Tris and I have a lengthly session on our net neutrality project that I'm owning, ironing out some kinks in the concept, and I spend some time writing up the current thinking.
Kat is getting her teeth into some juicy RadioDNS pieces, and works with Jo on finalising designs for an upcoming focus group on radio visualisation.
Wednesday comes and I'm over in W12 for meetings, then back to base where myself, Sean, Chris N, Tris & Kat catch up about the Twitter Firehose big data project, which is yielding (as expected) all sorts of interesting engineering challenges. We've hit a bit of a roadblock with some of the compression technology we've used to save storage, so that requires a rethink and a workaround. Meanwhile, Kat is continuing to refine the use cases for data analysis side of the project.
Sean spends most of the week up to his elbows in Java code, debugging some fairly new and fragile libraries - life at the cutting edge. Chris N implements LZO decompression into our data loader, and takes some detailed timings of running Elastic MapReduce jobs with different pool sizes & configs of worker machines. Initial results look encouraging.
There's a lot of other interesting big data things happening out there we're interested in, including , a distributed stream computing platform, and the lower-level (yet still very useful) - a parallellisable xargs command.
Our now-regular Show & Tell session on Thursdays kicks off with Mark doing a successful demo of his AudioSync work, which now features multiple frequency bands to improve the accuracy and tolerance of matches against background noise. Good discussion ensues about future possible applications for the work. Chris N & Sean give us a runthrough of the Firehose work so far and pull out some initial data to peruse. Duncan is finishing off his blog post and code release about the PubSub infrastructure for the Autumnwatch trial, whilst Theo is filming and editing a video demo of last week's experiment with a willing friend to help explain what the user experience was like on the day.
Friday sees me wrapping up some new recruitment and working with Chris G and Chris N finalising the plan for some important infrastructure work to happen next week. Lots of kinks and uncertainties have been straightened out, but in the end it's all down to a chap called Denis and a van. But then isn't everything?
Comments Post your comment