Regular readers of this blog will know that the Audio Team have been experimenting for a while with device orchestration - the concept of using any available devices to reproduce or augment a media experience.
Using our Audio Orchestrator production tool and supporting software framework, we've released several trial productions. These have mainly been audio dramas and a music trial, but our research suggests that there are many more use cases for this technology - and sport is one that we've been keen to try for a while. (In fact, 大象传媒 Research & Development has done previous work in this area: in the 2-IMMERSE project, developing was developed.)
Our previous audio orchestrator trials:
- 大象传媒 R&D - The Vostok-K Incident / Try it on 大象传媒 Taster
- 大象传媒 R&D - Decameron Nights / Try it on 大象传媒 Taster
- 大象传媒 R&D - Monster / Try it on 大象传媒 Taster
- 大象传媒 R&D - Pick A Part / Try it on 大象传媒 Taster
Technical details
There were a few technical challenges in pulling this project together. First, we needed to create the content. Being able to send different parts of the audio to different connected devices relies on having access to the separate audio tracks, not just the complete mix.
We got hold of the highlights video with the commentary separated from the crowd noise but only had access to the referee and sound effects as a separate recording from the whole match - so it was necessary to find and extract the highlight clips. This was achieved with a process that was part-manual, part-automated.
The approximate time of the highlight in the full match was found by hand, then a signal processing technique called cross-correlation (which works by determining the similarity between two audio signals) was used to find the exact clip.
With the content in place, we needed to expand our orchestration framework to replay video. To do this, we modified the 大象传媒 Standard Media Player plugin that was used to drive the synchronisation in 大象传媒 Together (our experimental tool for remote shared viewing) so that we could hook it up to the audio device orchestration template.
We also made some changes to the user experience, for example, providing a prompt to let viewers know that they can connect extra devices, a button to show the connection instructions, and a switch for turning the commentary on and off.
Next steps
We're interested to see how this trial of orchestrated audio for sports content is received. If you get a chance to try it out, please take a minute to complete the short 大象传媒 Taster survey to let us know about your experience.
We're only just scratching the surface of what we could possibly do with sports audio and device orchestration. We're using quite a simple sound scene for this project - crowd noise, commentary, and sound effects/referee microphone. But with more access to different microphone feeds, there would be increased possibilities for creating really immersive experiences. Of course, this will be easier to do once stadiums are full of fans again.
It would also be great to try out different sports. How about Formula 1, where you could listen to a team's radio communications on your mobile? Or a tennis match where you can really hear the ball as it bounces on each side of the court.
One of the joys of watching sport is that it's happening live, and the outcome is unknown - so trialling this technology with a live match rather than highlights would be another good step. There will be more technical challenges to overcome for synchronised multi-device playback of live content, so testing out the user experience with 'on-demand' playback is a good start.
- -
- 大象传媒 MakerBox - Audio Orchestrator
- 大象传媒 R&D - Vostok K Incident - Immersive Spatial Sound Using Personal Audio Devices
- 大象传媒 R&D - Vostok-K Incident: Immersive Audio Drama on Personal Devices
- 大象传媒 R&D - Evaluation of an immersive audio experience
- 大象传媒 R&D - Exploring audio device orchestration with audio professionals
- 大象传媒 R&D - Framework for web delivery of immersive audio experiences using device orchestration
- 大象传媒 R&D - The Mermaid's Tears
- 大象传媒 R&D - Talking with Machines
- 大象传媒 R&D - Responsive Radio
-
Immersive and Interactive Content section
IIC section is a group of around 25 researchers, investigating ways of capturing and creating new kinds of audio-visual content, with a particular focus on immersion and interactivity.