Our summer of Ultra HD streaming trials finished on Sunday. We successfully delivered all 29 大象传媒 One World Cup games, and every Centre Court match at Wimbledon - all live on 大象传媒 iPlayer.
In total, our trial Ultra HD coverage received over 1.6m live requests. The most popular days were the following, which together with our standard coverage, meant we were setting new streaming records for the 大象传媒. These figures are peak UHD streams:
- Sat 7 July - 60.3k - Sweden v England QF / Wimbledon Day 6
- Tues 10 July - 48.1k - France v Belgium QF / Wimbledon Day 8
- Sun 15 July - 44.3k - France v Croatia Final / Wimbledon Men鈥檚 Singles Final
- Fri 6 July - 41.6K - Brazil v Belgium QF / Wimbledon Day 5
- Mon 2 July - 37.3K - Belgium v Japan last 16 / Wimbledon Day 1
The trial is an important step forward, showing for the first time that Ultra HD and High Dynamic Range (HDR) can be delivered live and 鈥渇ree-to-air鈥� over the Internet. It鈥檚 part of the 大象传媒鈥檚 mission to ensure that future audiences can enjoy the benefits of improved picture quality and this trial follows on from , albeit with different technologies.
We鈥檝e had lots of viewer feedback which we are very grateful for and will help us to determine our next steps. We were able to address much of this feedback during the trial, , which explores the operational challenges we faced in running a trial of this scale for the first time. for those who would like more detail on this.
Why the Trial
From a research perspective the issues of broadcast Ultra HD are well understood. In conjunction with we ran an Ultra HD Digital Terrestrial trial at the Brazil 2014 World Cup and have made . And we鈥檝e had great co-operation from the production and outside broadcast community as we learn how to make good high dynamic range Ultra HD with different workflows.
But the focus of this trial was streaming Ultra HD over the Internet, where we offered a broader range of content and, with the addition of HDR and wider colour gamut, at a higher quality. Matthew Postgate, the 大象传媒's Chief Technology and Product Officer, has set out . Colleagues are working on a whole host of potential new services which only the Internet can support but, for this project, it meant delivering better picture quality and bringing deeper immersion, over the Internet.
To test Internet delivery we need scale, and scale only comes from there being compelling content that viewers wish to watch. It is only then that we can see how the Internet reacts to carrying high bit rate live Ultra HD streams. The and provided scale much greater than our earlier testing and hence why we had some of the issues that Jim and Lloyd reported.
Even with broadcasting there is complex functionality in TVs and set top boxes which have an important influence on the picture quality. With Internet delivery the coupling is even greater, and the trial was again designed to allow us to work with consumer equipment (CE) manufacturers to develop the capability to deliver a standards based approach to live Ultra HD via iPlayer. We were very pleased with the response from the CE industry in the large number of compatible devices we could enable.
The final purpose of the trial was to show that 大象传媒 iPlayer can be used to deliver new formats. We had shown previously we could deliver on-demand Ultra HD with Blue Planet II, and now we wanted to show that live Ultra HD was also possible using today鈥檚 infrastructure and TV sets.
The Trial
The trial was designed to bring together three key areas. Naturally, we wanted to demonstrate live end-to-end Ultra HD, but we have always felt that Ultra HD needed to be more than just extra pixels. So we also wanted to demonstrate a wide colour gamut and the that the 大象传媒 and have standardised. This is essential to improving the visual experience irrespective of the viewer鈥檚 screen size. Finally, we wanted to do this free-to-air, streamed to 大象传媒 iPlayer, at a scale never seen before in the UK.
By bringing together live Ultra HD production, the Ultra HD wide colour gamut, the HLG HDR signal format and Internet streaming at scale, we set ourselves quite some problems to solve!
Live
Whilst there is significant industry experience in live Ultra HD production, there is very little which includes wide colour gamut and HDR. Different events use different workflows, which means different cameras - and so on. All of these means operating equipment in modes not normally used, hence the need to make sure the whole production chain works. The World Cup is a complex multi venue outside broadcast and used a 鈥淪-Log3-Live鈥� workflow from Sony. Wimbledon is a single venue and we were able to use a native HLG workflow based upon our experience from the Royal Wedding.
The next challenge was encoding live Ultra HD. This is where the differences between live and on-demand Ultra HD programmes really become apparent. We use a video compression standard called High Efficiency Video Coding (HEVC), which is reasonably well established for on-demand. With on-demand, encoding times are not generally an issue and it is possible to get good picture quality at bit rates between 18-23Mbit/s. With PC-based encoders our current 鈥渞ule of thumb鈥� is 20 hours encode time for 1 hour of content in order to get a good set of different qualities and bit rates for an adaptive bit rate (ABR) ladder.
Obviously for live this is completely impractical and encoders capable of operating in real-time to provide multiple encodings must be used. We used GPU accelerated software based encoders but in order to maintain picture quality we had to use much higher bit rates. Our full Ultra HD resolution needed 36Mbit/s, as encoding an ABR ladder also brings an additional bit rate penalty beyond encoding a single representation.
For these reasons it is very hard to draw comparisons between on-demand Ultra HD content from the likes of Netflix, Amazon, or our Blue Planet II trial last year. The demands are much greater for live because all the processing needs to be done in real-time, and requires users to have a much higher speed connection to the Internet, as the live bit rates are significantly higher. Additionally all viewing is co-timed so downstream systems have to scale to much greater capacity than for on-demand.
HDR Production
Wimbledon gave us the opportunity to further test our HLG workflows that we had been developing through trial events and the Royal Wedding. Live television production requires constant adjustment of the cameras by vision engineers to ensure the next shot from the camera is correctly exposed. There is no opportunity for the colour grading and picture adjustments that form a key part of any non-live workflow. We invented HLG so TV production, and particularly live, could continue to be made in the same way.
The experiences of the trials, Royal Wedding and now Wimbledon has shown how well HLG works in a live outside broadcast. We have found some minor camera and camera control issues that we will work with the manufacturers to resolve. We have found areas where potentially the extra colour gamut is going to have to make us and audiences re-think what the correct colour of grass is! We have shown that we can simultaneously produce an SDR and HDR variant. We think that moving to a HDR based production with possible automatic conversion to provide a SDR variant for HD is entirely possible. All this has vindicated the and the .
The World Cup used a Sony S-Log3-Live workflow. For this the 大象传媒 took the S-Log3-Live feed from and converted it using our own technology to HLG. For both events there was a mix of cameras, and not all of them are Ultra HD. Also replays are often 1080p50 to maintain recording capacity, and studio augmented reality sets are rendered HD.
This means that a variety of different resolutions and standard dynamic range sources which need to be well matched. This mix of cameras is not unusual when the industry is adopting new standards. Large numbers of cameras are needed and there are limited camera positions available, so it鈥檚 inevitable that not all specialist cameras will support the new standards. It is clear that some viewers are good at spotting the different cameras.
Internet Streaming
We don鈥檛 believe anyone has quite tried to do Ultra HD at scale over the top (as opposed to a managed approach) for such an event. Our first consideration had to be to ensure that the normal full range of 大象传媒 Internet services weren鈥檛 going to be affected by the additional high bit rate Ultra HD streams.
Secondly, we needed to ensure we worked within the capabilities of the commercial content delivery networks (CDNs) we use. From sales data we could analyse with a reasonable degree of accuracy the number of Ultra HD compatible TVs in the UK, and we had further platform data from some operators. What we didn鈥檛 know is how many of you would watch the events nor how many of the TVs were connected to a suitable high speed Internet connection.
Hence we needed some means of controlling numbers and felt capping access on a first-come, first-served basis was the safest, and fairest, way of doing it. I鈥檓 glad to say this was only briefly used once for controlling access and once when we needed to move traffic between CDNs. Where we did use the cap we think very little growth was left and very few would have been affected.
The majority of viewers were receiving one of the two Ultra HD resolutions. This meant that we were routinely shipping over 1Tbit/s of Ultra HD content.
What has been interesting from the viewer feedback has been the concept of the Internet and how it works. We have had many reasonable comments along the lines 鈥淚 have a 80Mbit/s Internet connection, I suffered buffering, that must be the 大象传媒 not having enough capacity鈥�. The Internet is a complex interconnection of different networks with peering links between Internet Service Providers. ISPs have different architectures for connecting their customers to the wider Internet. Some ISPs have deployed nodes from the CDNs. Finally, local contention for capacity can be an issue.
All this means live over-the-top Internet distribution is very much more complex than broadcast and offers variable end-to-end capacity that some ABR implementations have not been able to fully cope with.
Out of interest I estimated that if all the viewers of the England v Croatia game had watched in the full Ultra HD resolution then we would need around 1 Pbit/s, that is around a 1000 times greater than we were delivering. But we are at the very beginning of this journey and improvements in encoding will have a big impact on reducing this.
Every codec we have worked with follows a curve of improved compression efficiency. Live HEVC encoding has a long way to go before it achieves the full potential of the codec; the on-demand bit rates show that a halving should be possible. We will continue to work with the encoding suppliers to develop the capabilities in this area. We already use cloud-based solutions for SD and HD and we will look at Ultra HD offerings and how they fit into a live end-to-end distribution architecture. New codecs are on the horizon and these propose better encoding efficiency than HEVC but with increased computational requirements.
We also continue to work on approaches for multicast delivery, which can also help reduce the overall load, and are participating in . Ultimately, we are in the hands of ISPs in deploying the necessary technology and are open to trials with ISPs. As many of you know multicast means that we no longer rely on each person鈥檚 device requiring an individual stream, which massively reduces the amount of capacity an Ultra HD live stream would consume in the UK Internet infrastructure. Managed solutions are deployed in some ISPs so the benefits of the technology are understood, the challenge is around OTT access and solving the issues this brings.
Your Feedback
I鈥檇 like to thank all of you that have provided feedback and I鈥檝e been able to read much of it. There鈥檚 no doubt this trial - with its combination of Ultra HD, wide colour gamut and HDR - produced some of the most spectacular live images the 大象传媒 has ever shown. And, overall, people were delighted with the increase in quality. But as we鈥檇 expect in a first-of-its-kind trial like this, there were reports of issues and I鈥檇 like to touch on a few here, as well as what we are doing about them.
Unsurprisingly the delay of the Ultra HD compared to broadcast and posts on social media was a very common theme. We recognise this as a real problem that needs solving for all streamed content. Some of the delay is inherent to the current streamed media formats and how end devices buffer segments to manage network variability.
It is fair to say that this is recognised as an industry issue for live events (although we're looking at ways to optimise our internal processing too) and we are participating in DVB work to look at low latency streaming. New industry standards such as the can shorten streaming latencies, multicast can also help and CDN suppliers are also looking at techniques to improve latencies. For Internet broadcasting to supplant broadcast then the latencies need to be much improved. My R&D team hopes to be demonstrating some advances in this area at IBC 2018 for those of you visiting.
There was a great deal of variability reported for the same event. One viewer would report perfect viewing, another with the same TV would report constant buffering. Sometimes this could be traced to the use of Wi-Fi or an insufficiently fast ISP connection. But there is more going on and we think that some devices are very sensitive to the overall network conditions. My own experience reflected this and sometimes one device would work fine, other times it would buffer, yet another device would then work fine and I could see no obvious local network issues. We have captured some network traces and now need to better understand exactly what is happening and then work with the CEs on their implementations.
A few comments regarding frame rates and motion were also received. This surprised us as both trials used 50 frames per second for all distributed content, with a few up-converted HD 25i (50 fields per second) sources. Research continues into the use of 100 frames per second for UHD but this is still some way from reaching any distribution trials.
Some of you rightly commented on the lack of audio description and subtitles, and this would be an essential part of any 大象传媒 service. Unfortunately it was not possible for this trial but we are working on it. In DVB, the 大象传媒 chaired the technical group specifying technical standards for the delivery of UHD subtitles. We will continue to work with CEs to prototype and develop these standards.
Our intention had been to provide 5.1 audio, but as previously explained we experienced issues with receivers that could have resulted in no audio so regrettably we had to resort to stereo audio. Again we will develop further tests and work with CEs to enable a reliable 5.1 audio experience.
The final theme was around dullness of pictures and lack of colour reported by some viewers. Again, this is something I鈥檝e seen on different TVs, sometimes one TV has been fine and another with the same content has shown the issues. Andrew Cotton has written a post which examines some of the potential issues at the TV end. By design HLG produces picture with colours that more accurately reproduce the scene that vision engineers and craft skills prefer. We need to work with CEs to understand why these aren鈥檛 always showing on TVs.
There are some quite complex colourimetry issues to work through, for example we don鈥檛 fully understand yet why the grass under sunlight at Wimbledon can have a blue tinge. Is this an equipment issue or is this a more accurate representation of reality and hence a benefit of wide colour gamut and HDR? We just don鈥檛 know yet. My eyes could see some blue in the grass in strong sunlight when I visited Wimbledon, although the SDR pictures don鈥檛 show this. It is probably a combination of issues.
The dullness of pictures is a more complex issue. We are aware of workflow issues that have had an impact and we have tried to improve these over the course of the events. This is all part of the learning on how to make good HDR and we will have to continue to develop to get greater consistency. For the TV, HDR is about how the extra brightness beyond what is required for SDR is used. Some TVs don鈥檛 seem to be using their available brightness, as I鈥檝e seen the same image looking quite dull on a 1000Nit display yet sparkling on a 600Nit display. Again this is something we will work on with the CE industry to understand the issues.
What comes next?
The project has left the 大象传媒 with the ongoing capability to receive an incoming Ultra HD contribution, and to encode, package and distribute via commercial CDNs. Over time we will add our own CDN to this. We鈥檝e worked with the production community to build knowledge of producing HDR content. Whilst not quite business-as-usual, the 大象传媒 now has the capability for streaming live Ultra HD content into 大象传媒 iPlayer on compatible devices.
The trial revealed issues around latency and buffering that were expected. However, it is clear from the feedback that a great number of viewers enjoyed seeing football and tennis in Ultra HD with wide colour gamut and high dynamic range.
The trial has been extremely useful for moving live Ultra HD production and distribution forward. We clearly have some issues to work on which is to be expected from a trial of this nature, but our overall summary is that the trial has been a great success. We have managed to go from a handful of devices in laboratories to full scale Internet delivery of Ultra HD in less than 6 months. Each test has grown in scale and we very much hope we can continue the trajectory.
I鈥檇 like to thank not just all the teams in the 大象传媒 that have contributed to this project but also all the external companies from outside broadcast, professional equipment suppliers, CDNs through to consumer equipment manufacturers who have combined to make this work.
- from the 大象传媒 Academy including: