Measuring perceptual quality in Internet television

Abstract

In this thesis we have evaluated the Quality of Experience (QoE) of Internet television through user tests of Absolutt Fotball. Absolutt Fotball is a Norwegian live football streaming service powered by adaptive streaming from Move Networks Inc. In our tests we found that the users rated the overall quality better than other Internet video services, such as Youtube, TV2 Sumo and NRK Nett-TV, but worse than football on TV. The main problems were coding artifacts, such as blurring, edge ringing and color bleeding, as well as problems with the smoothness of playback. Response time and adaptation period were in general satisfactory; all users preferred adaptive streaming with quick starts and no interruptions over traditional streaming with constant quality and buffering in the start and sometimes during sequences. The tests also revealed that factors other than video quality could have significance in the user s overall QoE. Most notably was the delay from other live services, such as SMS updates, radio and live updates on the Internet. We also found in our analyses tendencies of content and context dependencies to the QoE. E.g. the result of a user s favorite team, as well as his/her viewing environment, could have an impact on his/her perception of the quality. In order to improve the QoE the service provider should evaluate the encoding stage in particular. By increasing the bit rate of the encoding, many of the problems related to coding artifacts and smoothness of playback could be reduced. The client should be optimized with regards to adaptation period, response time and live-delay, however there is a compromise to be made with the robustness and reliability of the media player. The service provider can receive feedback on the QoE in three stages: 1. Full reference objective quality assessment at headend, such as VQM, 2. Bit rate statistics from the clients, and 3. An extended user profile and a QoE tool at user end. The proposed QoE tool in the form of a menu could include guides and tests related to user equipment and viewing environment, real-time feedback and support chat related to video quality problems, and service personalization in relation to quality/price and features. We found that controlling the QoE in Internet television is very difficult. QoE monitoring is however possible for the service provider, but a true end-to-end solution would require a better integration of client and user than is today.