Impact of Conversation Task When Testing QoE in WebRTC

Abstract

Video conferencing was previously associated with dedicated hardware and professional conferencing systems. The last years applications and services enabling Real-Time Communication with no need for additional software support, called WebRTC, have become more popular. Variety in telemeetings is huge, and the increasing complexity makes it challenging to evaluate telemeeting Quality of Experience (Qoe), by selecting one approach that is valid for all types of equipment and circumstances. Many studies have been conducted to achieve an understanding of which factors play a role/have an impact on telemeetings. Several of these have indicated that system factors alone are not sufficient to understand the QoE of an individual in a specific situation.

Previous studies have put participants in an experimental environment and given them a conversation task to simulate a natural conversation. The results from some of these earlier studies indicate that the chosen task have an impact when the test subject rates the QoE, and that the tolerance towards technical impairments may differ, depending on the task at hand. Given this context, the main motivation for this thesis is to contribute to literature in this respect, by investigating whether and how the conversation task influences QoE under different circumstances.

I have conducted an experiment by using a QoE-testbed. Participants were exposed to four different technical quality conditions (good quality, distorted audio and video, distorted audio and distorted video). Two conversation tasks were included in the experimental design, namely the LEGO-task (building blocks task) and the Free Conversation task. Data gathered in a previous study, using the Celebrity Name Guessing task, were also integrated into the dataset, in order to compare the findings.

The results of this study point in the direction that the task does matter and does influence QoE. When it comes to the traditional perceived quality ratings, there are significant differences when comparing the different conditions within each task, but not between the different tasks. Still, there are some tendencies. Overall, the quality ratings are slightly higher for the Free Conversation task than for the other two tasks (LEGO-task and Celebrity Name Guessing). In addition, it could be observed that in most cases, the Free Conversation yields more compact ratings with lower variability among the test subjects. When looking at affective state the test subjects clearly feel the least aroused during the Free Conversation task and more aroused in LEGO-task and Celebrity Name Guessing. A plausible explanation in this respect is the competition element that is inherently in these tasks.

To summarize, the choice of conversation task is complex, and many factors have to be taken in to consideration. There are advantages and disadvantages for both tasks and some differences are clear, while others are more implicit. In general, the Free Conversation task provides higher agreement of the quality ratings, but is clearly less engaging. The LEGO-task and Celebrity Name Guessing task are more engaging and implicitly the tolerance towards impairments seems to be lower.