O QoE, Where Art Thou? – ACM SIGMM Records

Authors and Affiliation: Christian Timmerer (Alpen-Adria-Universität (AAU) Klagenfurt and Bitmovin Inc., Austria) and Tobias Hoßfeld (University of Würzburg, Germany)

Once upon a time, when engineers measured networks in latency and packet loss, the idea of Quality of Experience (QoE) emerged — a myth whispered among researchers who dared to ask not what the system delivers, but what the user perceives. Decades later, QoE has evolved into a sprawling epic, spanning disciplines and domains, from humble MOS scores to immersive virtual realities. But as media experiences become ever more complex — adaptive, interactive, personalized — the question lingers: O QoE, where art thou?

1. Introduction

In this column, we revisit the notion of QoE and its evolution over time. We begin by reviewing early work from the 1990s to 2000s on the definitions of QoE (Section 2), where researchers first recognized the importance of user perception and the relevant QoE influence factors, as well as QoE modeling efforts. As a summary of this literature survey, QoE evolved from abstract notions of perception and satisfaction to a measurable, standardized concept encompassing the emotional, cognitive, and contextual responses of users to a service or application. The trends across time are:

1990s: Early focus on perception and interaction design.
Early 2000s: Growing focus on subjectivity, emotion, and context in user experience. QoE separated from QoS, emphasizing emotion, context, and expectation. Seen as key to commercial and user success.
Mid-2000s: Integration of technical and perceptual layers; need for metrics and quantification. Push for measurable models combining technical and user perspectives. Recognition of multiple definitions across domains.
Late 2000s–2010s: Standardization, recognition of multi-dimensionality, and development of cross-disciplinary definitions. QoE defined around subjective perception and system-wide impact.
2010s: Unified, multidisciplinary understanding established through initiatives like QUALINET; QoE as “delight or annoyance”.

This initial insight laid the foundation for larger initiatives like QUALINET, which helped to shape the field by providing widely accepted QoE definitions. We then examine how these developments have been formalized through standardization activities (Section 3), particularly within the ITU and the QUALINET whitepapers on the definition of QoE and immersive QoE.

The diverse and often conflicting definitions of QoE emerging in the 2000s highlighted the need for coordinated efforts and shared understanding across disciplines. This led to joint initiatives like QUALINET, which aimed to formalize and unify QoE research within a dedicated network. One of the results is the updated QoE definition, which is now taken in standardization.

2016: ITU-T Recommendation P.10/G.100 (2006) Amendment 5 (07/ 16), New Definitions for Inclusion in Recommendation ITU-T P.10/G.100, International Telecommunication Union, July 2016. ‘‘Quality of experience (QoE) is the degree of delight or annoyance of the user of an application or service’’.

Figure 1. Timeline on the notion and definitions of QoE in literature and standardization.

A timeline of the literature survey and the early definitions of QoE as well as the standardization activities is visualized in Figure 1. Finally, we discuss selected open issues in QoE research (Section 4) that continue to challenge both academia and industry.

2. Early Definitions of QoE: 1990s to 2000s

The term Quality of Experience (QoE) emerged in the late 1990s to early 2000s as a response to the limitations of traditional network-centric approaches. Although Quality of Service (QoS) had already been formally defined in ITU-T Recommendation E.800 (1994) [ITU-T E.800] for telephony and established a basis for assessing service quality from both technical and user viewpoints, QoS primarily addresses performance at the network level. QoS is commonly applied within communication networks to describe a system’s ability to meet predefined performance targets, ensuring consistent data transmission through metrics such as bandwidth, latency, jitter, and packet loss [Varela2014].

In contrast, researchers and industry practitioners began to recognize the importance of how users actually perceive the quality of a service in the late 1990s to early 2000s. In this context, a variety of alternative terms were used prior to the standardization and definition of QoE, including User-Perceived Quality, Perceived Quality, End-User Quality, User-Experience Quality, Multimedia Experience Quality, Subjective Quality of Service, and user-level QoS. These early terms reflected a growing awareness of the need to evaluate digital services from the user’s point of view, ultimately leading to the coining and adoption of QoE as a distinct and essential concept in the field of communication systems and multimedia applications.

The term QoE brought attention to the user’s subjective perception, marking a shift toward evaluating service quality from the end-user’s perspective in the mid of 2000s. In the following, a brief overview on first documents about “Quality of Experience” or “QoE” are provided to sketch the definition of terms. In particular, research articles from the ACM Digital Library and IEEE Xplore searching for “Quality of Experience” or “QoE” are collected.

Focus on user perception and interaction design

1990: Harman, G. “The intrinsic quality of experience.“ laims we’re not directly aware of our experiences’ intrinsic properties, but of those of the external objects they represent—like color, shape, texture, motion, and spatial relations.
1996: Austin Henderson. “What’s next?” explains the idea behind the ACM Award about QoE in interaction. “We really want to know what users experience! In short we are interested in the quality of a person’s experience in the interaction. […] factors contribute to the effective experience of interacting with the device.“ However, no QoE definition is proposed.
1996: Lauralee Alben. “Quality of experience: defining the criteria for effective interaction design“ is also related to the ACM interactions design award. “By ‘experience’ we mean all the aspects of how people use an interactive product: the way it feels in their hands, how well they understand how it works, how they feel about it while they’re using it, how well it serves their purposes, and how well it fits into the entire context in which they are using it. If these experiences are successful and engaging, then they are valuable to users and noteworthy to the interaction design awards jury. We call this ‘quality of experience’.” This early definition of QoE encompasses all aspects of a user’s interaction with a product, including its physical feel, usability, emotional impact, and the overall satisfaction derived from its use.
2000: Alan Turner and Lucy T. Nowell. “Beyond the desktop: diversity and artistry” relate QoE to the need for engaging, media-rich interactions across diverse devices, emphasizing the role of artistry in delivering compelling user experiences. A remarkable statement: “We also believe that the quality of experience will become the key metric of success for software, both commercially and socially.“

Focus on subjectivity, emotion, and context

2000: Marion Buchenau and Jane Fulton Suri. “Experience prototyping.” introduce a prototyping approach that immerses users in simulated interactions to explore and refine QoE, including sensory, emotional, and contextual dimensions beyond usability or function. QoE goes beyond usability or functionality, encompassing emotional and contextual factors.
2000: Anna Bouch, Allan Kuchinsky, and Nina Bhatti. “Quality is in the eye of the beholder: meeting users’ requirements for Internet quality of service.” They show that in Internet commerce, QoE depends on both technical QoS as well as user expectations and context. “Only through such integration of users’ requirements into systems design [of users’ requirements into systems design] will it be possible to achieve the customer satisfaction that leads to the success of any commercial system.”
2001: Public slide set by Touradj Ebrahimi (2012) “Quality of Experience Past, Present and Future Trends”, presented 23 Nov 2012, refers to a definition of QoE as follows. “The degree of fulfillment of an intended experience on a given user – as defined by Touradj Ebrahimi, 2001”.
2002: Heddaya, A. S. “An economically scalable Internet” uses the term “QoE rather than quality of service because QoS is not necessary for QoE, and QoE is sufficient for successful service.”

Focus on measurable models combining technical and user perspectives

1994: Nahrstedt, K., & Smith, J., Ralf Steinmetz. “Mapping User Level QoS from a Single Parameter” aims at quantifying QoE. “The ‘satisfaction’ concept has been introduced to quantify the QoS provided by the system. The transformations required to both map the cost into satisfaction and then configure the system are then developed.”
2003: Siller, M., & Woods, J. C. “QoS arbitration for improving the QoE in multimedia transmission.” propose a QoE-aware framework that adapts QoS to real-time user perception for multimedia networks. They define QoE as “the user’s perceived experience of what is being presented by the Application Layer, where the application layer acts as a user interface front-end that presents the overall result of the individual Quality of Services”.
They also review current related work at that time, which are taken from white papers, which are not accessible anymore:
- “A metric used for measuring the performance of this perceptual layer is Quality of Experience (QoE).”
- “QoE is referred to as; what a customer experiences and values to complete his tasks quickly and with confidence.”
- “QoE is considered as all the perception elements of the network and performance relative to expectations of the users/subscribers.“
- The QoE is defined as “the totality of the Quality of Service mechanisms, provided to ensure smooth transmission of audio and video over IP networks”.
2004: R. Jain. “Quality of Experience” asks the following questions. “But how do we quantitatively define the quality of experience? Can we extend QoS to QoE? What factors should we consider in developing measures for QoE?” He concludes with a remarkable statement. “In a sense, the challenges of QoE are nothing new. People in social sciences and marketing have always developed techniques to quantify people’s preferences and choices. That situation is similar to what goes into QoE.”
2004: Euro-NGI D.JRA.6.1.1 “State-of-the-art with regards to user-perceived Quality of Service and quality feedback” with Fiedler as lead for this deliverable reviews QoS from the user’s perspective. The notion of QoE is “The degree of satisfaction, i.e. the subjective quality, is influenced by the technical, objective quality stemming from the application and the interconnecting network(s). For this reason, subjective quality as perceived by the network has to be linked to objective, measurable quality, which is expressed in application and network performance parameters. “
2007: Hoßfeld, Tobias, Phuoc Tran-Gia, and Markus Fiedler. “Quantification of quality of experience for edge-based applications” provide a quantitative link between technical metrics and QoE. “Quality of Experience (QoE), a subjective measure from the user perspective of the overall value of the provided service or application”.

Diversity of definitions and interdisciplinarity

2007: Soldani, D., Li, M., & Cuny, R. “QoS and QoE management in UMTS cellular systems” define: “QoE is the term used to describe the perception of end-users on how usable the services are. […] The term ‘QoE’ refers to the perception of the user about the quality of a particular service or networks.” Notably, they already mentioned that “Browsing through the literature, one may find many different definitions for quality of end-user experience (QoE) and quality of service (QoS).”
2009: International Conference on Quality of Multimedia Experience (QoMEX) includes in the call for papers: “perceived user experience is psychological in nature and changes in different environmental conditions and with different multimedia devices.”

3. Definitions of QoE in Standardization

In standardization, the following definitions were introduced.

2007: ITU-T Rec. G.100/P.10 Amendment 1 (2007) New Appendix I – Definition of Quality of Experience (QoE). “The overall acceptability of an application or service, as perceived subjectively by the end user. NOTE 1: Quality of experience includes the complete end-to-end system effects (client, terminal, network, services infrastructure, etc.). NOTE 2: Overall acceptability may be influenced by user expectations and context.”
This definition has been superseded by the Qualinet Definition of QoE in 2016. It should be mentioned that acceptance and QoE are different concepts. acceptability refers more narrowly to whether a service or system is deemed “good enough” or usable under certain conditions. Approaches to link QoE and acceptance have been discussed in literature [Schatz2011,Hossfeld2016].
2008: ITU-T Recommendation E.800. “Definitions of terms related to quality of service” defines in as follows: “quality of service experienced/perceived by customer/user (QoSE): a statement expressing the level of quality that customers/users believe they have experienced. NOTE 1: The level of QoS experienced and/or perceived by the customer/user may be expressed by an opinion rating.”
2009: ETSI TR 102 643 V1.0.1 (2009-12) “Human Factors (HF); Quality of Experience (QoE) requirements for real-time communication services” defines QoE as “measure of user performance based on both objective and subjective psychological measures of using an ICT service or product”. It includes two notes on QoE: (1) Considers technical QoS, context, and measures both communication process and outcomes (e.g. effectiveness, satisfaction). (2) Uses objective (e.g. task time, errors) and subjective (e.g. perceived quality, satisfaction) psychological measures, depending on context.

2016: ITU-T Recommendation P.10/G.100 (2006) Amendment 5 (07/ 16), New Definitions for Inclusion in Recommendation ITU-T P.10/G.100, International Telecommunication Union, July 2016: ‘‘Quality of experience (QoE) is the degree of delight or annoyance of the user of an application or service’’.

QUALINET White Paper on Definitions of Quality of Experience

QUALINET is the European Network on Quality of Experience in Multimedia Systems and Service (COST Action IC 1003 from 2010 to 2014, later a network that meets regularly at QoMEX) with the aim to “to establish a strong network on Quality of Experience (QoE) with participation from both academia and industry” (https://www.cost.eu/actions/IC1003/). QUALINET was the driving force to further advance research in the context of QoE, producing three major, well-cited assets (among others), namely (1) QUALINET White Paper on Definitions of Quality of Experience, (2) QUALINET databases [QUALINET2019], and (3) QUALINET White Paper on Definitions of Immersive Media Experience (IMEx)

The white paper on definitions of QoE was the result from a consultation and collaborative writing process within the COST Action IC 1003 of 38 authors, contributors, and editors from 18 countries. A first draft was discussed and improved at the 2012 QoE Dagstuhl Seminar [Fiedler2012]. The final definition of QoE:

“Quality of Experience (QoE) is the degree of delight or annoyance of the user of an application or service. It results from the fulfillment of his or her expectations with respect to the utility and / or enjoyment of the application or service in the light of the user’s personality and current state.”
[QUALINET2013]

The white paper also defines influence factors (human, system, context) and features of QoE (level of direct perception, level of interaction, level of the usage situation, level of service) as well as the relationship between QoS and QoE, plus application areas, which allow “to provide specializations of a generally agreed definition of QoE pertaining to the respective application domain taking into account its requirements formulated by means of influence factors and features of QoE”.

QUALINET White Paper on Definitions of Immersive Media Experience (IMEx)

A follow-up white paper defines the QoE for immersive media as

“the degree of delight or annoyance of the user of an application or service which involves an immersive media experience. It results from the fulfillment of his or her expectations with respect to the utility and/or enjoyment of the application or service in the light of the user’s personality and current state.”
[QUALINET2020]

IMEx is defined as

“a high-fidelity simulation provided and communicated to the user through multiple sensory and semiotic modalities. Users are emplaced in a technology-driven environment with the possibility to actively partake and participate in the information and experiences dispensed by the generated world.”
[QUALINET2020]

Consequently, this white paper provides a “toolbox for definitions of IMEx including its Quality of Experience, application areas, influencing factors, and assessment methods.” [QUALINET2020].

4. Open Issues in QoE Research

We would like to conclude with some open issues regarding Quality of Experience. The upcoming 6G standard presents significant opportunities, such as QoE-aware orchestration of edge computing, cloud rendering, and network slicing [Tondwalkar2024] and native AI in 6G [Ziegler2020], while also considering tradeoff between QoE and CO2 emissions [Hossfeld2023]. As AI-generated content continues to rise, the evaluation of its quality remains in its early stages. The same applies to learning-based codecs, where existing quality assessment methods—both objective and subjective—are reaching their limits, particularly concerning media authenticity, which is becoming a critical issue. In this context, ethics and privacy are paramount, as user data plays a central role in QoE modeling. Future research must focus on privacy-preserving methods for QoE measurement and personalization. Finally, new modalities such as point clouds, light fields, and holograms necessitate the adaptation of existing techniques or the development of new methods. Moreover, multimodal or multisensory QoE, particularly concerning audio-visual-haptic or olfactory integration (previously referred to as Mulsemedia), is emerging as an important area that requires tailored QoE assessment methods and metrics. This is also reflected by the upcoming 17^th Int. Conf. on Quality of Multimedia Experiences (QoMEX’25) under the theme “Thinking of a QoE ®evolultion”. In particular, the call for papers requests: “On the edge of QoMEX ‘coming of age’, it is time to rethink the purpose and methods of QoE research: cross-fertilizing with adjacent fields, reaching more diverse populations, or exploring novel techniques and paradigms.” This addresses innovative approaches and novel paradigms in QoE research, technological innovations in the era of big data data and AI, but also on user-centricity in 6G. Interdisciplinary links in QoE include diversity, ethics, accessibility, but also novel interaction techniques and multimedia experiences. Specific applications such as gaming, healthcare, education, and immersive technologies, and multisensory perception are in the scope.

And so, like any true odyssey, the search for Quality of Experience continues — not as a destination, but as a path we shape with every interaction, every pixel tuned, every user understood. QoE is no longer a myth, but neither is it fully found. It lives at the intersection of perception and precision, where engineers meet psychologists, and systems learn to listen. In a world of immersive media and intelligent networks, perhaps the better question is no longer “O QoE, where art thou?” but rather — “Are we ready to meet it where it truly resides?”

References

[Alben1996] Lauralee Alben. 1996. Quality of experience: defining the criteria for effective interaction design. interactions 3, 3 (May/June 1996), 11–15. https://doi.org/10.1145/235008.235010
[Bouch2000]: Anna Bouch, Allan Kuchinsky, and Nina Bhatti. 2000. Quality is in the eye of the beholder: meeting users’ requirements for Internet quality of service. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (CHI ’00). Association for Computing Machinery, New York, NY, USA, 297–304. https://doi.org/10.1145/332040.332447
[Buchenau2000] Marion Buchenau and Jane Fulton Suri. 2000. Experience prototyping. In Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques (DIS ’00). Association for Computing Machinery, New York, NY, USA, 424–433. https://doi.org/10.1145/347642.347802
[Ebrahimi2001] Public slide set by Touradj Ebrahimi (2012) “Quality of Experience Past, Present and Future Trends”, presented at Alpen-Adria-Universität Klagenfurt, 23 Nov 2012
ETSI TR 102 643 V1.0.1 (2009-12) “Human Factors (HF); Quality of Experience (QoE) requirements for real-time communication services”
[EuroNGI2004] Euro-NGI D.JRA.6.1.1 : State-of-the-art with regards to user-perceived Quality of Service and quality feedback, Deliverable version No: 1.0 Sending date: 31/05-2004, Lead: Markus Fiedler, BTH Karlskrona. <a href=”https://www.diva-portal.org/smash/get/diva2:837296/FULLTEXT01.pdf”>Last accessed: 2025/04/22</a>
[Fiedler2012] Markus Fiedler, Sebastian Möller, and Peter Reichl. Quality of Experience: From User Perception to Instrumental Metrics (Dagstuhl Seminar 12181). In Dagstuhl Reports, Volume 2, Issue 5, pp. 1-25, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2012) https://doi.org/10.4230/DagRep.2.5.1
[Harman1990] Harman, G. (1990). The intrinsic quality of experience. Philosophical perspectives, 4, 31-52. https://doi.org/10.2307/2214186
[Heddaya2002] Heddaya, A. S. (2002). An economically scalable Internet. Computer, 35(9), 93-95. https://doi.org/10.1109/MC.2002.1033035
[Henderson1996] Austin Henderson. 1996. What’s next?—growing the notion of quality. Interactions 3, 3 (May/June 1996), 56–59. https://doi.org/10.1145/235008.235019
[Hestnes2009] Hestnes, B., Brooks, P., Heiestad, S. (2009). “QoE (Quality of Experience) – measuring QoE for improving the usage of telecommunication services”, Telenor R&I R 21/2009.
[Hossfeld2007] Hoßfeld, Tobias, Phuoc Tran-Gia, and Markus Fiedler. “Quantification of quality of experience for edge-based applications.” International Teletraffic Congress. Berlin, Heidelberg: Springer Berlin Heidelberg, 2007. https://doi.org/10.1007/978-3-540-72990-7_34
[Hossfeld2016] Hoßfeld, T., Heegaard, P. E., Varela, M., & Möller, S. (2016). QoE beyond the MOS: an in-depth look at QoE via better metrics and their relation to MOS. Quality and User Experience, 1, 1-23. https://doi.org/10.1007/s41233-016-0002-1
[Hossfeld2023] Hoßfeld, T., Varela, M., Skorin-Kapov, L., & Heegaard, P. E. (2023). A Greener Experience: Trade-Offs between QoE and CO 2 Emissions in Today’s and 6G Networks. IEEE communications magazine, 61(9), 178-184. https://doi.org/10.1109/MCOM.006.2200490
[ITU-T E.800] E.800: Terms and definitions related to quality of service and network performance including dependability”. ITU-T Recommendation. August 1994. Updated September 2008 as Definitions of terms related to quality of service. Last access: 2025/04/22
[ITU-T G.100/P.10 2007] ITU-T Rec. G.100/P.10 Amendment 1 (2007) New Appendix I—Definition of Quality of Experience (QoE). International Telecommunication Union, Geneva.
[Nahrstedt1994] Nahrstedt, K., & Smith, J., Ralf Steinmetz (Ed), 1994, “Service Kernel for Multimedia Endpoints”, Multimedia: Advanced Teleservices and High-speed Communication Architectures, Lecture Notes in Computer Science LNCS868, chanter I, pp. 8-22, Springer Verlag. https://doi.org/10.1007/3-540-58494-3_2
[QUALINET2013] Patrick Le Callet, Sebastian Möller and Andrew Perkis, eds., Qualinet White Paper on Definitions of Quality of Experience (2012). European Network on Quality of Experience in Multimedia Systems and Services (COST Action IC 1003) Lausanne, Switzerland, Version 1.2, March 2013. Last access: 2025/04/22
[QUALINET2019] Karel Fliegel, Lukáš Krasula, and Werner Robitza. 2022. Qualinet databases: central resource for QoE research – history, current status, and plans. SIGMultimedia Rec. 11, 3, Article 5 (September 2019), 1 pages. https://doi.org/10.1145/3524460.3524465
[QUALINET2020] Perkis, A., Timmerer, C., et al., “QUALINET White Paper on Definitions of Immersive Media Experience (IMEx)”, European Network on Quality of Experience in Multimedia Systems and Services, 14th QUALINET meeting (online), May 25, 2020. https://arxiv.org/abs/2007.07032
[Richards1998] Richards, A., Rogers, G., Witana, V., & Antoniades, M., 1998, “Mapping User Level QoS from a Single Parameter”, In Proceedings of the International Conference on MultimediaNetworks and Services (MMNS ‘98).
[Schatz2011] Schatz, R., Egger, S., & Platzer, A. (2011, June). Poor, good enough or even better? bridging the gap between acceptability and qoe of mobile broadband data services. In 2011 IEEE International Conference on Communications (ICC) (pp. 1-6). IEEE. https://doi.org/10.1109/icc.2011.5963220
[Siller2003] Siller, M., & Woods, J. C. (2003, July). QoS arbitration for improving the QoE in multimedia transmission. In International Conference on Visual Information Engineering (VIE 2003). Ideas, Applications, Experience (pp. 238-241). London UK: IEE. https://doi.org/10.1049/cp:20030531
[Soldani2006] Soldani, D., Li, M., & Cuny, R. (Eds.). (2007). QoS and QoE management in UMTS cellular systems. John Wiley & Sons. https://doi.org/10.1002/9780470034057
[Tondwalkar2024] Tondwalkar, A., Andres-Maldonado, P., Chandramouli, D., Liebhart, R., Moya, F. S., Kolding, T., & Perez, P. (2024). Provisioning Quality of Experience in 6G Networks. IEEE Access. https://doi.org/10.1109/ACCESS.2024.3455938
[Turner2000] Alan Turner and Lucy T. Nowell. 2000. Beyond the desktop: diversity and artistry. In CHI ’00 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’00). Association for Computing Machinery, New York, NY, USA, 35–36. https://doi.org/10.1145/633292.633317
[Varela2014] Varela, M., Skorin-Kapov, L., & Ebrahimi, T. (2014). Quality of service versus quality of experience. In Quality of Experience: Advanced Concepts, Applications and Methods (pp. 85-96). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-02681-7_6
[Ziegler2020] Ziegler, V., Viswanathan, H., Flinck, H., Hoffmann, M., Räisänen, V., & Hätönen, K. (2020). 6G architecture to connect the worlds. IEEE Access, 8, 173508-173520. https://doi.org/10.1109/ACCESS.2020.3025032