VQEG Column: VQEG Meeting May 2025

Introduction

From May 5^th to 9^th, 2025 Meta hosted the plenary meeting of the Video Quality Experts Group (VQEG) in their headquarters in Menlo Park (CA, United Sates). Around 150 participants registered to the meeting, coming from industry and academic institutions from 26 different countries worldwide.

The meeting was dedicated to present updates and discuss about topics related to the ongoing projects within VQEG. All the related information, minutes, and files from the meeting are available online in the VQEG meeting website, and video recordings of the meeting are available in Youtube.

All the topics mentioned bellow can be of interest for the SIGMM community working on quality assessment, but special attention can be devoted to the first activities of the group on Subjective and objective assessment of GenAI content (SOGAI) and to the advances on the contribution of the Immersive Media Group (IMG) group to the International Telecommunication Union (ITU) towards the Rec. ITU-T P.IXC for the evaluation of Quality of Experience (QoE) of immersive interactive communication systems.

Readers of these columns who are interested in VQEG’s ongoing projects are encouraged to subscribe to the corresponding mailing lists to stay informed and get involved.

Overview of VQEG Projects

Immersive Media Group (IMG)

The IMG group researches on the quality assessment of immersive media technologies. Currently, the main joint activity of the group is the development of a test plan to evaluate the QoE of immersive interactive communication systems, which is carried out in collaboration with ITU-T through the work item P.IXC. In this meeting, Pablo Pérez (Nokia XR Lab, Spain), Marta Orduna (Nokia XR Lab, Spain), and Jesús Gutiérrez (Universidad Politécnica de Madrid, Spain) presented the status of this recommendation and the next steps to be addressed towards a new contribution to ITU-T in its next meeting in September 2025. Also, in this meeting, it was decided that Marta Orduna will replace Pablo Pérez as vice-chair of IMG. In addition, the following presentations related to IMG topics were delivered:

• Gareth Rendle (Bauhaus-Universität Weimar, Germany) and Felix Immohr (TU Ilmenau, Germany) presented a user study on the influence of audiovisual realism on communication behaviour in group-to-group telepresence, showing that avatar realism has positive effects on subjective ratings of perceived message understanding and group cohesion, and yields behavioural differences that indicate more interactivity and engagement. Also, Anton Lammert (Bauhaus-Universität Weimar, Germany), presented his work (in collaboration with Gareth and Felix) on a system designed for the comprehensive analysis of social Virtual Reality (VR) studies, called Immersive Study Analyzer (ISA), which records all user actions, speech, and the contextual environment.
• Kamil Koniuch, Norbert Barczyk, Lucjan Janowski, and Mateusz Olszewski (AGH University of Krakow, Poland) presented their work on developing VR games based on circumplex model of group tasks for Quality of Experience (QoE) measurements.
• Xuemei Zhou (CWI & TU Delft) explained her research on point cloud quality assessment and visual saliency, including a task-free eye-tracking dataset and a task-depedent dataset for dynamic point clouds aimed at investigating visual attention, and the comparison of the task impact.
• Silvia Casino (Nokia XR Lab, Sapin) delivered a presentation on the evaluation (both objective and subjective) of segmentation algorithms for embodiment improvement in an eXtended Reality (XR) application, aiming at guiding the development of more accessible XR applications.
• Patrick Le Callet (NantesUniversité, France) delivered three presentations, the first one on his work, in collaboration with researchers from SJTU (China) on perceptual quality assessment of Vision-Pro-based egocentric spatial images; the second one focused on the study of interactions between vibroacoustic discomfort and visual stimuli, comparing real 3D and 360-degree environments; and the last one described a dataset and a tool for quantitative assessment of functional vision based on orientation and mobility tests in VR.

Statistical Analysis Methods (SAM)

The SAM group investigates on analysis methods both for the results of subjective experiments and for objective quality models and metrics. In relation with these topics, the following presentations were delivered during the meeting:

• Ludo Malfait (BT Group, United Kingdom) presented a study addressing Virtual Private Networks (VPN) and Virtual Private Servers (VPS) users when conducting subjective tests on crowdsourcing platforms.
• Dietmar Saupe (University of Konstanz, Germany) delivered two presentations. The first one covered the updates on the JPEG Assessment of Image Coding (AIC) project, especially the JPEG AIC-3, which is a standard (currently under review at ISO/IEC) for fine-grained subjective assessment of image quality in the high-fidelity range. The second one focused on the robustness and accuracy of Mean Opinion Scores (MOSs) with hard and soft outlier detection and proposed two new outlier detection methods with low complexity and excellent worst-case performance.
• Mohsen Jenadeleh (University of Konstanz, Germany) and Jon Sneyers (Cloudinary, Belgium) presented their work on fine-grained High dynamic range (HDR) image quality assessment, introducing the AIC-HDR2025 dataset, comprising 100 test images generated from five sources with different encoding configurations and presenting the results of a subjective tests with it. In addition, Mohsen also presented his research on subjective visual quality assessment for high-fidelity learning-based image compression, which covered a comprehensive subjective visual quality assessment of JPEG AI-compressed images using the JPEG AIC-3 methodology, which quantifies differences in Just Noticeable Difference (JND) units.
• Panagiotis Traganitis (Michigan State University, United States) presented a unified framework for learning from crowdsourced noisy labels, covering classical and modern methods for aggregating rankings while inferring annotator quality, as well as its application in ranking problems.

Joint Effort Group (JEG) – Hybrid

The group JEG addresses several areas of Video Quality Assessment (VQA), such as the creation of a large dataset for training such models using full-reference metrics instead of subjective metrics. The chair of this group, Enrico Masala (Politecnico di Torino, Italy) presented the updates on the latest activities of the group, including the current results of the Implementer’s Guide for Video Quality Metrics (IGVQM) project. In addition to this, the following presentations were delivered:

• Lohic Fotio Tiotsop (Politecnico di Torino, Italy) presented a software implementing and comparing 7 methods from 7 works in literature that aim at quality recovery in subjective quality assessment experiments, and a non-parametric approach to subjective media quality recovery in the presence of spammer annotators.
• Maria Martini (Kingston University London, United Kingdom) presented her research on quality assessment of transcoded video based on SSIM.

Emerging Technologies Group (ETG)

The ETG group focuses on various aspects of multimedia that, although they are not necessarily directly related to “video quality”, can indirectly impact the work carried out within VQEG and are not addressed by any of the existing VQEG groups. In particular, this group aims to provide a common platform for people to gather together and discuss new emerging topics, possible collaborations in the form of joint survey papers, funding proposals, etc. In this sense, the following topics were presented and discussed in the meeting:

• Avinab Saha (UT Austin, United States) presented the dataset of perceived expression differences, FaceExpressions-70k, which contains 70,500 subjective expression comparisons rated by over 1,000 study participants obtained via crowdsourcing.
• Mathias Wien (RWTH Aachen University, Germany) reported on recent developments in MPEG AG 5 and JVET for preparations towards a Call for Evidence (CfE) on video compression with capability beyond VVC.
• Effrosyni Doutsi (Foundation for Research and Technology – Hellas, Greece) presented her research on novel evaluation frameworks for spike-based compression mechanisms.
• David Ronca (Meta Platforms Inc. United States) presented the Video Codec Acid Test (VCAT), which is a benchmarking tool for hardware and software decoders on Android devices.

Subjective and objective assessment of GenAI content (SOGAI)

The SOGAI group seeks to standardize both subjective testing methodologies and objective metrics for assessing the quality of GenAI-generated content. In this first meeting of the group since its foundation, the following topics were presented and discussed:

• Ryan Lei and Qi Cai (Meta Platforms Inc., United states) presented their work on learning from subjective evaluation of Super Resolution (SR) in production use cases at scale, which included extensive benchmarking tests and subjective evaluation with external crowdsource vendors.
• Ioannis Katsavounidis, Qi Cai, Elias Kokkinis, Shankar Regunathan (Meta Platforms Inc., United States) presented their work on learning from synergistic subjective/objective evaluation of auto dubbing in production use cases.
• Kamil Koniuch (AGH University of Krakow, Poland) presented his research on cognitive perspective on Absolute Category Rating (ACR) scale tests
• Patrick Le Callet (Nantes Universite, France) presented his work, in collaboration with researchers from SJTU (China) on perceptual quality assessment of AI-generated omnidirectional images, including the annotated dataset called AIGCOIQA2024.

Multimedia Experience and Human Factors (MEHF)

The MEHF group focuses on the human factors influencing audiovisual and multimedia experiences, facilitating a comprehensive understanding of how human factors impact the perceived quality of multimedia content. In this meeting, the following presentations were given:

• Dawid Juszka (AGH University of Krakow, Oland) presented his study on the impact of valence and arousal of video content on subjective QoE assessment scores.
• Syed Uddin (AGH University of Krakow, Poland) presented a subjective evaluation of low-latency algorithms and compared their performance with traditional DASH-based Adaptive Bitrate Algorithms (ABR) algorithms across multiple QoE metrics, various network conditions, and diverse content types.
• Tomasz Konaszyński (AGH University of Krakow, Poland) presented his research on human and contextual bias in QoE, addressing the impact of testers’ psychophysical condition, declared at the beginning of the research process.
• Avrajyoti Dutta (AGH University of Krakow, Poland) presented his work on human factors influencing on crowdsourcing subjective video quality assessment, covering an analysis over 7,900 ratings from 47 participants that showed that cognitive burden and contextual biases impact perceived video quality.

5G Key Performance Indicators (5GKPI)

The 5GKPI group studies the relationship between key performance indicators of new 5G networks and QoE of video services on top of them. In this meeting, Pablo Pérez (Nokia XR Lab, Spain) and the rest of the team presented a first draft of the VQEG Whitepaper on QoE management in telecommunication networks, which shares insights and recommendations on actionable controls and performance metrics that the Content Application Providers (CAPs) and Network Service Providers (NSPs) can use to infer, measure and manage QoE.

In addition, Pablo Perez (Nokia XR Lab, Spain), Marta Orduna (Nokia XR Lab, Spain), and Kamil Koniuch (AGH University of Krakow, Poland) presented design guidelines and a proposal of a simple but practical QoE model for communication networks, with a focus on 5G/6G compatibility.

Quality Assessment for Health Applications (QAH)

The QAH group is focused on the quality assessment of health applications. It addresses subjective evaluation, generation of datasets, development of objective metrics, and task-based approaches. In this meeting, Lumi Xia (INSA Rennes, France) presented her research on task-based medical image quality assessment by numerical observer.

Other updates

Apart from this, Ajit Ninan (Meta Platforms Inc., United States) delivered a keynote on rethinking visual quality for perceptual display; a panel was organized with Christos Bampis (Netflix, United States), Denise Noyes (Meta Platforms Inc., United States), and Yilin Wang (Google, United States) addressing what more is left to do on optimizing video quality for adaptive streaming applications, which was moderated by Narciso García (Universidad Politécnica de Madrid, Spain); and there was a co-located ITU-T Q19 interim meeting. In addition, although no progresses were presented in this meeting, the groups on No Reference Metrics (NORM) and on Quality Assessment for Computer Vision Applications (QACoViA) are still active.  

Finally, as already announced in the VQEG website, the next VQEG plenary meeting will be online or hybrid online/in-person, probably in November or December 2025.