VQEG Column: VQEG Meeting July 2024

Introduction

The University of Klagenfurt (Austria) hosted from July 01-05, 2024 a plenary meeting of the Video Quality Experts Group (VQEG). More than 110 participants from 20 different countries could attend this meeting in person and remotely.

The first three days of the meeting were dedicated to presentations and discussions about topics related to the ongoing projects within VQEG, while during the last two days an IUT-T Study Group G12 Question 19 (SG12/Q9) interim meeting took place. All the related information, minutes, and files from the meeting are available online in the VQEG meeting website, and video recordings of the meeting are available in Youtube.

All the topics mentioned bellow can be of interest for the SIGMM community working on quality assessment, but special attention can be devoted to the workshop on quality assessment towards 6G held within the 5GKPI group, and to the dedicated meeting of the IMG group hosted by the Distributed and Interactive Systems Group (DIS) of the CWI in September 2024 to work on ITU-T P.IXC recommendation. In addition, during those days there was a co-located ITU-T SG12 Q19 interim meeting.

Readers of these columns interested in the ongoing projects of VQEG are encouraged to subscribe to their corresponding reflectors to follow the activities going on and to get involved in them.

Another plenary meeting of VQEG has taken place from 18th 22nd of November 2024 and will be reported in a following issue of the ACM SIGMM Records.

Overview of VQEG Projects

Audiovisual HD (AVHD)

The AVHD group works on developing and validating subjective and objective methods to analyze commonly available video systems. During the meeting, there were 8 presentations covering very diverse topics within this project, such as open-source efforts, quality models, and subjective assessment methodologies:

• Jonas Birmé (Eyevinn Technology, Sweden) presented their work on lowering the barrier for using open source and contributing to a sustainable business. In this sense, he presented Open Source Cloud that offers open source as a service, removing the need for users of to maintain their own infrastructure in applications such as video encoding and quality assurance.
• Hadi Amirpour (University of Klagenfurt, Austria) and Jingwen Zhu (Nantes Université, France) presented their joint work that explored the use of Just Noticeable Differences (JND) to select bitratere-solution pairs for constructing a bitrate ladder with respect to the proportion of Satisfied User Ratio (SUR).
• Rafał Mantiuk (University of Cambridge, UK) talked about a family of metrics that directly model low-level human vision by incorporating the models of contrast sensitivity, contrast masking, and colour vision, which can bring many advantages, such as explainability, robustness to unseen distortion, etc. Those metrics include HDR-VDP-3, Foveated Video VDP, and Colour Video VDP, all publicly available as open-source projects.
• Dounia Hammou (University of Cambridge, UK) presented a study on the effect of viewing distance and display luminance on the visibility of HDR video streaming distortions, including a new video quality dataset, HDR-VDC, which captures the quality degradation of HDR content due to AV1 coding artifacts and the resolution reduction.
• Tomasz Konaszynski (AGH University of Krakow, Poland) talked about the impact of the structure and order of the stimuli presented to the viewers during subjective quality tests.
• Dominik Keller (Technische Universität Ilmenau, Germany) an open 8K HDR source dataset for video quality research (AVT-VQDB-UHD-2-HDR).
• Syed Uddin (AGH University of Krakow, Poland) presented his analysis on how effectively low-latency algorithms in DASH.JS enhance the user experience.
• Avrajyoti Dutta (AGH University of Krakow, Poland) presented a study that investigates the evaluation of subjective video quality utilizing short video clips on a crowd-sourcing platform.

Quality Assessment for Health applications (QAH)

The QAH group is focused on the quality assessment of health applications. It addresses subjective evaluation, generation of datasets, development of objective metrics, and task-based approaches. Joshua Maraval and Meriem Outtas (INSA Rennes, France) a dual rig approach for capturing multi-view video and spatialized audio capture for medical training applications, including a dataset for quality assessment purposes.

Statistical Analysis Methods (SAM)

The group SAM investigates on analysis methods both for the results of subjective experiments and for objective quality models and metrics. The following presentations were delivered during the meeting:  

• Rafał Mantiuk (University of Cambridge, UK) presented lessons learned, covering the main strengths and caveats, from a large experience performing pairwise comparison experiments, which includes the publication of datasets, software tools, and methods.
• Mohsen Jenadeleh (University of Konstanz, Germany) presented an experiment and an annotated dataset on image quality evaluation with triplet comparisons, in the particular case of multi-dimensional scaling. Also, he presented a study on the effects of immediate feedback on crowdworkers’ performance in subjective image quality assessment tasks using paired comparisons.
• Simon H. Del Pin (Norwegian University of Science and Technology, Norway) and Dietmar Saupe (University of Konstanz, Germany) presented their study on national differences in image quality assessment using discrete rating based on 3 large-scale datasets.
• Andréas Pastor (Nantes Université, France) proposed a new framework for perceptually-optimized encoding using the “libaom” of the AV1 codec, which aims to improve perceptual quality and compression efficiency.
• Hadi Amirpour (University of Klagenfurt, Austria) and Jingwen Zhu (Nantes Université, France) presented their joint work on analyzing the uncertainty of Satisfied User Ratios (SUR) and studying how different video quality metrics perform to estimate SUR.

No Reference Metrics (NORM)

The group NORM addresses a collaborative effort to develop no-reference metrics for monitoring visual service quality. In this sense, the following topics were covered:

• Yixu Chen (Amazon, US) presented their development of a metric tailored for video compression and scaling, which can extrapolate to different dynamic ranges, is suitable for real-time video quality metrics delivery in the bitstream, and can achieve better correlation than VMAF and P.1204.3.
• Filip Korus (AGH University of Krakow, Poland) talked about the detection of hard-to-compress video sequences (e.g., video content generated during e-sports events) based on objective quality metrics, and proposed a machine-learning model to assess compression difficulty.
• Hadi Amirpour (University of Klagenfurt, Austria) provided a summary of activities in video complexity analysis, covering from VCA to DeepVCA and describing a Grand Challenge on Video Complexity.
• Pierre Lebreton (Capacités & Nantes Université, France) presented a new dataset that allows studying the differences among existing UGC video datasets, in terms of characteristics, covered range of quality, and the implication of these quality ranges on training and validation performance of quality prediction models.
• Zhengzhong Tu (Texas A&M University, US) introduced a comprehensive video quality evaluator (COVER) designed to evaluate video quality holistically, from a technical, aesthetic, and semantic perspective. It is based on leveraging three parallel branches: a Swin Transformer backbone to predict technical quality, a ConvNet employed to derive aesthetic quality, and a CLIP image encoder to obtain semantic quality.

Emerging Technologies Group (ETG)

The ETG group focuses on various aspects of multimedia that, although they are not necessarily directly related to “video quality”, can indirectly impact the work carried out within VQEG and are not addressed by any of the existing VQEG groups. In particular, this group aims to provide a common platform for people to gather together and discuss new emerging topics, possible collaborations in the form of joint survey papers, funding proposals, etc. During this meeting, the following presentations were delivered:

• Abhijay Ghildyal (Portland State University, Canada) talked about the current status, gaps, shortcomings, and opportunities that the new paradigms of AI-generated image and video content brings.
• Kjell Brunnström (RISE, Sweden) presented his work on augmented reality head-up displays and digital rear view mirrors in cars, analyzing different factors, such as height of cameras, size of the field of view, etc.
• Mohammad Ghasempour (University of Klagenfurt, Austria) presented their approach for energy-aware video streaming, based on the appropriate selection of spatial and temporal resolution of videos.
• Mathias Wien (RWTH Aachen University, Germany) provided the updates on testing activities on video quality assessment within MPEG, including the creation of the CVQM database (Coded Video for study of Quality Metrics) covering both conventional and neural network-based video coding schemes.
• Henrique Souza Rossi (Luleå University of Technology, Sweden) presented his study on subjective QoE assessment for VR cloud-based gaming, focused on a first-person shooter game.

Joint Effort Group (JEG) – Hybrid

The group JEG-Hybrid addresses several areas of Video Quality Assessment (VQA), such as the creation of a large dataset for training such models using full-reference metrics instead of subjective metrics. In addition, the group includes the VQEG project Implementer’s Guide for Video Quality Metrics (IGVQM). The chair of this group,  Enrico Masala (Politecnico di Torino, Italy) presented the updates on the latest activities going on, including the status of the IGVQM project and a new image dataset, which will be partially subjectively annotated, to train DNN models to predict single user’s subjective quality perception. In addition to this:

• Lohic Fotio Tiotsop (Politecnico di Torino, Italy) presented various advances on modeling subject scoring behaviors, such as a new approach to estimate the subjective quality from noisy subjective ratings and a novel subject scoring model that allows to highlight several peculiar. He also presented the development of a DNN-based model to predict individual subjective quality of images with multiple distortions, which included the creation of a dataset comprising two million samples with synthetic labels derived from human annotation.
• Maria Martini (Kingston University London, UK) followed up from a presentation delivered in a previous VQEG meeting, highlighting the relationship between PSNR and SSIM for DCT-based compressed images and video, including comparisons with other approximations of the relationships between the two.

Immersive Media Group (IMG)

The IMG group researches on the quality assessment of immersive media technologies. Currently, the main joint activity of the group is the development of a test plan to evaluate the QoE of immersive interactive communication systems, which is carried out in collaboration with ITU-T through the work item P.IXC. In this meeting, Pablo Pérez (Nokia XR Lab, Spain) and Jesús Gutiérrez (Universidad Politécnica de Madrid, Spain) provided an update on the progress of the test plan, reviewing the status of the subjective tests that were being performed at the 13 involved labs. Also in relation with this test plan:

• Jesús Gutiérrez and Miguel Die (Universidad Politécnica de Madrid, Spain) presented preliminary results from the subjective tests carried out to study the impact on remote communication of display technology with the real-time FVV Live system.
• Felix Immohr (Technische Universität Ilmenau, Germany) presented the results of a study to assess the effect of spatial audio on audiovisual plausibility and presence perception in a three-user interactive communication scenario.

In relation with other topics addressed by IMG:

•  Kamran Javidi and Maria Martini (Kingston University London, UK) presented two light field datasets: 1) a display-specific turntable-based dataset for subjective quality assessment (KULF-TT53), and 2) a video dataset of scenes with moving objects captured with a plenoptic video camera.
• Stephan Fremerey (Technische Universität Ilmenau, Germany) presented an open-source dataset to evaluate cognitive performance including source audiovisual 360° video and immersive CGI multi-talker content.

In addition, a specific meeting of the group was held at Distributed and Interactive Systems Group (DIS) of CWI in Amsterdam (Netherlands) from the 2nd to the 4th of September to progress on the joint test plan for evaluating immersive communication systems. A total of 26 international experts from seven countries (Netherlands, Spain, Italy, UK, Sweden, Germany, US, and Poland) participated, with 7 attending online. In particular, the meeting featured presentations on the status of tests run by 13 participating labs, leading to insightful discussions and progress towards the ITU-T P.IXC recommendation.

Quality Assessment for Computer Vision Applications (QACoViA)

The group QACoViA addresses the study the visual quality requirements for computer vision methods, where the final user is an algorithm. In this meeting, Mikołaj Leszczuk (AGH University of Krakow, Poland) presented a study introducing a novel evaluation framework designed to address accurately predicting the impact of different quality factors on recognition algorithm, by focusing on machine vision rather than human perceptual quality metrics.

5G Key Performance Indicators (5GKPI)

The 5GKPI group studies relationship between key performance indicators of new 5G networks and QoE of video services on top of them. In this meeting, a workshop was organized by Pablo Pérez (Nokia XR Lab, Spain) and Kjell Brunnström (RISE, Sweden) on “Future directions of 5GKPI: Towards 6G“.

The workshop consisted of a set of diverse topics such as: QoS and QoE management in 5G/6G networks by (Michelle Zorzi, University of Padova, Italy); parametric QoE models and QoE management by Tobias Hoßfeld (University of. Würzburb, Germany) and Pablo Pérez (Nokia XR Lab, Spain); current status of standardization and industry by Kjell Brunnström (RISE, Sweden) and Gunilla Berndtsson (Ericsson); content and applications provider perspectives on QoE management by François Blouin (Meta, US); and communications service provider perspectives by Theo Karagioules and Emir Halepovic (AT&T, US). In addition, a panel moderated by Narciso García (Universidad Politécnica de Madrid, Spain) with Christian Timmerer (University of Klagenfurt, Austria), Enrico Masala (Politecnico di Torino, Italy) and Francois Blouin (Meta, US) as speakers.

Human Factors for Visual Experiences (HFVE)

The HFVE group covers human factors related to audiovisual experiences and upholds the liaison relation between VQEG and the IEEE standardization group P3333.1. In this meeting, there were two presentations related to these topics:

• Mikołaj Leszczuk and Kamil Koniuch (AGH University of Krakow, Poland) presented a two-part insight into the realm of image quality assessment: 1) it provided an overview of the TUFIQoE project (Towards Better Understanding of Factors Influencing the QoE by More Ecologically-Valid Evaluation Standards) with a focus on challenges related to ecological validity; and 2) it delved into the ‘Psychological Image Quality’ experiment, highlighting the influence of emotional content on multimedia quality perception.

• Ali Ak (Capacités & Nantes Université, France) presented a video quality dataset with Iphone HDR videos and AV1 encoding (Nantes-MobileHDRVQA) and a study on the potential use of crowdsourcing platforms for acceptability and annoyance experiments.