MediaEval is a benchmarking initiative that offers challenges in multimedia retrieval, analysis and exploration. The tasks offered by MediaEval concentrate specifically on the human and social aspects of multimedia. They encourage researchers to bring together multiple modalities (visual, text, audio) and to think in terms of systems that serve users. Our larger aim is to promote reproducible research that makes multimedia a positive force for society. In order to provide an impression of the topical scope of MediaEval, we describe a few examples of typical tasks.
Historically, MediaEval tasks have often involved social media analysis. One of the first tasks offered by MediaEval, called the “Placing” Task, focused on the geo-location of social multimedia. This task ran from 2010-2016 and studied the challenge of automatically predicting the location at which an image has been taken. Over the years, the task investigated the benefits of combining text and image features, and also explored the challenges involved with geo-location prediction of video.
MediaEval “Placing” Task (2010-2016)
The “Placing” Task gave rise to two daughter tasks, which are focused on the societal impact of technology that can automatically predict the geo-location of multimedia shared online. One is Flood-related Multimedia, which challenges researchers to extract information related to flooding disasters from social media posts (combining text and images). The other is Pixel Privacy, which allows researchers to explore ways in which adversarial images can be used to protect sensitive information from being automatically extracted from images shared online.
The MediaEval Pixel Privacy Task (currently ongoing) had its own “trailer” in 2019
MediaEval has also offered a number of tasks that focus on how media content is received by users. The interest of MediaEval in the emotional impact of music is currently continued by the Emotion and Theme Recognition in Music Task. Also, the Predicting Media Memorability Task explores the aspects of video that are memorable to users.
The MediaEval Predicting Media Memorability Task (currently ongoing)
Recently, MediaEval has widened its focus to include multimedia analysis in systems. The Sports Video Annotation Task works towards improving sports training systems and the Medico Task focuses on multimedia analysis for more effective and efficient medical diagnosis.
Recent years have seen the rise of the use of sensor data in MediaEval. The No-audio Multimodal Speech Detection Task uses a unique data set captured by people wearing sensors and having conversations in a social setting. In addition to the sensor data, the movement of the speakers is captured by an overhead camera. The challenge is to detect the moments at which the people are speaking without making use of audio recordings.
The Insight for Wellbeing Task uses a data set of lifelog images, sensor data and tags captured by people walking through a city wearing sensors and using smartphones. The challenge is to relate the data that is captured to the local pollution conditions.
MediaEval 10th Anniversary Workshop
Each year, MediaEval holds a workshop that brings researchers together to share their findings, discuss, and plan next year’s tasks. The 2019 workshop marked the 10th anniversary of MediaEval, which became an independent benchmark in 2010. The MediaEval 2019 Workshop was hosted by EURECOM in Sophia Antipolis, France. The workshop took place 27-29 October 2020, right after ACM Multimedia 2019, in Nice, France.
MediaEval 2019 Workshop at EURECOM, Sophia, Antipolis, France (Photo credit: Mathias Lux)
The MediaEval 2019 Workshop is grateful to SIGMM for their support. This support contributed to helping ten students to attend the workshop, across a variety of tasks and also made it possible to record all of the workshop talks. We also gratefully acknowledge the Multimedia Computing Group at Delft University of Technology and EURECOM.
Links to MediaEval 2019 tasks, videos and slides are available on the MediaEval 2019 homepage http://multimediaeval.org/mediaeval2019/. The link to the 2019 proceedings can be found there as well.
Presenting results of a MediaEval task (Photo credit: Vajira Thambawita)
MediaEval has compiled a bibliography of papers that have been published using MediaEval data sets. This list includes not only MediaEval workshop papers, but also papers published at other workshops, conferences, and in journals. In total, around 750 papers have been written that use MediaEval data, and this number continues to grow. Check out the bibliography at https://multimediaeval.github.io/bib.
The Medieval in MediaEval
A long-standing tradition in MediaEval is to incorporate some aspect of medieval history into the social event of the workshop. This tradition is a wordplay on our name (“mediaeval” is an older spelling of “medieval”). Through the years the medieval connection has served to provide a local context for the workshop and has strengthened the bond among participants. At the MediaEval 2019 Workshop, we offered the chance to take a nature walk to the medieval town of Biot.
A journey of discovery at the MediaEval 2019 workshop (Photo credit: Vajira Thambawita)
The walking participants and the participants taking the bus convened on the “Place des Arcades” in the medieval town of Biot, where we enjoyed a dinner together under historic arches.
The MediaEval 2019 workshop gathers in Place des Arcades in Biot, near EURECOM (Photo credit: Vajira Thambawita)
MediaEval 2020
MediaEval has just announced the task line-up for 2020. Registration will open in July 2020 and the runs will be due at the end of October 2020. The workshop will be held in December, with dates to be announced.
This year, the MediaEval workshop will be fully online. Since the MediaEval 2017 in Dublin, MediaEval has offered the possibility for remote workshop participation. Holding the workshop online this year is a natural extension of this trend, and we hope that researchers around the globe will take advantage of the opportunity to participate.
We are happy to introduce the new website: https://multimediaeval.github.io/. More information will be posted there as the season moves forward.
The day-to-day operations of MediaEval are handled by the MediaEval logistics committee, which grows stronger with each passing year. The authors of this article are logistics committee members from 2019.
Welcome to the first column on the ACM SIGMM Records from the Video Quality Experts Group (VQEG). VQEG is an international and independent organisation of technical experts in perceptual video quality assessment from industry, academia, and government organisations. This column briefly introduces the mission and main activities of VQEG, establishing a starting point of a series of columns that will provide regular updates of the advances within the current ongoing projects, as well as reports of the VQEG meetings. The editors of these columns are Jesús Gutiérrez (upper photo, jesus.gutierrez@upm.es), co-chair of the Immersive Media Group of VQEG and Kjell Brunnström (lower photo, kjell.brunnstrom@ri.se), general co-chair of VQEG. Feel free to contact them for any further questions, comments or information, and also to check the VQEG website: www.vqeg.org.
Introduction
The Video Quality Experts Group (VQEG) was born from a need to bring together experts in subjective video quality assessment and objective quality measurement. The first VQEG meeting, held in Turin in 1997, was attended by a small group of experts drawn from ITU-T and ITU-R Study Groups. VQEG was first grounded in basic subjective methodology and objective tool development/verification for video quality assessment such that the industry could be moved forward with standardization and implementation. At the beginning it was focused around measuring the perceived video quality since the distribution path for video and audio were limited and known.
Over the last 20 years from the formation of VQEG the ecosystem has changed dramatically and thus so must the work. Multimedia is now pervasive on all devices and methods of distribution from broadcast to cellular data networks. This shift has the expertise within VQEG to move from the visual (no-audio) quality of video to Quality of Experience (QoE).
The march forward of technologies means that VQEG needs to react and be on the leading edge of developing, defining and deploying methods and tools that help address these new technologies and move the industry forward. This also means that we need to embrace both qualitative and quantitative ways of defining these new spaces and terms. Taking a holistic approach to QoE will enable VQEG to drive forward and faster with unprecedented collaboration and execution
VQEG is open to all interested from industry, academia, government organizations and Standard-Developing Organizations (SDOs). There are no fees involved, no membership applications and no invitations are needed to participate in VQEG activities. Subscription to the main VQEG email list (ituvidq@its.bldrdoc.gov) constitutes membership in VQEG.
VQEG conducts work via discussions over email reflectors, regularly scheduled conference calls and, in general, two face-to-face meetings per year. There are currently more than 500 people registered across 11 email reflectors, including a main reflector for general announcements relevant to the entire group, and different project reflectors dedicated to technical discussions of specific projects. A LinkedIn group exists as well.
Objectives
The main objectives of VQEG are:
To provide a forum, via email lists and face-to-face meetings for video quality assessment experts to exchange information and work together on common goals.
To formulate test plans that clearly and specifically define the procedures for performing subjective assessment tests and objective models validations.
To produce open source databases of multimedia material and test results, as well as software tools.
To conduct subjective studies of multimedia and immersive technologies and provide a place for collaborative model development to take place.
Projects
Currently, several working groups are active within VQEG, classified under four main topics:
Subjective Methods: Based on collaborative efforts to improve subjective video quality test methods.
Audiovisual HD (AVHD), project “Advanced Subjective Methods” (AVHD-SUB): This group investigates improved audiovisual subjective quality testing methods. This effort may lead to a revision of ITU-T Rec. P.911. As examples of its activities, the group has investigated alternative experiment designs for subjective tests, to validate subjective testing of long video sequences that are only viewed once by each subject. In addition, it conducted a joint investigation into the impact of the environment on mean opinion scores (MOS).
Psycho-Physiological Quality Assessment (PsyPhyQA): The aim of this project is to establish novel psychophysiology based techniques and methodologies for video quality assessment and real-time interaction of humans with advanced video communication environments. Specifically, some of the aspects that the project is looking at include: video quality assessment based on human psychophysiology (including, eye gaze, EEG, EKG, EMG, GSR, etc.), computational video quality models based on psychophysiological measurements, signal processing and machine learning techniques for psychophysiology based video quality assessment, experimental design and methodologies for psychophysiological assessment, correlates of psychophysics and psychophysiology. PsyPhyQA has published a dataset and testplan for a common framework for the evaluation of psychophysiological visual quality assessment.
Statistical Analysis Methods (SAM): This group addresses problems related to how to better analyze and improve data quality coming from subjective experiments and how to consider uncertainty in objective media quality predictors/models development. Its main goals are: to improve methods used to draw conclusions from subjective experiments, to understand the process of expressing opinion in a subjective experiment, to improve subjective experiment design to facilitate analysis and applications, to improve the analysis of objective model performances, and to revisit standardised methods for the assessment of the performance of objective model performances.
Objective Metrics: Working towards developing and validating objective video quality metrics.
Audiovisual HD (AVHD), project “AVHD-AS / P.NATS phase 2”: It is a joint project of VQEG and ITU Study Group 12 Question 14. The main goal is to develop a multitude of objective models, varying in terms of complexity/type of input/use-cases for the assessment of video quality in HTTP/TCIP based adaptive bitrate streaming services (e.g., YouTube, Vimeo, Amazon Video, Netflix, etc). For these services quality experienced by the end user is affected by video coding degradations, and delivery degradations due to initial buffering, re-buffering and media adaptations caused by the changes in bitrate, resolution, and frame rate
Computer Generated Imagery (CGI): focuses on the computer generated content for both images and videos material. The main goals are as follows: creating a large database of computer generated content, analyzing the content (feature extraction before and after rendering), analyzing the performance of objective quality metrics, evaluating/developing existing/new quality metrics/models for CGI material, studying rendering adaptation techniques (depending on the network constraints). This activity is in-line with the ITU-T work item P.BBQCG (Parametric Bitstream-based Quality Assessment of Cloud Gaming Services).
No Reference Metrics (NORM): This group is an open collaborative for developing No-Reference metrics and methods for monitoring use case specific visual service quality. The NORM group is a complementary, industry-driven alternative of QoE to measure automatically the visual quality by using perceived indicators. Its main activities are to maintain a list of real-world use cases for visual quality monitoring, a list of potential algorithms and methods for no reference MOS and/or key indicators (visual artifact detection) for each use case, a list of methods (including datasets) to train and validate the algorithms for each use case, and a list of methods to provide root cause indication for each use case. In addition, the group encourages open discussions and knowledge sharing on all aspects related to no-reference metric research and development.
Joint Effort Group (JEG) – Hybrid: This group is an open collaboration working together to develop a robust Hybrid Perceptual/Bit-Stream model. It has developed and made available routines to create and capture bit-stream data and parse bit-streams into HMIX files. Efforts are underway into developing subjectively rated video quality datasets with bit-stream data that can be used by all JEG researchers. The goal is to produce one model that combines metrics developed separately by a variety of researchers.
Quality Assessment for Computer Vision Applications (QACoViA): the goal of this group is to study the visual quality requirements for computer vision methods, especially focusing on: testing methodologies and frameworks to identify the limit of computer vision methods with respect to the visual quality of the ingest; the minimum quality requirements and objective visual quality measure to estimate if a visual content is the operating region of computer vision; and delivering implementable algorithms being a proof/demonstrate of the new proposal concept of an objective video quality assessment methods for recognition tasks.
Industry and Applications: Focused on seeking improved understanding of new video technologies and applications.
5G Key Performance Indicators (5GKPI): Studies the relationship between the Key Performance Indicators (KPI) of new communication networks (namely 5G, but extensible to others) and the QoE of the video services on top of them. With this aim, this group addresses: the definition of relevant use cases (e.g., video for industrial applications, or mobility scenarios), the study of global QoE aspects for video in mobility and industrial scenarios, the identification of the relevant network KPIs(e.g., bitrate, latency, etc.) and application-level video KPIs (e.g., picture quality, A/V sync, etc.) and the generation of open datasets for algorithm testing and training.
IMG (Immersive Media Group): This group researches on quality assessment of immersive media, with the main goals of generating datasets of immersive media content, validating subjective test methods, and baseline quality assessment of immersive systems providing guidelines for QoE evaluation. The technologies covered by this group include: 360-degree content, virtual/augmented mixed reality, stereoscopic 3D content, Free Viewpoint Video, multiview technologies, light field content, etc.
Support and Outreach: Responsible for the support for VQEG’s activities.
eLetter: The goal of VQEG eLetter is to provide up-to-date technical advances on video quality related topics. Each issue of VQEG eletter features a collection of papers authored by well-known researchers. These papers are contributed by invited authors or authors responding to a call-for-paper, and they can be: technical papers, summary/review of other publications, best practice anthologies, reprints of difficult to obtain articles, and responses to other articles. VQEG wants the eLetter to be interactive in nature.
Human Factors for Visual Experiences (HFVE): The objectives of this group is to uphold the liaison relation between VQEG and the IEEE standardization group P3333.1. Some examples of the activities going on within this group are the standard for the (deep learning-based) assessment based on human factors of visual experiences with virtual/augmented/mixed reality and the standards on human factors for the quality assessment of light field imaging (IEEE P3333.1.4) and on quality assessment of high dynamic range technologies.
Independent Lab Group (ILG): The ILG act as independent arbitrators, whose generous contributions make possible the VQEG validation tests. Their goal is to ensure that all VQEG validation testing is unbiased and done to high quality standards.
Joint Effort Group (JEG): is an activity within VQEG that promotes collaborative efforts addressed to: validate metrics through both subjective dataset completion and metric design, extend subjective datasets in order to better identify the limitations of quality metrics, improve subjective methodologies to address new scenarios and use cases that involve QoE issues, and increase the knowledge about both subjective and objective video quality assessment.
Joint Qualinet-VQEG team on Immersive Media: The objectives of this joint team from Qualinet and VQEG are: to uphold the liaison relation between both bodies, to inform both QUALINET and VQEG on the activities in respective organizations (especially on the topic of immersive media), to promote collaborations on other topics (i.e., form new joint teams), and to uphold the liaison relation with ITU-T SG12, in particular on topics around interactive, augmented and virtual reality QoE.
Tools and Subjective Labs Setup: The objective of this project is to provide the video quality research community with a wide variety of software tools and guidance in order to facilitate research. Tools are available in the following categories: quality analysis (software to run quality analyses), encoding (video encoding tools), streaming (streaming and extracting information from video streams), subjective test software (tools for running and analyzing subjective tests), and helper tools (miscellaneous helper tools).
In addition, the Intersector Rapporteur Group on Audiovisual Quality Assessment (IRG-AVQA) studies topics related to video and audiovisual quality assessment (both subjective and objective) among ITU-R Study Group 6 and ITU-T Study Group 12. VQEG colocates meetings with the IRG-AVQA to encourage a wider range of experts to contribute to Recommendations.
VQEG activities are documented in reports and submitted to relevant ITU Study Groups (e.g., ITU-T SG9, ITU-T SG12, ITU-R WP6C), and other SDOs as appropriate. Several VQEG studies have resulted in ITU Recommendations.
Examined the performance of FR and NR models on standard definition video. The test materials used in this test plan and the subjective tests data are freely available to researchers.
Examined the performance of FR, RR and NR models for HDTV. Some of the video sequences used in this test are publicly available in the Consumer Digital Video Library.
Investigated how to assess 3DTV subjective video quality, covering methodologies, display requirements and evaluation of visual discomfort and fatigue.
On one side, addressed the subjective evaluation of audio-video quality metrics. On the other side, developed model standards for video quality assessment of streaming services over reliable transport for resolutions up to 4K/UHD, in collaboration with ITU-T SG12.
The contribution to current ITU standardization efforts is still ongoing. For example, updated texts have been contributed by VQEG on statistical analysis in ITU-T Rec. P.1401, and on subjective quality assessment of 360-degree video in ITU-T P.360-VR.
Apart from this, VQEG is supporting the research on QoE by providing for the research community tools and datasets. For instance, it is worth noting the wide variety of software tools and guidance in order to facilitate research provided by VQEG Tools and Subjective Labs Setup via GitHub. Another example, is the VQEG Image Quality Evaluation Tool (VIQET), which is an objective no-reference photo quality evaluation tool. Finally, several datasets have been published which can be found in the websites of the corresponding projects, in the Consumer Digital Video Library or in other repositories.
General articles for the interested reader about the work of VQEG, especially covering the previous works are [1, 2].
The 87th JPEG meeting initially planned to be held in Erlangen, Germany, was held online from 25-30, April 2020 because of the Covid-19 outbreak. JPEG experts participated in a number of online meetings attempting to make them as effective as possible while considering participation from different time zones, ranging from Australia to California, U.S.A.
JPEG decided to proceed with a Second Call for Evidence on JPEG Pleno Point Cloud Coding and continued work to prepare for contributions to the previous Call for Evidence on Learning-based Image Coding Technologies (JPEG AI).
The 87th JPEG meeting had the following highlights:
JPEG Pleno Point Cloud Coding issues a Call for Evidence on coding solutions supporting scalability and random access of decoded point clouds.
JPEG AI defines evaluation methodologies of the Call for Evidence on machine learning based image coding solutions.
JPEG XL defines the file format compatible with existing formats.
JPEG exploration on Media Blockchain releases use cases and requirements.
JPEG Systems releases a first version of JPEG Snack use cases and requirements.
JPEG XS announces significant improvement of the quality of raw-Bayer image sensor data compression.
JPEG Pleno Point Cloud
JPEG Pleno is working towards the integration
of various modalities of plenoptic content under a single and seamless framework.
Efficient and powerful point cloud representation is a key feature within this
vision. Point cloud data supports a wide range of applications including
computer-aided manufacturing, entertainment, cultural heritage preservation,
scientific research and advanced sensing and analysis. During the 87th JPEG meeting,
the JPEG Committee released a Second Call for Evidence on JPEG Pleno Point Cloud
Coding that focuses specifically on point cloud coding solutions supporting
scalability and random access of decoded point clouds. The Second Call for Evidence
on JPEG Pleno Point Cloud Coding has a revised timeline reflecting changes in
the activity due to the 2020 COVID-19 Pandemic. A Final Call for Evidence on
JPEG Pleno Point Cloud Coding is planned to be released in July 2020.
JPEG AI
The main focus of JPEG AI was on the promotion
and definition of the submission and evaluation methodologies of the Call for
Evidence (in coordination with the IEEE MMSP 2020 Challenge) that was issued as
outcome of the 86th JPEG meeting, Sydney, Australia.
JPEG XL
The File Format has been defined for JPEG XL
(ISO/IEC 18181-1) codestream, metadata and extensions. The file format enables
compatibility with ISOBMFF, JUMBF, XMP, Exif and other existing standards.
Standardization has now reached the Committee Draft stage and the DIS ballot is
ongoing. A white paper about JPEG XL’s features and tools was approved at this
meeting and is available on the jpeg.org website.
JPEG exploration on Media Blockchain – Call for
feedback on use cases and requirements
JPEG has determined that blockchain and
distributed ledger technologies (DLT) have great potential as a technology
component to address many privacy and security related challenges in digital
media applications. This includes digital rights management, privacy and
security, integrity verification, and authenticity, that impacts society in
several ways including the loss of income in the creative sector due to piracy,
the spread of fake news, or evidence tampering for fraud purposes.
JPEG is exploring standardization needs related
to media blockchain to ensure seamless interoperability and integration of
blockchain technology with widely accepted media standards. In this context,
the JPEG Committee announces a call for feedback from interested stakeholders
on the first public release of the use cases and requirements
document.
JPEG Systems initiates standardisation of JPEG Snack
Media “snacking”, the consumption of multimedia
in short bursts (less than 15 minutes) has become globally popular. JPEG
recognizes the need for standardizing how snack images are constructed to
ensure interoperability. A first version of JPEG Snack use cases and
requirements is now complete and publicly available on JPEG website inviting feedback
from stakeholders.
JPEG made progress on a fundamental capability
of the JPEG file structure with enhancements to JPEG Universal Metadata Box
Format (JUMBF) to support embedding common file types; the DIS text for JUMBF
Amendment 1 is ready for ballot. Likewise JPEG 360 Amendment 1 DIS text is
ready for ballot; this amendment supports stereoscopic 360 degree images,
accelerated rendering for regions-of-interest, and removes the XMP signature
block from the metadata description.
JPEG XS – The JPEG committee is pleased to
announce significant improvement of the quality of its upcoming Bayer
compression.
Over the past year, an improvement of around
2dB has been observed for the new coding tools currently being developed for
image sensor compression within JPEG XS. This visually lossless low-latency and
lightweight compression scheme can be used as a mezzanine codec in various
markets like real-time video storage inside and outside of cameras, and data
compression onboard autonomous cars. Mathematically lossless capability is also
investigated and encapsulation within MXF or SMPTE ST2110-22 is currently being
finalized.
Final Quote
“JPEG is committed to the development of new standards that provide state of the art imaging solutions to the largest spectrum of stakeholders. During the 87th meeting, held online because of the Covid-19 pandemic, JPEG progressed well with its current and even launched new activities. Although some timelines had to be revisited, overall, no disruptions of the workplan have occurred.” said Prof. Touradj Ebrahimi, the Convenor of the JPEG Committee.
About JPEG
The
Joint Photographic Experts Group (JPEG) is a Working Group of ISO/IEC, the
International Organisation for Standardization / International Electrotechnical
Commission, (ISO/IEC JTC 1/SC 29/WG 1) and of the International
Telecommunication Union (ITU-T SG16), responsible for the popular JPEG, JPEG
2000, JPEG XR, JPSearch, JPEG XT and more recently, the JPEG XS, JPEG Systems,
JPEG Pleno and JPEG XL families of imaging standards.
More
information about JPEG and its work is available at jpeg.org or by contacting Antonio Pinheiro or Frederik Temmermans (pr@jpeg.org)
of the JPEG Communication Subgroup.
If
you would like to stay posted on JPEG activities, please subscribe to the
jpeg-news mailing list on http://jpeg-news-list.jpeg.org.
Future JPEG meetings are planned as
follows:
No 88, initially planned in Geneva, Switzerland, July 4 to 10, 2020, will be held online from July 7 to 10, 2020.
The original blog post can be found at the Bitmovin Techblog and has been modified/updated here to focus on and highlight research aspects.
The 130th MPEG meeting concluded on April 24, 2020, in Alpbach, Austria … well, not exactly, unfortunately. The 130th MPEG meeting concluded on April 24, 2020, but not in Alpbach, Austria.
I attended the 130th MPEG meeting remotely.
Because of the Covid-19 pandemic, the 130th MPEG meeting has been converted from a physical meeting to a fully online meeting, the first in MPEG’s 30+ years of history. Approximately 600 experts attending from 19 time zones worked in tens of Zoom meeting sessions supported by an online calendar and by collaborative tools that involved MPEG experts in both online and offline sessions. For example, input contributions had to be registered and uploaded ahead of the meeting to allow for efficient scheduling of two-hour meeting slots, which have been distributed from early morning to late night in order to accommodate experts working in different time zones as mentioned earlier. These input contributions have been then mapped to GitLab issues for offline discussions and the actual meeting slots have been primarily used for organizing the meeting, resolving conflicts, and making decisions including approving output documents. Although the productivity of the online meeting could not reach the level of regular face-to-face meetings, the results posted in the press release show that MPEG experts managed the challenge quite well, specifically
MPEG ratifies MPEG-5 Essential Video Coding (EVC) standard;
MPEG issues the Final Draft International Standards for parts 1, 2, 4, and 5 of MPEG-G 2nd edition;
MPEG expands the coverage of ISO Base Media File Format (ISOBMFF) family of standards;
A new standard for large scale client-specific streaming with MPEG-DASH;
Other Important Activities at the 130th MPEG meeting: (i) the carriage of visual volumetric video-based coding data, (ii) Network-Based Media Processing (NBMP) function templates, (iii) the conversion from MPEG-21 contracts to smart contracts, (iv) deep neural network-based video coding, (v) Low Complexity Enhancement Video Coding (LCEVC) reaching DIS stage, and (vi) a new level of the MPEG-4 Audio ALS Simple Profile for high-resolution audio among others
The corresponding press release of the 130th MPEG meeting can be found here: https://mpeg.chiariglione.org/meetings/130. This report focused on video coding (EVC) and systems aspects (file format, DASH).
MPEG ratifies MPEG-5 Essential Video Coding Standard
At its 130th meeting, MPEG announced the completion of the new ISO/IEC 23094-1 standard which is referred to as MPEG-5 Essential Video Coding (EVC) and has been promoted to Final Draft International Standard (FDIS) status. There is a constant demand for more efficient video coding technologies (e.g., due to the increased usage of video on the internet), but coding efficiency is not the only factor determining the industry’s choice of video coding technology for products and services. The EVC standard offers improved compression efficiency compared to existing video coding standards and is based on the statements of all contributors to the standard who have committed announcing their license terms for the MPEG-5 EVC standard no later than two years after the FDIS publication date.
The MPEG-5 EVC defines two important profiles, including “Baseline profile” and “Main profile”. The “Baseline Profile” contains only technologies that are older than 20 years or otherwise freely available for use in the standard. In addition, the “Main Profile” adds a small number of additional tools, each of which can be either cleanly disabled or switched to the corresponding baseline tool on an individual basis.
It will be interesting to see how EVC profiles (baseline and main) will find its path into products and services given the existing number of codecs already in use (e.g., AVC, HEVC, VP9, AV1) and those still under development but being close to ratification (e.g., VVC, LCEVC). That is, in total, we may end up with about seven video coding formats that probably need to be considered for future video products and services. In other words, the multi-codec scenario I have envisioned some time ago is becoming reality raising some interesting challenges to be addressed in the future.
Research aspects: as for all video coding standards, the most important research aspect is certainly coding efficiency. For EVC it might be also interesting to research its usability of the built-in tool switching mechanism within a practical setup. Furthermore, the multi-codec issue, the ratification of EVC adds another facet to the already existing video coding standards in use or/and under development.
MPEG expands the Coverage of ISO Base Media File Format (ISOBMFF) Family of Standards
At the 130th WG11 (MPEG) meeting, the ISOBMFF family of standards has been significantly amended with new tools and functionalities. The standards in question are as follows:
ISO/IEC 14496-12: ISO Base Media File Format;
ISO/IEC 14496-15: Carriage of network abstraction layer (NAL) unit structured video in the ISO base media file format;
ISO/IEC 23008-12: Image File Format; and
ISO /IEC 23001-16: Derived visual tracks in the ISO base media file format.
In particular, three new amendments to the ISOBMFF family have reached their final milestone, i.e., Final Draft Amendment (FDAM):
Amendment 4 to ISO/IEC 14496-12 (ISO Base Media File Format) allows the use of a more compact version of metadata for movie fragments;
Amendment 1 to ISO/IEC 14496-15 (Carriage of network abstraction layer (NAL) unit structured video in the ISO base media file format) adds support of HEVC slice segment data track and additional extractor types for HEVC such as track reference and track groups; and
Amendment 2 to ISO/IEC 23008-12 (Image File Format) adds support for more advanced features related to the storage of short image sequences such as burst and bracketing shots.
At the same time, new amendments have reached their first milestone, i.e., Committee Draft Amendment (CDAM):
Amendment 2 to ISO/IEC 14496-15 (Carriage of network abstraction layer (NAL) unit structured video in the ISO base media file format) extends its scope to newly developed video coding standards such as Essential Video Coding (EVC) and Versatile Video Coding (VVC); and
the first edition of ISO/IEC 23001-16 (Derived visual tracks in the ISO base media file format) allows a new type of visual track whose content can be dynamically generated at the time of presentation by applying some operations to the content in other tracks, such as crossfading over two tracks.
Both are expected to reach their final milestone in mid-2021.
Finally, the final text for the ISO/IEC 14496-12 6th edition Final Draft International Standard (FDIS) is now ready for the ballot after converting MP4RA to the Maintenance Agency. WG11 (MPEG) notes that Apple Inc. has been appointed as the Maintenance Agency and MPEG appreciates its valuable efforts for the many years while already acting as the official registration authority for the ISOBMFF family of standards, i.e., MP4RA (https://mp4ra.org/). The 6th edition of ISO/IEC 14496-12 is expected to be published by ISO by the end of this year.
Research aspects: the ISOBMFF family of standards basically offers certain tools and functionalities to satisfy the given use case requirements. The task of the multimedia systems research community could be to scientifically validate these tools and functionalities with respect to the use cases and maybe even beyond, e.g., try to adopt these tools and functionalities for novel applications and services.
A New Standard for Large Scale Client-specific Streaming with DASH
Historically, in ISO/IEC 23009 (Dynamic Adaptive Streaming over HTTP; DASH), every client has used the same Media Presentation Description (MPD) as it best serves the scalability of the service (e.g., for efficient cache efficiency in content delivery networks). However, there have been increasing requests from the industry to enable customized manifests for more personalized services. Consequently, MPEG has studied a solution to this problem without sacrificing scalability, and it has reached the first milestone of its standardization at the 130th MPEG meeting.
ISO/IEC 23009-8 adds a mechanism to the Media Presentation Description (MPD) to refer to another document, called Session-based Description (SBD), which allows per-session information. The DASH client can use this information (i.e., variables and their values) provided in the SBD to derive the URLs for HTTP GET requests. This standard is expected to reach its final milestone in mid-2021.
Research aspects: SBD’s goal is to enable personalization while maintaining scalability which calls for a tradeoff, i.e., which kind of information to put into the MPD and what should be conveyed within the SBD. This tradeoff per se could be considered already a research question that will be hopefully addressed in the near future.
An overview of the current status of MPEG-DASH can be found in the figure below.
The next MPEG meeting will be from June 29th to July 3rd and will be again an online meeting. I am looking forward to a productive AhG period and an online meeting later this year. I am sure that MPEG will further improve its online meeting capabilities and can certainly become a role model for other groups within ISO/IEC and probably also beyond.
Crowdsourcing is a well-established concept in the scientific community, used for instance by Jeff Howe and Mark Robinson in 2005 to describe how businesses were using the Internet to outsource work to the crowd [2], but can be dated back up to 1849 (weather prediction in the US). Crowdsourcing has enabled a huge number of new engineering rules and commercial applications. To better define crowdsourcing in the context of network measurements, a seminar was held in Würzburg, Germany 25-26 September 2019 on the topic “Crowdsourced Network and QoE Measurements”. It notably showed the need for releasing a white paper, with the goal of providing a scientific discussion of the terms “crowdsourced network measurements” and “crowdsourced QoE measurements”. It describes relevant use cases for such crowdsourced data and its underlying challenges.
The outcome of the seminar is the white paper [1], which is – to our knowledge – the first document covering the topic of crowdsourced network and QoE measurements. This document serves as a basis for differentiation and a consistent view from different perspectives on crowdsourced network measurements, with the goal of providing a commonly accepted definition in the community. The scope is focused on the context of mobile and fixed network operators, but also on measurements of different layers (network, application, user layer). In addition, the white paper shows the value of crowdsourcing for selected use cases, e.g., to improve QoE, or address regulatory issues. Finally, the major challenges and issues for researchers and practitioners are highlighted.
This article now summarizes the current state of the art in crowdsourcing research and lays down the foundation for the definition of crowdsourcing in the context of network and QoE measurements as provided in [1]. One important effort is first to properly define the various elements of crowdsourcing.
1.1 Crowdsourcing
The word crowdsourcing itself is a mix of the crowd and the traditional outsourcing work-commissioning model. Since the publication of [2], the research community has been struggling to find a definition of the term crowdsourcing [3,4,5] that fits the wide variety of its applications and new developments. For example, in ITU-T P.912, crowdsourcing has been defined as:
Crowdsourcing consists of obtaining the needed service by a large group of people, most probably an on-line community.
The above definition has been written with the main purpose of collecting subjective feedback from users. For the purpose of this white paper focused on network measurements, it is required to clarify this definition. In the following, the term crowdsourcing will be defined as follows:
Crowdsourcing is an action by an initiator who outsources tasks to a crowd of participants to achieve a certain goal.
The following terms are further defined to clarify the above definition:
A crowdsourcing action is part of a campaign that includes processes such as campaign design and methodology definition, data capturing and storage, and data analysis.
The initiator of a crowdsourcing action can be a company, an agency (e.g., a regulator), a research institute or an individual.
Crowdsourcingparticipants (also “workers” or “users”) work on the tasks set up by the initiator. They are third parties with respect to the initiator, and they must be human.
The goal of a crowdsourcing action is its main purpose from the initiator’s perspective.
The goals of a crowdsourcing action can be manifold and may include,
for example:
Gathering subjective feedback from users about an application (e.g., ranks expressing the experience of users when using an application)
Leveraging existing capacities (e.g., storage, computing, etc.) offered by companies or individual users to perform some tasks
Leveraging cognitive efforts of humans for problem-solving in a scientific context.
In general, an initiator adopts a
crowdsourcing approach to remedy a lack of resources (e.g., running a
large-scale computation by using the resources of a large number of users to
overcome its own limitations) or to broaden a test basis much further than classical
opinion polls. Crowdsourcing thus covers a wide range of actions with various
degrees of involvement by the participants.
In crowdsourcing, there are various methods of identifying, selecting, receiving, and retributing users contributing to a crowdsourcing initiative and related services. Individuals or organizations obtain goods and/or services in many different ways from a large, relatively open and often rapidly-evolving group of crowdsourcing participants (also called users). The use of goods or information obtained by crowdsourcing to achieve a cumulative result can also depend on the type of task, the collected goods or information and final goal of the crowdsourcing task.
1.2
Roles and Actors
Given the above definitions, the actors involved in a crowdsourcing action are the initiator and the participants. The role of the initiator is to design and initiate the crowdsourcing action, distribute the required resources to the participants (e.g., a piece of software or the task instructions, assign tasks to the participants or start an open call to a larger group), and finally to collect, process and evaluate the results of the crowdsourcing action.
The role of participants depends on their degree of contribution or involvement. In general, their role is described as follows. At least, they offer their resources to the initiator, e.g., time, ideas, or computation resources. In higher levels of contributions, participants might run or perform the tasks assigned by the initiator, and (optionally) report the results to the initiator.
Finally, the relationships between the initiator and the participants are governed by policies specifying the contextual aspects of the crowdsourcing action such as security and confidentiality, and any interest or business aspects specifying how the participants are remunerated, rewarded or incentivized for their participation in the crowdsourcing action.
2 Crowdsourcing in the Context of Network
Measurements
The above model considers crowdsourcing at large. In this section, we
analyse crowdsourcing for network measurements, which creates crowd data. This exemplifies the
broader definitions introduced above, even if the scope is more restricted but
with strong contextual aspects like security and confidentiality rules.
2.1
Definition: Crowdsourced Network Measurements
Crowdsourcing enables a distributed and scalable approach to perform network measurements. It can reach a large number of end-users all over the world. This clearly surpasses the traditional measurement campaigns launched by network operators or regulatory agencies able to reach only a limited sample of users. Primarily, crowd data may be used for the purpose of evaluating QoS, that is, network performance measurements. Crowdsourcing may however also be relevant for evaluating QoE, as it may involve asking users for their experience – depending on the type of campaign.
With regard to the previous section and the special aspects of network measurements, crowdsourced network measurements/crowd data are defined as follows, based on the previous, general definition of crowdsourcing introduced above:
Crowdsourced network measurements are actions by an initiator who outsources tasks to a crowd of participants to achieve the goal of gathering network measurement-related data.
Crowd data is the data that is generated in the context of crowdsourced network measurement actions.
The format of the crowd data is specified by the initiator and depends
on the type of crowdsourcing action. For instance, crowd data can be the
results of large scale computation experiments, analytics, measurement data,
etc. In addition, the semantic interpretation of crowd data is under the
responsibility of the initiator. The participants cannot interpret the crowd
data, which must be thoroughly processed by the initiator to reach the
objective of the crowdsourcing action.
We consider in this paper the contribution of human participants only. Distributed measurement actions solely made by robots, IoT devices or automated probes are excluded. Additionally, we require that participants consent to contribute to the crowdsourcing action. This consent might, however, vary from actively fulfilling dedicated task instructions provided by the initiator to merely accepting terms of services that include the option of analysing usage artefacts generated while interacting with a service.
It follows that in the present document, it is assumed that
measurements via crowdsourcing (namely, crowd data) are performed by human
participants aware of the fact that they are participating in a crowdsourcing
campaign. Once clearly stated, more details need to be provided about the
slightly adapted roles of the actors and their relationships in a crowdsourcing
initiative in the context of network measurements.
2.2
Active and Passive Measurements
For a better classification of crowdsourced network measurements, it is
important to differentiate between active
and passive measurements. Similar to
the current working definition within the ITU-T Study Group 12 work item
“E.CrowdESFB” (Crowdsourcing Approach for
the assessment of end-to-end QoS in Fixed Broadband and Mobile Networks),
the following definitions are made:
Active measurements create
artificial traffic to generate crowd data.
Passive
measurements do not create artificial traffic, but measure crowd
data that is generated by the participant.
For example, a typical case of an active measurement is a
speed test that generates artificial traffic against a test server in order to
estimate bandwidth or QoS. A passive measurement instead may be realized by
fetching cellular information from a mobile device, which has been collected
without additional data generation.
2.3
Roles of the Actors
Participants have to commit to participation in the crowdsourcing measurements. The level of contribution can vary depending on the corresponding effort or level of engagement. The simplest action is to subscribe to or install a specific application, which collects data through measurements as part of its functioning – often in the background and not as part of the core functionality provided to the user. A more complex task-driven engagement requires a more important cognitive effort, such as providing subjective feedback on the performance or quality of certain Internet services. Hence, one must differentiate between participant-initiated measurements and automated measurements:
Participant-initiated measurements require
the participant to initiate the measurement. The measurement data are typically
provided to the participant.
Automated
measurements can be performed without the need for the participant
to initiate them. They are typically performed in the background.
A participant can thus be a user or a worker. The distinction depends on the main focus of the person doing the contribution and his/her engagement:
A
crowdsourcing user is providing
crowd data as the side effect of another activity, in the context of passive,
automated measurements.
A crowdsourcing
worker is providing crowd data as a consequence of his/her engagement when
performing specific tasks, in the context of active, participant-initiated
measurements.
The term “users” should, therefore, be used when the crowdsourced activity is not the main focus of engagement, but comes as a side effect of another activity – for example, when using a web browsing application which collects measurements in the background, which is a passive, automated measurement.
“Workers” are involved when the crowdsourced activity is the main driver of engagement, for example, when the worker is paid to perform specific tasks and is performing an active, participant-initiated measurement. Note that in some cases, workers can also be incentivized to provide passive measurement data (e.g. with applications collecting data in the background if not actively used).
In general,
workers are paid on the basis of clear guidelines for their specific
crowdsourcing activity, whereas users provide their contribution on the basis
of a more ambiguous, indirect engagement, such as via the utilization of a
particular service provided by the beneficiary of the crowdsourcing results, or
a third-party crowd provider. Regardless of the participants’ level of
engagement, the data resulting from the crowdsourcing measurement action is
reported back to the initiator.
The initiator of the crowdsourcing measurement action often has to design a crowdsourcing measurement campaign, recruit the participants (selectively or openly), provide them with the necessary means (e.g. infrastructure and/or software) to run their action, provide the required (backend) infrastructure and software tools to the participants to run the action, collect, process and analyse the information, and possibly publish the results.
2.4 Dimensions of Crowdsourced Network
Measurements
In light of the previous section, there are multiple dimensions to
consider for crowdsourcing in the context of network measurements. A
preliminary list of dimensions includes:
Level of
subjectivity (subjective vs. objective measurements) in the crowd data
Level of
engagement of the participant (participant-initiated or background) or their
cognitive effort, and awareness (consciousness) of the measurement level of
traffic generation (active vs. passive)
Type and
level of incentives (attractiveness/appeal, paid or unpaid)
Besides these key dimensions, there are other features which are relevant in characterizing a crowdsourced network measurement activity. These include scale, cost, and value; the type of data collected; the goal or the intention, i.e. the intention of the user (based on incentives) versus the intention of the crowdsourcing initiator of the resulting output.
Figure 1: Dimensions for network measurements crowdsourcing definition, and relevant characterization features (examples with two types of measurement actions)
In Figure 1, we have illustrated some dimensions of network measurements based on crowdsourcing. Only the subjectivity, engagement and incentives dimension are displayed, on an arbitrary scale. The objective of this figure is to show that an initiator has a wide range of combinations for crowdsourcing action. The success of a measurement action with regard to an objective (number of participants, relevance of the results, etc.) is multifactorial. As an example, action 1 may indicate QoE measurements from a limited number of participants and action 2 visualizes the dimensions for network measurements by involving a large number of participants.
3
Summary
The attendees of the Würzburg seminar on “Crowdsourced Network and QoE Measurements” have produced a white paper, which defines terms in the context of crowdsourcing for network and QoE measurements, lists of relevant use cases from the perspective of different stakeholders, and discusses the challenges associated with designing crowdsourcing campaigns, analyzing, and interpreting the data. The goal of the white paper is to provide definitions to be commonly accepted by the community and to summarize the most important use-cases and challenges from industrial and academic perspectives.
Describe your journey into research from your youth up to the present. What foundational lessons did you learn from this journey? Why were you initially attracted to multimedia?
Looking back at the early days of my life, my love for science started quite young. I loved solving puzzles and small recreational mathematical problems. Actually, I still do this. It may also be because my mother “seeded” many stories with great scientific people like Thomas Edison or Marie Curie every night. I admired them a lot and often dreamed of being like them. I also love to play video games. I played them a lot, and I think that I am quite good, especially in games like The Legend of Zelda and the Castlevania series. I also love to travel, and perhaps that is why I have a nomad’s journey over the last ten years, starting from Vietnam to Japan, Italy, Ireland, and now Norway. While living in Vietnam, I would often travel to the countryside on my motorbike. Solving puzzles, playing video games, collecting things, and travelling; these tiny things play an essential role in making me who I am today.
Now back to the story. I come from Vietnam, where it is very normal for my generation to grow through endless competitions. My first challenge was a math competition when I was eight. I then became a math student and followed many competitions like the current MIT Mystery Hunt. When I was 12, a friend of my father gave me his old PC as a present. It was a 486 (we called it that since it has an Intel 486 core), and it changed my life. I played with it endlessly. I learned Pascal by myself, and in the last year of my secondary schools (K-9), I proudly won the first rank at both Math and Informatics in the regional contests. Thanks to that, I entered one of the best high schools in Vietnam. I joined the Informatics class, and as you might already guess, we were dealing with programming challenges every day. We learned mainly algorithms and data structures, discrete mathematics, and computational complexity through solving challenging problems from the International Olympiad of Informatics. It is quite similar to Topcoder now. It was tough and very competitive, but it was exciting to me since it was like solving hard puzzles.
Moving to my bachelor’s, I took an honor program in Computer Science, which was one of the best Computer Science programs in Vietnam. In the third year of my bachelor’s in an Image Processing course, I did a project about image annotation. It was a pure K-means for image segmentation based on pixel color values, followed by a k-NN on a pre-trained set of images. It sounds pretty basic now, but this was in 2001, and “I did it my way” so it was a fantastic achievement! It was from this project I became a multimedia researcher.
After my bachelor’s, I continued researching computer vision and image retrieval in my master’s. In my first year as a Ph.D. student, I was working on a multimedia retrieval project, but just three months before the qualifying exam (you need to present your research proposal to continue your Ph.D.), I changed my research topic to Image Forensics, thanks to the course of the same name. I found everything I love in this new research field. It is like solving a puzzle, collecting evidence, and playing a game simultaneously. So, I became an image forensics researcher.
Some people say, “Choose a job you love, and you will never have to work a day in your life”, perhaps they are missing the last part “because no one will hire you”. Yes, it’s just a joke, but it can also be quite true in may circumstances. It was hard to find a job that needs image forensics when I finished my Ph.D. However, since I know image processing, computer vision, and machine learning, it was not that hard for me to find a postdoc in those fields. I was then doing both multimedia forensics and multimedia retrieval. This “evolvement” introduced me to a new field, lifelogging, a research direction that tries to discover insights from personal data archives. At first, it was an “okay” field to me, but later, after digging more into it, I found many interesting challenges that need to be solved. And that was a very long story about how I reach the starting point of my research.
Can you profile your current research, its challenges, opportunities, and implications?Tell us more about your vision and objectives behind your current roles.
Bergen, where I mainly focus on image forensics and lifelogging. Multimedia forensics is about discovering the history of modifications to multimedia content such as videos, images, audio, etc. Mainly, I work with images and have dabbled a bit in video forensics. Audio is nice too, but I mostly enjoy working with the visual side of multimedia. People tend to think about multimedia forensics as a tool to check if an image or a video is real or fake. However, we also try to look at the specifics for the media in question. Some potential questions for an image could, for example, be where was it first posted? What type of camera was it taken with? These are questions that help identify the reliability of the image in questions and give more information than fake or real. Also, I believe that we should also take a further step by considering the context of use (how, where, and when) of the multimedia content. The expectation of truthfulness is radically different if the image is hanging in an art gallery than if it is being used as evidence in a court case.
As previously mentioned, I also work with lifelogging. This work is still in its early stages. We have not proposed any novel approaches yet. Instead, we are building a community by organizing research activities as workshops and bench-marking initiatives. We believe that by holding such events, we are preparing a solid user-base for the next phase when people are more familiar with such technologies, the phase of personal data analytics. We have witnessed great applications of AI during the last decade. Since AI needs data, and people need more personalized solutions, I believe that very soon we will be doing lifelogging in our everyday life. Let’s wait and see if my prediction is becoming true or not.
How would you describe your top innovative achievements in terms of the problems you were trying to solve, your solutions, and the impact it has today and into the future?
In multimedia forensics, I am quite happy that I was among the first to propose an approach for discriminating between computer graphics and natural human faces. People are well aware of “Deepfake”, and many great people are working on this problem. However, when I presented my first study in 2011, many people, including computer graphics researchers, were laughing when I told them that they would soon not be able to distinguish computer-generated faces from the real ones. In image forensics, we try to reveal all traces of the image acquisition history, and since digital images are based on pixels, they are susceptible to changes. For example, many traces of modifications become incredibly hard to find if the image is resized. Most of my approaches are thus physical or geometrical based, which makes them more robust against changes as well as more reliable in terms of decision explanation.
Over your distinguished career, what are your top lessons you want to share with the audience?
I believe that I am still at the start of my career, and perhaps the first and the most important lesson I have is about “causes and effects” or what Steve Jobs described as “Connecting the dots”. There are dots in our life that are very hard to understand or predict how everything is connected, but eventually, when looking back, the connections will reveal themselves over time. Just follow whatever you think is good for you and try very hard to make it a good “dot”. Everyone wants to work with something we love, but finding what we love in our current work is even more important.
What is the best joke you know?
Most of the jokes I love are in Vietnamese, and unless you are Vietnamese, you can’t get them. I am trying to think about some “Western” jokes that share some commonalities with Vietnamese humor and culture. That should be a politics joke. I believe that you can find a similar version with KGB or Stasi. This one was very famous, and surprisingly, it is very well suited to my current research on lifelogging 🙂
“Why do Stasi officers make such good taxi drivers? — You get in the car and they already know your name and where you live.”
A recent photo of Tien.
Bio: Duc-Tien Dang-Nguyen is an associate professor at the University of Bergen. His main research interests are multimedia forensics, lifelogging, and machine learning.
