Standardization, open source software and datasets
Jump to Introduction, Reports from Standardization, Open Source, Datasets, Quality-of-Experience
Introduction
Standards Column: JPEG and MPEG
Introduction ISO/IEC JTC 1/SC 29 area of work comprises the standardization of coded representation of audio, picture, multimedia and hypermedia information and sets of compression and control functions for use with such information. SC29 basically hosts two working groups responsible for the development of international standards for the compression, decompression, processing, and coded representation of media content, ...
Reports from Standardization
JPEG Column: 100th meeting in Covilha, Portugal
JPEG AI reaches Committee Draft stage at the 100th JPEG meeting The 100th JPEG meeting was held in Covilhã, Portugal, from July 17th to 21st, 2023. At this meeting, in addition to its usual standardization activities, the JPEG Committee organized a celebration on the occasion of its 100th meeting. This face-to-face meeting, the second after the ...
VQEG Column: VQEG Meeting June 2023
Introduction This column provides a report on the last Video Quality Experts Group (VQEG) plenary meeting, which took place from 26 to 30 June 2023 in San Mateo (USA), hosted by Sony Interactive Entertainment. More than 90 participants worldwide registered for the hybrid meeting, counting with the physical attendance of more than 40 people. This ...
MPEG Column: 143rd MPEG Meeting in Geneva, Switzerland
The 143rd MPEG meeting took place in person in Geneva, Switzerland. The official press release can be accessed here and includes the following details: MPEG finalizes the Carriage of Uncompressed Video and Images in ISOBMFF MPEG reaches the First Milestone for two ISOBMFF Enhancements MPEG ratifies Third Editions of VVC and VSEI MPEG reaches the First Milestone of AVC ...
JPEG Column: 99th JPEG Meeting
JPEG Trust on a mission to re-establish trust in digital media The 99th JPEG meeting was held online, from 24th to 28th April 2023. Providing tools suitable for establishing provenance, authenticity and ownership of multimedia content is one of the most difficult challenges faced nowadays, considering the technological models that allow effective multimedia data manipulation and generation. ...
MPEG Column: 142nd MPEG Meeting in Antalya, Türkiye
The 142nd MPEG meeting was held as a face-to-face meeting in Antalya, Türkiye, and the official press release can be found here and comprises the following items: MPEG issues Call for Proposals for Feature Coding for Machines MPEG finalizes the 9th Edition of MPEG-2 Systems MPEG reaches the First Milestone for Storage and Delivery of Haptics Data MPEG completes ...
VQEG Column: Emerging Technologies Group (ETG)
Introduction This column provides an overview of the new Video Quality Experts Group (VQEG) group called the Emerging Technologies Group (ETG), which was created during the last VQEG plenary meeting in December 2022. For an introduction to VQEG, please check the VQEG homepage or this presentation. The works addressed by this new group can be of ...
JPEG Column: 98th JPEG meeting in Sydney, Australia
JPEG explores standardization in event-based imaging The 98th JPEG meeting was held in Sydney, Australia, from the 16th to 20th January 2023. This was a welcome return to face-to-face meetings after a long period of online meetings due to Covid-19 pandemics. Interestingly, the previous face-to-face meeting of the JPEG Committee was also held in Sydney, in ...
VQEG Column: VQEG Meeting December 2022
Introduction This column provides an overview of the last Video Quality Experts Group (VQEG) plenary meeting, which took place from 12 to 16 December 2022. Around 100 participants from 21 different countries around the world registered for the meeting that was organized online by Brightcove (United Kingdom). During the five days, there were more than ...
MPEG Column: 140th MPEG Meeting in Mainz, Germany
After several years of online meetings, the 140th MPEG meeting was held as a face-to-face meeting in Mainz, Germany, and the official press release can be found here and comprises the following items: MPEG evaluates the Call for Proposals on Video Coding for Machines MPEG evaluates Call for Evidence on Video Coding for Machines Feature Coding MPEG reaches ...
JPEG Column: 97th JPEG Meeting
JPEG initiates specification on fake media based on responses to its call for proposals The 97th JPEG meeting was held online from 24 to 28 October 2022. JPEG received responses to the Call for Proposals (CfP) on JPEG Fake Media, the first multimedia international standard designed to facilitate the secure and reliable annotation of media assets ...
Open Source
Procedural Content Generation – The Open Source Success Story of Wave Function Collapse
With OpenAI’s Dall-E, Midjourney, and Adobe Firefly, computer-generated visual content has hit the mass market. Machine learning-based algorithms can now create, and re-mix multimedia content based on huge corpora of images and videos and relieve creative professionals of tedious work. While this has gained much momentum lately, procedurally generated content (PCG) has been around for ...
Autonomous driving research with CARLA simulator
Introduction The autonomous driving industry, in order to advance through its six levels of automation (as defined by SAE, Society of Automotive Engineers ), is going to be increasingly more data-driven. While the number of sensors and their technology has been increasing it is still both cost-effective and, in some cases, necessary to use a simulator, ...
COCO Annotator – Web-based Image Segmentation Tool for Object Detection, Localization, and Keypoints
Introduction COCO Annotator is an image annotation tool that allows the labelling of images to create training data for object detection and localization. It provides many features, including the ability to label an image segment by drawing, label objects with disconnected visible parts, efficiently store and export annotations in the well-known COCO format as well as ...
Multimedia Retrieval and Analysis with Cottontail DB
Introduction Analysis and retrieval of media collections get more and more challenging the larger the collections become. Keeping everything in the main memory becomes less feasible, and more and more time and effort have to be spent to deal with the data management. However, traditional relational databases do not support primitives often used in multimedia workloads, ...
Open Source for Video Games – A Shortlist of Game Engines
Introduction Open-source software is a relevant topic in video game development. Taking a look at the most frequently employed game engines for developing Android games we can see that seven out of ten ranked engines are OSS. Over the last decade, more and more game studios and individual developers switched to open-source software. Oliver Franzke ...
nteract – A Desktop-Based Tool for Notebooks
Introduction Writing source code for programs with lightweight text editors or fully featured integrated development environments is considered the main method of programming. Notebooks, however, are an extremely practical tool. In contrast to IDEs, projects are set up more easily and they allow for running programs in a read-eval-print loop (REPL) environment. The Jupyter Notebooks Quick ...
Open Source Video Streaming & Recording
By the time of writing, the CoViD-19 disease has spread, impacting the world. Education and research are affected as well, but fortunately the community can continue working due to all the developments over the last few years. Even more so, streaming tech and video conferences allow us to keep in touch with family and friends. ...
OpenVSLAM: A Versatile Visual SLAM Framework
Visual simultaneous localization and mapping (VSLAM) systems are essential for augmented reality devices, autonomous control of robots and drones, etc. However, conventional open-source visual SLAM frameworks are not designed to be called as libraries from third-party programs. To overcome this situation, we have developed OpenVSLAM , a novel visual SLAM framework, and released it as ...
Configurable image annotation on the Web
Image annotations are required in a wide range of applications including image classification (which requires textual labels), object detection (bounding boxes), or image segmentation (pixel-wise classification). The application we show here was presented in the ACM Multimedia 2018 Open Source Software Competition track, and provides a number of configurable manual image annotation tools for detection and segmentation. It is available online, but in ...
Deep Learning in the Browser: TensorFlow JS
Having already discussed MatConvNet and Keras, let us continue with an open source framework for deep learning, which takes a new and interesting approach. TensorFlow.js is not only providing deep learning for JavaScript developers, but it’s also making applications of deep learning available in the WebGL enabled web browsers, or more specifically, Chrome, Chromium-based browsers, ...
Datasets
Overview of Benchmarking Platforms and Software for Multimedia Applications
In a time where Artificial Intelligence (AI) continues to push the boundaries of what was previously thought possible, the demand for benchmarking platforms that allow to fairly assess and evaluate AI models has become paramount. These platforms serve as connecting hubs between data scientists, machine learning specialists, industry partners, and other interested parties. They mostly ...
Overview of Open Dataset Sessions and Benchmarking Competitions in 2022 – Part 2 (MDRE at MMM 2022, ACM MM 2022)
In this Dataset Column, we present a review of some of the notable events related to open datasets and benchmarking competitions in the field of multimedia. This year’s selection highlights the wide range of topics and datasets currently of interest to the community. Some of the events covered in this review include special sessions on ...
Overview of Open Dataset Sessions and Benchmarking Competitions in 2022 – Part 3 (ImageCLEF 2022, MediaEval 2022)
In this Dataset Column, we present a review of some of the notable events related to open datasets and benchmarking competitions in the field of multimedia. This year’s selection highlights the wide range of topics and datasets currently of interest to the community. Some of the events covered in this review include special sessions on ...
Overview of Open Dataset Sessions and Benchmarking Competitions in 2022 – Part 1 (QoMEX 2022, ODS at MMSys ’22)
In this Dataset Column, we present a review of some of the notable events related to open datasets and benchmarking competitions in the field of multimedia. This year’s selection highlights the wide range of topics and datasets currently of interest to the community. Some of the events covered in this review include special sessions on ...
Two Interviews with renown Datasets Researchers
This issue of the Dataset Column provides two interviews with the researchers responsible for novel datasets of recent years. In particular, we first interview Nacho Reimat (https://www.cwi.nl/people/nacho-reimat), the scientific programmer responsible for the CWIPC-SXR, one of the first datasets on dynamic, interactive volumetric media. Second, we interview Pierre-Etienne Martin (https://www.eva.mpg.de/comparative-cultural-psychology/staff/pierre-etienne-martin/), responsible for contributions to datasets ...
Overview of Open Dataset Sessions and Benchmarking Competitions in 2021.
This issue of the Dataset Column proposes a review of some of the most important events in 2021 related to special sessions on open datasets or benchmarking competitions associated with multimedia data. While this is not meant to represent an exhaustive list of events, we wish to underline the great diversity of subjects and dataset ...
Dataset Column: Overview, Scope and Call for Contributions
Overview and Scope The Dataset Column (https://records.sigmm.org/open-science/datasets/) of ACM SIGMM Records provides timely updates on the developments in the domain of publicly available multimedia datasets as enabling tools for reproducible research in numerous related areas. It is intended as a platform for further dissemination of useful information on multimedia datasets and studies of datasets covering various ...
Report from the MMM 2020 Special Session on Multimedia Datasets for Repeatable Experimentation (MDRE 2020)
Introduction Information retrieval and multimedia content access have a long history of comparative evaluation, and many of the advances in the area over the past decade can be attributed to the availability of open datasets that support comparative and repeatable experimentation. Hence, sharing data and code to allow other researchers to replicate research results is needed ...
MediaEval Multimedia Evaluation Benchmark: Tenth Anniversary and Counting
MediaEval Multimedia Challenges MediaEval is a benchmarking initiative that offers challenges in multimedia retrieval, analysis and exploration. The tasks offered by MediaEval concentrate specifically on the human and social aspects of multimedia. They encourage researchers to bring together multiple modalities (visual, text, audio) and to think in terms of systems that serve users. Our larger ...
Dataset Column: ToCaDa Dataset with Multi-Viewpoint Synchronized Videos
This column describes the release of the Toulouse Campus Surveillance Dataset (ToCaDa). It consists of 25 synchronized videos (with audio) of two scenes recorded from different viewpoints of the campus. An extensive manual annotation comprises all moving objects and their corresponding bounding boxes, as well as audio events. The annotation was performed in order to ...
Quality-of-Experience
Explainable Artificial Intelligence for Quality of Experience Modelling
Data-driven Quality of Experience (QoE) modelling using Machine Learning (ML) arose as a promising alternative to the cumbersome and potentially biased manual QoE modelling. However, the reasoning of a majority of ML models is not explainable due to their black-box characteristics, which prevents us from gaining insights about how the model actually related QoE influence ...
Sustainability vs. Quality of Experience: Striking the Right Balance for Video Streaming
The exponential growth in internet data traffic, driven by the widespread use of video streaming applications, has resulted in increased energy consumption and carbon emissions. This outcome is primarily due to higher resolution or higher framerates content and the ability to watch videos on various end-devices. However, efforts to reduce energy consumption in video streaming ...
Green Video Streaming: Challenges and Opportunities
Introduction Regarding the Intergovernmental Panel on Climate Change (IPCC) report in 2021 and Sustainable Development Goal (SDG) 13 “climate action”, urgent action is needed against climate change and global greenhouse gas (GHG) emissions in the next few years . This urgency also applies to the energy consumption of digital technologies. Internet data traffic is responsible for ...
Towards the design and evaluation of more sustainable multimedia experiences: which role can QoE research play?
In this column, we reflect on the environmental impact and broader sustainability implications of resource-demanding digital applications and services such as video streaming, VR/AR/XR and videoconferencing. We put emphasis not only on the experiences and use cases they enable but also on the “cost” of always striving for high Quality of Experience (QoE) and better ...
What is the trade-off between CO2 emission and video-conferencing QoE?
It is a natural thing that users of multimedia services want to have the highest possible Quality of Experience (QoE), when using said services. This is especially so in contexts such as video-conferencing and video streaming services, which are nowadays a large part of many users’ daily life, be it work-related Zoom calls, or relaxing ...
Towards an updated understanding of immersive multimedia experiences
Bringing theories and measurement techniques up to date Development of technology for immersive multimedia experiences Immersive multimedia experiences, as its name is suggesting are those experiences focusing on media that is able to immerse users with different interactions into an experience of an environment. Through different technologies and approaches, immersive media is emulating a physical world through ...
MPEG Visual Quality Assessment Advisory Group: Overview and Perspectives
Introduction The perceived visual quality is of utmost importance in the context of visual media compression, such as 2D, 3D, immersive video, and point clouds. The trade-off between compression efficiency and computational/implementation complexity has a crucial impact on the success of a compression scheme. This specifically holds for the development of visual media compression standards which ...
ITU-T Standardization Activities Targeting Gaming Quality of Experience
Motivation for Research in the Gaming Domain The gaming industry has eminently managed to intrinsically motivate users to interact with their services. According to the latest report of Newzoo, there will be an estimated total of 2.7 billion players across the globe by the end of 2020. The global games market will generate revenues of $159.3 billion in 2020 . This ...
Immersive Media Experiences – Why finding Consensus is Important
An introduction to the QUALINET White Paper on Definitions of Immersive Media Experience (IMEx) . Introduction Immersive media are reshaping the way users experience reality. They are increasingly incorporated across enterprise and consumer sectors to offer experiential solutions to a diverse range of industries. Current technologies that afford an immersive media experience (IMEx) include Augmented Reality (AR), ...
Towards Interactive QoE Assessment of Robotic Telepresence
Telepresence robots (TPRs) are remote-controlled, wheeled devices with an internet connection. A TPR can “teleport” you to a remote location, let you drive around and interact with people. A TPR user can feel present in the remote location by being able to control the robot position, movements, actions, voice and video. A TPR facilitates human-to-human ...