Definitions of Crowdsourced Network and QoE Measurements

1 Introduction and Definitions

Crowdsourcing is a well-established concept in the scientific community, used for instance by Jeff Howe and Mark Robinson in 2005 to describe how businesses were using the Internet to outsource work to the crowd [2], but can be dated back up to 1849 (weather prediction in the US). Crowdsourcing has enabled a huge number of new engineering rules and commercial applications. To better define crowdsourcing in the context of network measurements, a seminar was held in Würzburg, Germany 25-26 September 2019 on the topic “Crowdsourced Network and QoE Measurements”. It notably showed the need for releasing a white paper, with the goal of providing a scientific discussion of the terms “crowdsourced network measurements” and “crowdsourced QoE measurements”. It describes relevant use cases for such crowdsourced data and its underlying challenges.

The outcome of the seminar is the white paper [1], which is – to our knowledge – the first document covering the topic of crowdsourced network and QoE measurements. This document serves as a basis for differentiation and a consistent view from different perspectives on crowdsourced network measurements, with the goal of providing a commonly accepted definition in the community. The scope is focused on the context of mobile and fixed network operators, but also on measurements of different layers (network, application, user layer). In addition, the white paper shows the value of crowdsourcing for selected use cases, e.g., to improve QoE, or address regulatory issues. Finally, the major challenges and issues for researchers and practitioners are highlighted.

This article now summarizes the current state of the art in crowdsourcing research and lays down the foundation for the definition of crowdsourcing in the context of network and QoE measurements as provided in [1]. One important effort is first to properly define the various elements of crowdsourcing.

1.1 Crowdsourcing

The word crowdsourcing itself is a mix of the crowd and the traditional outsourcing work-commissioning model. Since the publication of [2], the research community has been struggling to find a definition of the term crowdsourcing [3,4,5] that fits the wide variety of its applications and new developments. For example, in ITU-T P.912, crowdsourcing has been defined as:

Crowdsourcing consists of obtaining the needed service by a large group of people, most probably an on-line community.

The above definition has been written with the main purpose of collecting subjective feedback from users. For the purpose of this white paper focused on network measurements, it is required to clarify this definition. In the following, the term crowdsourcing will be defined as follows:

Crowdsourcing is an action by an initiator who outsources tasks to a crowd of participants to achieve a certain goal.

The following terms are further defined to clarify the above definition:

A crowdsourcing action is part of a campaign that includes processes such as campaign design and methodology definition, data capturing and storage, and data analysis.

The initiator of a crowdsourcing action can be a company, an agency (e.g., a regulator), a research institute or an individual.

Crowdsourcing participants (also “workers” or “users”) work on the tasks set up by the initiator. They are third parties with respect to the initiator, and they must be human.

The goal of a crowdsourcing action is its main purpose from the initiator’s perspective.

The goals of a crowdsourcing action can be manifold and may include, for example:

  • Gathering subjective feedback from users about an application (e.g., ranks expressing the experience of users when using an application)
  • Leveraging existing capacities (e.g., storage, computing, etc.)  offered by companies or individual users to perform some tasks
  • Leveraging cognitive efforts of humans for problem-solving in a scientific context.

In general, an initiator adopts a crowdsourcing approach to remedy a lack of resources (e.g., running a large-scale computation by using the resources of a large number of users to overcome its own limitations) or to broaden a test basis much further than classical opinion polls. Crowdsourcing thus covers a wide range of actions with various degrees of involvement by the participants.

In crowdsourcing, there are various methods of identifying, selecting, receiving, and retributing users contributing to a crowdsourcing initiative and related services. Individuals or organizations obtain goods and/or services in many different ways from a large, relatively open and often rapidly-evolving group of crowdsourcing participants (also called users). The use of goods or information obtained by crowdsourcing to achieve a cumulative result can also depend on the type of task, the collected goods or information and final goal of the crowdsourcing task.

1.2 Roles and Actors

Given the above definitions, the actors involved in a crowdsourcing action are the initiator and the participants. The role of the initiator is to design and initiate the crowdsourcing action, distribute the required resources to the participants (e.g., a piece of software or the task instructions, assign tasks to the participants or start an open call to a larger group), and finally to collect, process and evaluate the results of the crowdsourcing action.

The role of participants depends on their degree of contribution or involvement. In general, their role is described as follows. At least, they offer their resources to the initiator, e.g., time, ideas, or computation resources. In higher levels of contributions, participants might run or perform the tasks assigned by the initiator, and (optionally) report the results to the initiator.

Finally, the relationships between the initiator and the participants are governed by policies specifying the contextual aspects of the crowdsourcing action such as security and confidentiality, and any interest or business aspects specifying how the participants are remunerated, rewarded or incentivized for their participation in the crowdsourcing action.

2 Crowdsourcing in the Context of Network Measurements

The above model considers crowdsourcing at large. In this section, we analyse crowdsourcing for network measurements, which creates crowd data. This exemplifies the broader definitions introduced above, even if the scope is more restricted but with strong contextual aspects like security and confidentiality rules.

2.1 Definition: Crowdsourced Network Measurements

Crowdsourcing enables a distributed and scalable approach to perform network measurements. It can reach a large number of end-users all over the world. This clearly surpasses the traditional measurement campaigns launched by network operators or regulatory agencies able to reach only a limited sample of users. Primarily, crowd data may be used for the purpose of evaluating QoS, that is, network performance measurements. Crowdsourcing may however also be relevant for evaluating QoE, as it may involve asking users for their experience – depending on the type of campaign.

With regard to the previous section and the special aspects of network measurements, crowdsourced network measurements/crowd data are defined as follows, based on the previous, general definition of crowdsourcing introduced above:

Crowdsourced network measurements are actions by an initiator who outsources tasks to a crowd of participants to achieve the goal of gathering network measurement-related data.

Crowd data is the data that is generated in the context of crowdsourced network measurement actions.

The format of the crowd data is specified by the initiator and depends on the type of crowdsourcing action. For instance, crowd data can be the results of large scale computation experiments, analytics, measurement data, etc. In addition, the semantic interpretation of crowd data is under the responsibility of the initiator. The participants cannot interpret the crowd data, which must be thoroughly processed by the initiator to reach the objective of the crowdsourcing action.

We consider in this paper the contribution of human participants only. Distributed measurement actions solely made by robots, IoT devices or automated probes are excluded. Additionally, we require that participants consent to contribute to the crowdsourcing action. This consent might, however, vary from actively fulfilling dedicated task instructions provided by the initiator to merely accepting terms of services that include the option of analysing usage artefacts generated while interacting with a service.

It follows that in the present document, it is assumed that measurements via crowdsourcing (namely, crowd data) are performed by human participants aware of the fact that they are participating in a crowdsourcing campaign. Once clearly stated, more details need to be provided about the slightly adapted roles of the actors and their relationships in a crowdsourcing initiative in the context of network measurements.

2.2 Active and Passive Measurements

For a better classification of crowdsourced network measurements, it is important to differentiate between active and passive measurements. Similar to the current working definition within the ITU-T Study Group 12 work item “E.CrowdESFB” (Crowdsourcing Approach for the assessment of end-to-end QoS in Fixed Broadband and Mobile Networks), the following definitions are made:

Active measurements create artificial traffic to generate crowd data.

Passive measurements do not create artificial traffic, but measure crowd data that is generated by the participant.

For example, a typical case of an active measurement is a speed test that generates artificial traffic against a test server in order to estimate bandwidth or QoS. A passive measurement instead may be realized by fetching cellular information from a mobile device, which has been collected without additional data generation.

2.3 Roles of the Actors

Participants have to commit to participation in the crowdsourcing measurements. The level of contribution can vary depending on the corresponding effort or level of engagement. The simplest action is to subscribe to or install a specific application, which collects data through measurements as part of its functioning – often in the background and not as part of the core functionality provided to the user. A more complex task-driven engagement requires a more important cognitive effort, such as providing subjective feedback on the performance or quality of certain Internet services. Hence, one must differentiate between participant-initiated measurements and automated measurements:

Participant-initiated measurements require the participant to initiate the measurement. The measurement data are typically provided to the participant.

Automated measurements can be performed without the need for the participant to initiate them. They are typically performed in the background.

A participant can thus be a user or a worker. The distinction depends on the main focus of the person doing the contribution and his/her engagement:

A crowdsourcing user is providing crowd data as the side effect of another activity, in the context of passive, automated measurements.

A crowdsourcing worker is providing crowd data as a consequence of his/her engagement when performing specific tasks, in the context of active, participant-initiated measurements.

The term “users” should, therefore, be used when the crowdsourced activity is not the main focus of engagement, but comes as a side effect of another activity – for example, when using a web browsing application which collects measurements in the background, which is a passive, automated measurement.

“Workers” are involved when the crowdsourced activity is the main driver of engagement, for example, when the worker is paid to perform specific tasks and is performing an active, participant-initiated measurement. Note that in some cases, workers can also be incentivized to provide passive measurement data (e.g. with applications collecting data in the background if not actively used).

In general, workers are paid on the basis of clear guidelines for their specific crowdsourcing activity, whereas users provide their contribution on the basis of a more ambiguous, indirect engagement, such as via the utilization of a particular service provided by the beneficiary of the crowdsourcing results, or a third-party crowd provider. Regardless of the participants’ level of engagement, the data resulting from the crowdsourcing measurement action is reported back to the initiator.

The initiator of the crowdsourcing measurement action often has to design a crowdsourcing measurement campaign, recruit the participants (selectively or openly), provide them with the necessary means (e.g. infrastructure and/or software) to run their action, provide the required (backend) infrastructure and software tools to the participants to run the action, collect, process and analyse the information, and possibly publish the results.

2.4 Dimensions of Crowdsourced Network Measurements

In light of the previous section, there are multiple dimensions to consider for crowdsourcing in the context of network measurements. A preliminary list of dimensions includes:

  • Level of subjectivity (subjective vs. objective measurements) in the crowd data
  • Level of engagement of the participant (participant-initiated or background) or their cognitive effort, and awareness (consciousness) of the measurement level of traffic generation (active vs. passive)
  • Type and level of incentives (attractiveness/appeal, paid or unpaid)

Besides these key dimensions, there are other features which are relevant in characterizing a crowdsourced network measurement activity. These include scale, cost, and value; the type of data collected; the goal or the intention, i.e. the intention of the user (based on incentives) versus the intention of the crowdsourcing initiator of the resulting output.

Figure 1: Dimensions for network measurements crowdsourcing definition, and relevant characterization features (examples with two types of measurement actions)

In Figure 1, we have illustrated some dimensions of network measurements based on crowdsourcing. Only the subjectivity, engagement and incentives dimension are displayed, on an arbitrary scale. The objective of this figure is to show that an initiator has a wide range of combinations for crowdsourcing action. The success of a measurement action with regard to an objective (number of participants, relevance of the results, etc.) is multifactorial. As an example, action 1 may indicate QoE measurements from a limited number of participants and action 2 visualizes the dimensions for network measurements by involving a large number of participants.

3 Summary

The attendees of the Würzburg seminar on “Crowdsourced Network and QoE Measurements” have produced a white paper, which defines terms in the context of crowdsourcing for network and QoE measurements, lists of relevant use cases from the perspective of different stakeholders, and discusses the challenges associated with designing crowdsourcing campaigns, analyzing, and interpreting the data. The goal of the white paper is to provide definitions to be commonly accepted by the community and to summarize the most important use-cases and challenges from industrial and academic perspectives.

References

[1] White Paper on Crowdsourced Network and QoE Measurements – Definitions, Use Cases and Challenges (2020). Tobias Hoßfeld and Stefan Wunderer, eds., Würzburg, Germany, March 2020. doi: 10.25972/OPUS-20232.

[2] Howe, J. (2006). The rise of crowdsourcing. Wired magazine, 14(6), 1-4.

[3] Estellés-Arolas, E., & González-Ladrón-De-Guevara, F. (2012). Towards an integrated crowdsourcing definition. Journal of Information science, 38(2), 189-200.

[4] Kietzmann, J. H. (2017). Crowdsourcing: A revised definition and introduction to new research. Business Horizons, 60(2), 151-153.

[5] ITU-T P.912, “Subjective video quality assessment methods for recognition tasks “, 08/2016

[6] ITU-T P.808 (ex P.CROWD), “Subjective evaluation of speech quality with a crowdsourcing approach”, 06/2018

Report from ACM SIG Heritage Workshop

What does history mean to computer scientists?” – that was the first question that popped up in my mind when I was to attend the ACM Heritage Workshop at Minneapolis few months back. And needless to say, the follow up question was “what does history mean for a multimedia systems researcher?” As a young graduate student, I had the joy of my life when my first research paper on multimedia authoring (a hot topic those days) was accepted for presentation in the first ACM Multimedia in 1993, and that conference was held along side SIGGRAPH. Thinking about that, it gives multimedia systems researchers about 25 to 30 years of history. But what a flow of topics this area has seen: from authoring to streaming to content-based retrieval to social media and human-centered multimedia, the research area has been hot as ever. So, is it the history of research topics or the researchers or both? Then, how about the venues hosting these conferences, the networking events, or the grueling TPC meetings that prepped the conference actions?

Figure 1. Picture from the venue

With only questions and no clear answers, I decided to attend the workshop with an open mind. Most SIGs (Special Interest Groups) in ACM had representation at this workshop. The workshop itself was organized by the ACM History Committee. I understood this committee, apart from the workshop, organizes several efforts to track, record, and preserve computing efforts across disciplines. This includes identifying distinguished persons (who are retired but made significant contributions to computing), coming up with a customized questionnaire for the persons, training the interviewer, recording the conversations, curating them, archiving, and providing them for public consumption. Efforts at most SIGs were mostly based on the website. They were talking about how they try to preserve conference materials such as paper proceedings (when only paper proceedings were published), meeting notes, pictures, and videos. For instance, some SIGs were talking about how they tracked and preserved ACM’s approval letter for the SIG! 

It was very interesting – and touching – to see some attendees (senior Professors) coming to the workshop with boxes of materials – papers, reports, books, etc. They were either downsizing their offices or clearing out, and did not feel like throwing the material in recycling bins! These materials were given to ACM and Babbage Institute (at University of Minnesota, Minneapolis) for possible curation and storage.

Figure 2. Galleries with collected material

ACM History committee members talked about how they can fund (at a small level) projects that target specific activities for preserving and archiving computing events and materials. ACM History Committee agreed that ACM should take more responsibility in providing technical support to web hosting – obviously, not sure whether anything tangible would result.

Over the two days at the workshop, I was getting answers to my questions: History can mean pictures and videos taken at earlier MM conferences, TPC meetings, SIGMM sponsored events and retreats. Perhaps, the earlier paper proceedings that have some additional information than what is found in the corresponding ACM Digital Library version. Interviews with different research leaders that built and promoted SIGMM.

It was clear that history meant different things to different SIGs, and as SIGMM community, we would have to arrive at our own interpretation, collect and preserve that. And that made me understand the most obvious and perhaps, the most important thing: today’s events become tomorrow’s history! No brainer, right? Preserving today’s SIGMM events will give us a richer, colorful, and more complete SIGMM history for the future generations!

For the curious ones:

ACM Heritage Workshop website is at: https://acmsigheritage.dash.umn.ed

Some of the workshop presentation materials are available at: https://acmsigheritage.dash.umn.edu/uncategorized/class-material-posted/

Reports from ACM Multimedia 2019

Introduction

The annual ACM Multimedia Conference was held in Nice, France during October 21st to 25th, 2019. Being the 27th of its series, it attracted approximately 800 participants from all over the World. Among them were the student volunteers who supported the smooth organization of the Conference. In this article, I would like to introduce the reports and comments provided by each of them.

Figure. Student volunteers at ACM Multimedia 2019

Reports from student volunteers

Hui Chen (Tsinghua University, China)

It was such an honor for me to be granted for the student travel funding. During my stay in Nice, as a Ph.D. researcher, I read a lot of nice academical works which inspired me a lot. And I had wonderful conversations with authors from all over the world. Meanwhile, as a session volunteer, I was glad to help speakers and the audience during sessions. Their nice works and warm smiles impressed me a lot. What I most valued about is the friendship with other volunteers. We often discussed the attractive places and the delicious food in Nice, and cared for each other along the journey. I am deeply thankful for this wonderful experience in Nice. Some advice: (1) I think the beret was not necessary for the volunteers. Majority of us seemed to dislike it, because I did not see many volunteers wearing them. (2) Notifications about the room changing for sessions should be made clear early. (3) The manner of being punctual can be emphasized in the ice-break meeting. (4) Reminding of volunteered sessions could be shown in the Whova app.

Shizhe Chen (Renmin University of China, China)

It was a great pleasure to attend the ACM Multimedia this year. I have attended MM twice and the organizations are getting better and better. One big change was the deployment of the Whova APP, which really improved our experience at MM. On the one hand, it made connections among different attendants and organizations more convenient and efficient. On the other hand, it was nice to share photos in the APP about the conference. The volunteers are very devoted to serve the conference and uploaded many good pictures. The conference banquet at Nice also improved a lot. I really enjoyed local foods and magic shows. Even though there were so many people at that night, the organization was very ordered and made everyone satisfied. I also liked some multimedia modern art pieces exhibited at the conference which were wonderful. The conference session I enjoyed most was the Multimedia Grand Challenge, which provided a great opportunity for us academics to get involved in real-life problems in industries. It would have been better if there were more opportunities off-line to communicate with industry people in the conference. In summary, thanks for all the efforts the organizers have put on the conference. I am also proud to be able to contribute a little as a volunteer this time.

Yang Chen (University of Science and Technology of China, China)

This was my first time attending an international conference and needed to be a session volunteer during the conference. It was also my first time abroad. So I felt a litter nervous before going abroad for the conference. Fortunately, everything went smoothly in the end. The MM conference has been held for many years, so the experience of organizing the conference is rich, and the scale is also large. The MM conference provided a lot of convenience for the participants. All conference schedules can be found at the venue, so attendees can easily find the sessions that they needed to participate or were interested in. In addition, this year, the MM conference had many local characteristics of Nice, France. All attendees were given the famous local soap of Nice. The French food provided at the venue was also very delicious. All in all, it was a very impressive MM conference experience.

Amanda Duarte (Universitat Politècnica de Catalunya, Spain)

ACM Multimedia 2019 for me was a different and great experience. This was the first time that I attended this conference and it was very different of what I am used to find in a big conference. For the past four years I have been going to conferences more focused on Computer Vision and Machine Learning which nowadays have a large number of attendees, accepted papers, parallel sessions, and all the stress of being in a large venue and need to find the sessions that interest you across large rooms full of people.
ACM Multimedia on the other way around was held in a smaller venue with less attendees but yet with a very large amount of high quality researchers. Thus, I had the chance of talking more to great researchers in the areas that I have interest and also were interested in my work. In addition to my great experience during the conference in general, I had a great experience participating in the Doctoral Symposium during the conference. This event gave me the opportunity to present my work to great researchers that work on topics related to my doctoral thesis and were able of giving me great feedback and suggestions on how to improve my research.

Gelli Francesco (National University of Singapore, Singapore)

Although I am still a student, this edition of ACM Multimedia has been my third. Similar to the previous times, I met with the now more familiar community and allocated my time between attending sessions, walking around the posters, and rehearsing my presentation. My observation is that this year, there has been a major focus on applications rather than on the technical aspects. For example, the Best Paper session included works on zooming audio together with video, multi-modal dialogue system and privacy. The Brave New Ideas session, in which I presented, saw some more unusual and daring applications, such as the automatic creation of a sequence of images to match a short story. I had a great time presenting my paper on ranking images by subjective attributes, as I did my best to engage the audience with multiple questions. I learned from the senior organizers that their goal is to push the Multimedia community on applications such as Wellness and Human-Machine interaction, which naturally involves multimedia data. It was also inspiring to see so many engaged volunteers all dressed in blue running around with that very traditional beret. Definitely looking forward to attend the next edition.

Trung-Hiếu Hoàng (University of Science, Vietnam National University Ho Chi Minh City, Vietnam)

I am excited to share my experience in ACMMM 2019, as a person who received the student travel grant. Living in Vietnam, I cannot believe that I had such a great opportunity to travel thousands of kilometers and attend one of the top conferences in the world. On the first day, I met a lot of friends who received the same travel grant like me. We hung out together sharing different stories and experiences, all of us were enthusiastic and couldn’t wait to become a part of the volunteer team and contribute to the success of this year’s conference. During the last two years, I have had a strong interest in medical image processing. In detail, my research focuses on abnormality detection in the endoscopic image. Attending ACMMM 2019 gave me a wonderful chance to present my work, and discuss with experts in this field. I enjoyed the Healthcare Multimedia workshop, where I met the organizers of the BioMedia Grand Challenge track. I loved talking with them and discussing the future and their interests. In conclusion, I am so glad that the student grant brought me to Europe for the first time, opened up my mind and showed me wonderful things that I had never seen before.

Chia-Wei Hsieh (National Chiao Tung University, Taiwan)

I attended the ACM Multimedia 2019 in Nice, France, and listened to new AI approaches by experts and scholars from various countries. In this conference, I got the chance to learn about the latest studies’ results from world-renowned universities and research institutions, and learn about the latest developments in the industry. These most advanced tools broadened my view and realized the disabilities that can be improved in our future research. Furthermore, I appreciated serving as a volunteer at the conference. This forced me to interact with people and have made many good friends from all over the world. Everything is really well to attend MM’19, but a fly in the ointment is that the attendance of the last two days was pretty low. With some special benefits for people to stay, there could be more academic exchanges at the conference.

Michael Kerr (RMIT University, Australia)

I came to the conference this year hoping to learn about some very specific research that was being presented in my own field of employment of video surveillance. My expectations around these presentations was well met, but additionally I also took away new insights into other areas that were previously not of great interest to me, mainly as I had not explored their application to my own field.
I particularly enjoyed the Tutorials on Multimedia Forensics and was interested to see the work done in areas that had been developed in recent years. I was very engaged by the application of CNN to solve forensic challenges and quickly found that the application of these systems was a major theme in the entire conference. So, whilst I enjoyed many of the practical applications such as the Tutorials, the System Demonstrations, and the Open Source Software Competition, I also learnt a great deal about the growth of CNN technologies within the multimedia discipline as a whole. This has had a positive effect by helping to develop my own research plans and in particular enabling the identification of new applications that may be of interest to those working in multimedia as well as my specific field of interest.

Saurabh Kumar (Indian Institute of Technology Bombay, India)

I had an enjoyable experience at ACM Multimedia and learned a lot as this was my first big international conference. The papers were from diverse applications, and it was great talking to the speakers after the talks and at the posters. This allowed me to meet many amazing people from various backgrounds and talk about the exciting research they are doing. It was easy to approach anyone at the conference for casual or technical discussions. These days conferences are recorded with recording and proceedings are put up online, but that is just the tip of the iceberg. Attending a conference is a much broader experience, and I got an opportunity to experience this thanks to this travel grant. I made friends from many countries, thanks to the friendly atmosphere, and learned how my research fits in. I would like to highlight that being a volunteer was the primary reason all of this was possible. As a volunteer, it was so much easier to talk to people, and it was great helping them around. I would love to come and help out again anytime. The conference was just perfect, and I will remember my experience as a volunteer, which made it way more fun and especially the people I interacted with. I am certainly submitting to the next MM and coming back again with more exciting research and to meet this fantastic community. Also, visiting Nice was a delight, and it is a magnificent city, and the food was delicious.

Yadan Luo (University of Queensland, Australia)

It has been a great experience attending ACM Multimedia 2019 in Nice this October, where I met many brilliant people working in the same field. The Invited Talks offered impressive ideas, inspiring visions of the future and excellent coverage of many areas, like preserving audiovisual archives and data protection law. The most impressive part of the conference was the Art Exhibition, which showed a great power of installation art and interactive multimedia. Moreover, this great meeting brought me a lot of precious opportunities of meeting other researchers working in other subfields like video streaming, domain adaptation, and image generation. All chatting with them helped me quickly pick up plenty of new knowledge and opened a door to other research directions. In conclusion, I would like to sincerely express my thanks to people who have prepared the conference, in which I have benefited a lot from this fantastic event.

Kwanyong Park (Korea Advanced Institute of Science and Technology, Korea)

ACM Multimedia 2019 was especially special to me in terms of my improvement. Honestly speaking, my paper, presented in ACM Multimedia 2019, is my first international research accomplishment. So I really lacked experiences and skills about presenting my work and communicating with other researchers. But after ACM Multimedia 2019, I have confidence that at least I can do better and better. Combination of Oral and Poster sessions was really impressive and effective to obtain a lot of information in a short time. Every paper had at least 2 minutes oral presentation, and I could catch the core concept. Based on that, I easily decided whether the paper is closely related to my interest or not. I agree that this kind of configuration is a really efficient way. Through the conference, I saw which topics the students, who have mostly academic perspective, are focusing on. Although it is a great stimulus to me, I think practical perspective from various companies is also important to broaden the horizon. However, research from companies was relatively hard to find in ACM Multimedia 2019. I think that having some interactive booths from companies would be helpful.

K. R. Prajawal (International Institute of Information Technology, India)

ACM Multimedia was not only my first top-tier conference, but my first conference as well. I was pleased to see a lot of interesting and impactful papers from people from various backgrounds and universities. I particularly liked the conference venue as well, as it was spacious and comfortable to encourage a healthy discussion. I personally feel the food and meals could have been better curated. For example, I’m a vegetarian. I understand I have few items to eat, but the vegetarian items were not clearly labeled. This can be rectified in the future editions of the conference. I also believe that most of the presentation rooms were well prepared and organized for the presentation. During my oral presentation, however, I had an issue in playing a demo video. This issue had occurred because the conference organizers were not fully prepared to play a video during the presentation. That is rather odd, I felt, given this is a top-tier multimedia conference, which means it will have lots of audio and visual content. But, other than that, I had a very pleasant and fruitful time at the conference. I was able to connect and socialize with eminent researchers at ACM Multimedia and I hope to attend the next edition as well.

Estêvão Bissoli Saleme (Federal University of Espírito Santo, Brazil)

ACM Multimedia 2019 in Nice was such a unique experience. I volunteered for six sessions and attended a couple more, including the Best Paper session which I particularly liked the most. Not only because it brought original ideas, but also because I had the opportunity to witness an innovative presentation of the paper “Multimodal Dialog System: Generating Responses via Adaptive Decoders,” in which the speakers kept a dialog between them to give their talk. Besides that, I enjoyed the poster presentation hall, which we could mingle with other participants, get to know other people’s work better, and interact with them. One presentation that impressed me was entitled “Editing Text in the Wild.” In this work, the researchers proposed a method to replace any text in a picture keeping the background intact. The outcome looked like a real figure. Just impressive! Technically, I was more interested in Quality of Experience and Interaction, but I thought the subject of the papers in this session was spread out, which hindered the interaction with other presenters. It lacked a bit of work related to QoE itself. Finally, another aspect that deserves praise was the organization. Whova helped hugely, and we could post photos and interact with other people there. Moreover, Martha, Laurent, and Benoit were omnipresent and tireless. They were just on fire and worked very well to deliver such a great conference!

David Semedo (Universidade NOVA de Lisboa, Portugal)

My experience at ACM MM 2019 was very positive. I presented two full papers: one as a full oral and one as a short presentation. As such, the whole event was quite intense for me but also very personally enriching. I could do a lot of networking, with both students and senior researchers (the ConfLab contributed in this regard). As I am in my last Ph.D. year, I could talk with several researchers, from which I got valuable advices on how to take the next steps towards pursuing a career in research. At the poster sessions, I had the opportunity to discuss in detail my work with several people, from which I received constructive feedback. While I liked the fact that posters stayed posted during the whole conference, some were hard to find or were a bit hidden (e.g. the ones facing the wall). The conference program covered a wide range of topics on Multimedia. This allowed me to understand which techniques are being used on different tasks, and identify common technical aspects across these different tasks. It not only helped me in being updated, in terms of state-of-the-art approaches, but also in defining potential future research directions.

Junbo Wang (Institute of Automation, Chinese Academy of Sciences, China)

From 21-25 October 2019, I attended the ACM Multimedia 2019 Conference in Nice, France. This conference is a premier international conference in the area of multimedia within the field of computer science and I am very proud of attending this professional conference thanks to the ACM student travel grant. In this conference, I met many famous researchers in the area of multimedia, such as Tao Mei, Tat-Seng Chua, and Changsheng Xu. During the Poster or Oral sessions, I discussed many academic problems with these researchers, which really gave me new vision and insight. In addition to many academic talks, I also enjoyed a lot of French food, such as Macaroon and Foie Gras. As a session volunteer, I was also very happy to help the attendees in some session talks. The interesting and professional talks inspired me and guided my interest to many different research areas. Moreover, the conference was held at the NICE ACROPOLIS Convention Center in Nice, which is a beautiful and peaceful city. The fresh air and pleasant sea breeze gave us a good mood every day and made us have an unforgettable experience in this city. Overall, I think this conference was very successful to reach its fundamental objective: free communication. However, I also found that the sponsors this year was far less than that for last year, which can be expected to be better in the next year.

Xin Wang (Donghua University, China)

In my experience, I think MM’19 was very impressive and easy to follow. The arrangement of the conference was very reasonable especially the Whova APP helped me a lot whenever I wanted to figure on what is going on during the conference. Except one thing that I found in the first two days, there were still some workshops that had different room numbers between the session volunteer schedule (a Google sheet). That made me confused for a while, but luckily Martha told us use the APP as the standard. I really loved the Demo session and I think there must be people who had the same feeling like me. I met and talked with many researchers from all of the world, such as NUS, DCU, Nagoya University, Shandong University, National Chiao Tung University, etc. I still keep contact with some of them and exchange our research ideas. Besides, the weather in Nice was very comfortable. The food during the conference was rich and delicious. All of these reasons make me look forward to the next year’s MM conference.

Yitian Yuan (Tsinghua University, China)

It was very enjoyable to attend the ACM MM 2019 conference. As a volunteer, I could meet peers from other countries and schools and communicate with them, which is of great benefit to my scientific research knowledge. I think the agenda of this ACM MM conference was compact and reasonably arranged, but there are still the following problems that I think need to be improved: (1) The entrance of the main conference hall was dimly lit and the signs were not obvious, so volunteers needed to guide, otherwise it was difficult for participants to find the place. (2) I wish the stage at the Banquet had a bigger screen, so that everyone can see the name of the winners and the prize information. Finally, I wish the ACM MM better and better and more international influence.

Zhengyu Zhao (Radboud University, The Netherlands)

This was my second time to attend ACM Multimedia, after the first time in Korea in 2018. Overall, I felt the conference this year was a very successful edition, reflected by the perfect location, delicious food, well-designed program and especially the efforts from the volunteers. But still, I have some suggestions for further improvement. Specifically, from the experience of the poster presentation of my reproducibility paper, I realized that most people actually know nothing about this new reproducibility track. This made most of my time spent on explaining the general background of the track and so less time for my own research. I was happy to explain and get more people involved in this track but it would be better if the organization team could give more exposure of this track beforehand. From this experience serving as one of the poster session chairs, I figured out that many people do not use the official communication APP Whova, so the instructions and important announcements could not reach all the participants timely. In my opinion, more offline solutions (e.g., a big screen on the spot) would help.

Summary

In general, the student volunteers seemed to have enjoyed the event to the full extent, but some of them have proposed constructive suggestions that organizers and participants to future versions of the conference could take in account to provide better experiences!

All in all, I think we can see from the submitted reports that providing the chance to experience top-level research and to mix with all-range of researchers at a top-level Conference to young researchers who may one day become leaders in our community, would surely benefit us in the future.

Report on QoMEX 2019: QoE and User Experience in Times of Machine Learning, 5G and Immersive Technologies

qomex2019_logo

The QoMEX 2019 was held from 5 to 7 June 2019 in Berlin, with Sebastian Möller (TU Berlin and DFKI) and Sebastian Egger-Lampl (AIT Vienna) as general chairs. The annual conference celebrated its 10th birthday in Berlin since the first edition in 2009 in San Diego. The latter focused on classic multimedia voice, video and video services. Among the fundamental questions back then were how to measure and how to quantify quality from the user’s point of view in order to improve such services? Answers to these questions were also presented and discussed at QoMEX 2019, where technical developments and innovations in terms of video and voice quality were considered. The scope has however broadened significantly over the last decade: interactive applications, games and immersive technologies, which require new methods for the subjective assessment of perceived quality of service and QoE, were addressed. With a focus on 5G and its implications for QoE, the influence of communication networks and network conditions for the transmission of data and the provisioning of services were also examined. In this sense, QoMEX 2019 looked at both classic multimedia applications such as voice, audio and video as well as interactive and immersive services: gaming QoE, virtual realities such as VR exergames, and augmented realities such as smart shopping, 360° video, Point Clouds, Web QoE, text QoE, perception of medical ultrasound videos for radiologists, QoE of visually impaired users with appropriately adapted videos, QoE in smart home environments, etc.

In addition to this application-oriented perspective, methodological approaches and fundamental models of QoE were also discussed during QoMEX 2019. While suitable methods for carrying out user studies and assessing quality remain core topics of QoMEX, advanced statistical methods and machine learning (ML) techniques emerged as another focus topic at this year’s QoMEX. The applicability, performance and accuracy of e.g. neural networks or deep learning approaches have been studied for a wide variety of QoE models and in several domains: video quality in games, content of image quality and compression methods, quality metrics for high-dynamic-range (HDR) images, instantaneous QoE for adaptive video streaming over the Internet and in wireless networks, speech quality metrics, and ML-based voice quality improvement. Research questions addressed at QoMEX 2019 include the impact of crowdsourcing study design on the outcomes, or the reliability of crowdsourcing, for example, in assessing voice quality. In addition to such data-driven approaches, fundamental theoretical work on QoE and its quantification in systems as well as fundamental relationships and model approaches were presented.

The TPC Chairs were Lynne Baillie (HWU Edinburgh), Tobias Hoßfeld (Univ. Würzburg), Katrien De Moor (NTNU Trondheim), Raimund Schatz (AIT Vienna). In total, the program included 11 sessions on the above topics. From those 11 sessions, 6 sessions on dedicated topics were organized by various Special Session organizers in an open call. A total of 82 full paper contributions were submitted, out of which 35 contributions were accepted (acceptance rate: 43%). Out of the 77 short papers submitted, 33 were accepted and presented in two dedicated poster sessions. The QoMEX 2019 Best Paper Award went to Dominik Keller, Tamara Seybold, Janto Skowronek and Alexander Raake for “Assessing Texture Dimensions and Video Quality in Motion Pictures using Sensory Evaluation Techniques”. The Best Student Paper Award went to Alexandre De Masi and Katarzyna Wac for “Predicting Quality of Experience of Popular Mobile Applications in a Living Lab Study”.

The keynote speakers addressed several timely topics. Irina Cotanis gave an inspiring talk on QoE in 5G. She addressed both the emerging challenges and services in 5G and the question of how to measure quality and QoE in these networks. Katrien De Moor highlighted the similarities and differences between QoE and User Experience (UX), considering the evolution of the two terms QoE and UX in the past and current status. An integrated view of QoE and UX was discussed and how the two concepts develop in the future. In particular, she posed the question how the two communities could empower each other and what would be needed to bring both communities together in the future. The final day of QoMEX 2019 began with the keynote of artist Martina Menegon, who presented some of her art projects based on VR technology.

Additional activities and events within QoMEX 2019 comprised the following. (1) In the Speed ​​PhD mentoring organized by Sebastian Möller and Saman Zadtootaghaj, the participating doctoral students could apply for a short mentoring session (10 minutes per mentor) with various researchers from industry and academia in order to ask technical or general questions. (2) In a session organized by Sebastian Egger-Lampl, the best works of the last 5 years of the simultaneous TVX Conference and QoMEX were presented to show the similarities and differences between the QoE and the UX communities. This was followed by a panel discussion. (3) There was a 3-minute madness session organized by Raimund Schatz and Tobias Hoßfeld, which featured short presentations of “crazy” new ideas in a stimulating atmosphere. The intention of this second session is to playfully encourage the QoMEX community to generate new unconventional ideas and approaches and to provide a forum for mutual creative inspiration.

The next edition, QoMEX 2020, will be held May 26th to 28th 2020 in Athlone, Ireland. More information:  http://qomex2020.ie/

Report from MMSYS 2019 – by Alia Sheikh

Alia Sheikh (@alteralias) is researching immersive and interactive content. At present she is interested in the narrative language of immersive environments and how stories can best be choreographed within them.

Being part of an international academic research community and actually meeting said international research community are not exactly the same thing it turns out. After attending the 2019 ACM MMSys conference this year, I have decided that leaving the office and actually meeting the people behind the research is very worth doing.

This year I was invited to give an overview presentation at ACM MMSys ’19, which was being hosted at the University of Massachusetts. The MMSys, NOSSDAV and MMVE (International Workshop on Immersive Mixed and Virtual Environment Systems) conferences happen back to back, in a different location each year. I was asked to talk about some of our team’s experiments in immersive storytelling at MMVE. This included our current work on lightfields and my work on directing attention in, and the cinematography of, immersive environments.

To be honest it wasn’t the most convenient time to decide to catch a plane to New York and then a train to Boston for a multi-day conference, but it felt like the right time to take a break from the office and find out what the rest of the community had been working on.

Fig.1: A picturesque scene from the wonderful University of Massachussetts Amherst campus

Fig.1: A picturesque scene from the wonderful University of Massachussetts Amherst campus

I arrived at Amherst the day before the conference and (along with another delegate who had taken the same bus) wandered the tranquil university grounds slightly lost before being rescued by the ever calm and cheerful Michael Zink. Michael is the chair of the MMSys organising committee and someone who later spent much of the conference introducing people with shared interests to each other – he appeared to know every delegate by name.

Once installed in my UMass hotel room, I proceeded to spend the evening on my usual pre-conference ritual: entirely rewriting my presentation.

As the timetable would have it, I was going to be the first speaker.

Fig 2: Attendees at MMSys 2019 taking their seats

Fig. 2: Attendees at MMSys 2019 taking their seats

Fig 3: Alia in full flow during our talk on day 1

Fig. 3: Alia in full flow during our talk on day 1

I don’t actually know why I do this to myself, but there is something about turning up to the event proper that gives you a sense of what will work for that particular audience, and Michael had given me a brilliantly concise snapshot of the type of delegate that MMSys attracts – highly motivated, expert on the nuts and bolts of how to get data to where it needs to be and likely to be interested in a big picture overview of how these systems can be used to create a meaningful human connection.

Using selected examples from our research, I put together a talk on how the experience of stories in high tech immersive environments differs from more traditional formats, but, once the language of immersive cinematography is properly understood, we find that we are able to create new narrative experiences that are both meaningful and emotionally rich.

The next morning I walked into an auditorium full of strangers filing in, gave my talk (I thought it went well?) and then sank happily into a plush red flip-seat chair safe in the knowledge that I was free to enjoy the rest of the event.

The next item was the keynote and easily one of the best talks I have ever experienced at a conference. Presented by Professor Nimesha Ranasinghe it was a masterclass in taking an interesting problem (how do we transmit a full sensory experience over a network?) And presenting it in such a way as to neatly break down and explain the science (we can electrically stimulate the tongue to recreate a taste!) while never losing sight of the inherent joy in working on the kind of science you dream of as a child (therefore electrified cutlery!).

Fig. 4: Professor Nimesha Ranasinghe during his talk on Multisensory experiences

Fig. 4: Professor Nimesha Ranasinghe during his talk on Multisensory experiences

Fig 5: Multisensory enhanced multimedia - experiences of the future ?

Fig. 5: Multisensory enhanced multimedia – experiences of the future ?

Fig6: Networking and some delicious lunch

Fig. 6: Networking and some delicious lunch

At lunch I discovered the benefit of having presented my talk early – I made a lot of friends with people who had specific questions about our work, and got a useful heads up on work they were presenting either in the afternoon’s long papers session or the poster session.

We all spent the evening at the welcome reception on the top floor of UMass Hotel, where we ate a huge variety of tiny, delicious cakes and got to know each other better. It was obvious that in some cases, researchers that might collaborate remotely all year, were able to use MMSys as an excellent opportunity to catch up. As a newcomer to this ACM conference however, I have to say that I found it a very welcoming event, and I met a lot of very friendly people many of them working on research that was entirely different to my own, but which seemed to offer an interesting insight or area of overlap.

I wasn’t surprised that I really enjoyed MMVE – virtual environments are very much my topic of interest right now. But I was delighted by how much of MMSys was entirely up my street. ACM MMSys provides a forum for researchers to present and share their latest research findings in multimedia systems, and the conference cuts across all media/data types to showcase the intersections and the interplay of approaches and solutions developed for different domains. This year, the work presented on how to best encode and transport mixed reality content, as well as predict head motion to better encode and deliver the part of a spherical panorama a viewer was likely to be looking at, was particularly interesting to me. I wondered whether comparing the predicted path of user attention to the desired path of user attention, would teach us how to better control a users attention within a panoramic scene, or whether peoples viewing patterns were simply too variable. In the Open Datasets & Software track, I was fascinated by one particular dataset: “ A Dataset of Eye Movements for the Children with Autism Spectrum Disorder”. This was a timely reminder for me that diversity within the audience needed to be catered for when designing multimedia systems, to avoid consigning sections of our audience to a substandard experience.

Of the demos, there were too many interesting ones to list, but I was hugely impressed by the demo for Multi-Sensor Capture and Network Processing for Virtual Reality Conferencing. This used cameras and Kinects to turn me into a point cloud and put a live 3D representation of my own physical body in a virtual space.A brilliantly simple and incredibly effective idea and I found myself sitting next to the people responsible for it at a talk later that day and discussing ways to optimise their data compression.

Despite wearing a headset that allowed me to see the other participants, I was still able to see and therefore use my own hands in the real world – even extending to picking up and using my phone.

Fig7: Trying out some cool demos during a bustling demo session

Fig. 7: Trying out some cool demos during a bustling demo session

Fig. 8: An example of the social media interaction from my "tweeting"

Fig. 8: An example of the social media interaction from my “tweeting”

Amusingly, I found that I was (virtually) sat next to a point-cloud of TNO researcher Omar Niamut which led to my favourite twitter exchange of the whole conference. I knew Omar from online, but we had never actually managed to meet in real life. Still, this was the most life-like digital incarnation yet!

I really should mention the Women’s and Diversity lunch event which (pleasingly) was attended by both men and women and offered some absolutely fascinating insights.

These included: the value of mentors over the course of a successful academic life, how a gender pay-gap is inextricably related to work family policies and steps that have successfully been taken by some countries and organisations to improve work-life balance for all genders.

It was incredibly refreshing to see these topics being discussed both scientifically and openly. The conversations I had with people afterwards as they opened up about their own experiences of work and parenthood, were among the most interesting I have ever had on the topic.

Another nice surprise – MMSys offers childcare grants available for conference attendees who are bringing small children to the conference and require on-site childcare or who incur extra expenses in leaving their children at home. It was very cheering to see that the Inclusion Policy did not stop at simply providing interesting talks, but also translated into specific inclusive action.

Fig. 9:  Women’s and Diversity lunch! What a wonderful initiative - well done MMSys and SIGMM

Fig. 9: Women’s and Diversity lunch! What a wonderful initiative – well done MMSys and SIGMM

I am delighted that I made the decision to attend MMSys. I had not realised that I was feeling somewhat detached from my peers and the academic research community in general, until I was put in an environment which contained a concentrated amount of interesting research, interesting researchers and an air of collaboration and sheer good will. It is easy to get tunnel vision when you are focused on your own little area of work, but every conversation I had at the conference reminded me that research does not happen in a vacuum.

Fig. 10: A fascinating talk at the  Women’s and Diversity lunch - it initiated great post event discussions!

Fig. 10: A fascinating talk at the Women’s and Diversity lunch – it initiated great post event discussions!

Fig. 11: The food truck experience - one of many wonderful social aspects to MMSys 2019

Fig. 11: The food truck experience – one of many wonderful social aspects to MMSys 2019

I could write a thousand more words about every interesting thing I saw or person I met at MMSys, but that would only give you my own specific experience of the conference. (I did live tweet* a lot of the talks and demos just for my own records and that can all be found here: https://twitter.com/Alteralias/status/1148546945859952640?s=20)

Fig. 12: Receiving the SIGMM Social Media Reporter Award for MMSys 2019!

Fig. 12: Receiving the SIGMM Social Media Reporter Award for MMSys 2019!

Whether you were someone I was sitting next to at a paper session, a person I spoke to standing next to in line at the food truck (one of the many sociable meal events) or someone who demoed their PhD work to me, thank you so much for sharing this event with me.

Maybe I will see you at MMSys 2020.

* p.s it turns out that if you live-tweet an entire conference, Niall gives you a Social Media Reporter award.

Report from QoE-Management 2019

The 3rd International Workshop on Quality of Experience Management (QoE-Management 2019) was a successful full day event held on February 18, 2019 in Paris, France, where it was co-located with the 22nd Conference on Innovation in Clouds, Internet and Networks (ICIN). After the success of the previous QoE-Management workshops, the third edition of the workshop was also endorsed by the QoE and Networking Initiative (http://qoe.community). It was organized by workshop co-chairs Michael Seufert (AIT, Austrian Institute of Technology, Austria, who is now at University of Würzburg, Germany), Lea Skorin-Kapov (University of Zagreb, Croatia) and Luigi Atzori (University of Cagliari, Italy). The workshop attracted 24 full paper and 3 short paper submissions. The Technical Program Committee consisted of 33 experts in the field of QoE Management, which provided at least three reviews per submitted paper. Eventually, 12 full papers and 1 short paper were accepted for publication, which gave an acceptance rate of 48%.

On the day of the workshop, the co-chairs welcomed 30 participants. The workshop started with a keynote given by Martín Varela (callstats.io, Finland) who elaborated on “Some things we might have missed along the way”. He presented open technical and business-related research challenges for the QoE Management community, which he supported with examples from his current research on the QoE monitoring of WebRTC video conferencing. Afterwards, the first two technical sessions focused on video streaming. Susanna Schwarzmann (TU Berlin, Germany) presented a discrete time analysis approach to compute QoE-relevant metrics for adaptive video streaming. Michael Seufert (AIT Austrian Institute of Technology, Austria) reported the results of an empirical comparison, which did not find any differences in the QoE between QUIC- and TCP-based video streaming for naïve end users. Anika Schwind (University of Würzburg, Germany) discussed the impact of virtualization on video streaming behavior in measurement studies. Maria Torres Vega (Ghent University, Belgium) presented a probabilistic approach for QoE assessment based on user’s gaze in 360° video streams with head mounted displays. Finally, Tatsuya Otoshi (Osaka University, Japan) outlined how quantum decision making-based recommendation methods for adaptive video streaming could be implemented.

The next session was centered around machine learning-based quality prediction. Pedro Casas (AIT Austrian Institute of Technology) presented a stream-based machine learning approach for detecting stalling in real-time from encrypted video traffic. Simone Porcu (University of Cagliari, Italy) reported on the results of a study investigating the potential of predicting QoE from facial expressions and gaze direction for video streaming services. Belmoukadam Othmane (Cote D’Azur University & INRIA Sophia Antipolis, France) introduced ACQUA, which is a lightweight platform for network monitoring and QoE forecasting from mobile devices. After the lunch break, Dario Rossi (Huawei, France) gave the second keynote, entitled “Human in the QoE loop (aka the Wolf in Sheep’s clothing)”. He used the main leitmotiv of Web browsing and showed relevant practical examples to discuss the challenges towards QoE-driven network management and data-driven QoE models based on machine learning.

The following technical session was focused on resource allocation. Tobias Hoßfeld (University of Würzburg, Germany) elaborated on the interplay between QoE, user behavior and system blocking in QoE management. Lea Skorin-Kapov (University of Zagreb, Croatia) presented studies on QoE-aware resource allocation for multiple cloud gaming users sharing a bottleneck link. Quality monitoring was the topic of the last technical session. Tomas Boros (Slovak University of Technology, Slovakia) reported how video streaming QoE could be improved by 5G network orchestration. Alessandro Floris (University of Cagliari, Italy) talked about the value of influence factors data for QoE-aware management. Finally, Antoine Saverimoutou (Orange, France) presented WebView, a measurement platform for web browsing QoE. The workshop co-chairs closed the day with a short recap and thanked all speakers and participants, who joined in the fruitful discussions. To summarize, the third edition of the QoE Management workshop proved to be very successful, as it brought together researchers from both academia and industry to discuss emerging concepts and challenges related to managing QoE for network services. As the workshop has proven to foster active collaborations in the research community over the past years, a fourth edition is planned in 2020.

We would like to thank all the authors, reviewers, and attendants for their precious contributions towards the successful organization of the workshop!

Michael Seufert, Lea Skorin-Kapov, Luigi Atzori
QoE-Management 2019 Workshop Co-Chairs

Report from ACM MM 2018 – by Ana García del Molino

Seoul, what a beautiful place to host the premier conference on multimedia! Living in never-ending summer Singapore, I fell in love with the autumn colours of this city. The 26th edition of the ACM International Conference on Multimedia was held on October 22-26 of 2018 at the Lotte Hotel in Seoul, South Korea. It packed a full program including a very diverse range of workshops and tutorials, oral and poster presentations, art exhibits, interactive demos, competitions, industrial booths, and plenty of networking opportunities.

For me, this edition was a special one. About to graduate, with my thesis half written, I was presenting two papers. So of course, I was both nervous and excited. I had to fly to Seoul a few days ahead just to prepare myself! I was so motivated, I somehow managed to get myself a Best Social Media Reporter Award (who would have said… Me! A reporter!).

So, enough with the intro. Let’s get to the juice. What happened in Seoul between the 22nd and 26th of October 2018?

The first and last day of the conference were dedicated to Workshops and Tutorials. Those were a mix between Deep Learning themed and social applications of multimedia. The sessions included tutorials like “Interactive Video Search: Where is the User in the Age of Deep Learning?” that discussed the importance of the user in the collection of datasets, evaluation, and also interactive search, as opposed to using deep learning to solve challenges with big labelled datasets. In “Deep Learning Interpretation” Jitao Sang presented the main multimedia problems that can’t be addressed using deep learning. On the other hand, new and important trends related to social media (analysis of information diffusion and contagion, user activities and networking, prediction of real-world events, etc) were discussed in the tutorial “Social and Political Event Analysis using Rich Media”. The workshops were mainly user-centred, with special interest in affective computing and emotion analysis and use for multimedia (EE-USAD, ASMMC – MMAC 2018, AVEC 2018).

The conference kick-started with a wonderful keynote by Marianna Obrist. With “Don’t just Look – Smell, Taste, and Feel the Interaction” she showed us how to bring art into 4D by using technology, driving us through a full sensory experience that let us see, hear, and almost touch and smell. Ernest Edmonds also delved into how to mix art and multimedia in “What has art got to do with it?” but this time the other way around: what can multimedia research learn from the artists? Three industry speakers completed the keynote program. Xian-Sheng Hua from Alibaba Group shared their efforts towards visual Intelligence in “Challenges and Practices of Large-Scale Visual Intelligence in the Real-World”. Gary Geunbae Lee shared Samsung’s AI user experience strategy in “Living with Artificial Intelligence Technology in Connected Devices around Us.” And Bowen Zhou presented JD.com’s brand-new concept of Retail as a Service in “Transforming Retailing Experiences with Artificial Intelligence”.

This year’s program included 209 full papers, from a total of 757 submissions. 64 papers were allocated 15-minute oral presentations, while the others got a 90-second spotlight slot in the fast-forward sessions.  The poster sessions and the oral sessions run at the same time. While this was an inconvenience for poster presenters having to leave the poster to attend the oral sessions or miss them, the coffee breaks took place at the same location as the posters, so that was a win-win: chit-chat while having cookies and fruits? I’m in! In terms of content, half of the submissions were to only two areas: Multimedia and Vision and Deep Learning for Multimedia. But who am I to judge, when I had two of those myself! Many members of the community noted that the conference is becoming more and more deep learning, and less multimodal. To compensate, the workshops, tutorials and demos were mostly pure multimedia.

The challenges, competitions, art exhibits and demos happened in the afternoons, so at times it was hard to choose where to head to. So many interesting things happening all around the place! The art exhibit had some really cool interactive art installations, such as “Cellular Music”, that created music from visual motion. Among the demos, I found particularly interesting AniDance, an LSTM-based algorithm that made 3D models dance to the given music; SoniControl, an ultrasonic firewall for NFC protection; MusicMapp, a platform to augment how we experience music; and The Influence Map project, to explore who has influenced each scientist, and who did they most influence through their career.

Regarding diversity, I feel there is still a long way to go. Being in Asia, it makes sense that almost half of the attendees came from China. However, the submission numbers speak by themselves: less than 20% of submissions came from out of Asia, with just one submission from Africa (that’s a 0.13%!) Diversity is not only about gender, folks! I feel like more efforts are needed to facilitate the integration of more collectives in the multimedia community. One step at a time.

The next edition will take place at the NICE ACROPOLIS Convention Center in Nice, France from 21-25 October 2019. The ACM reproducibility badge system will be implemented for the first time at this 27th edition, so we may be seeing many more open-sourced projects. I am so looking forward to this!

First Combined ACM SIGMM Strategic Workshop and Summer School in Stellenbosch, South Africa

The first combined ACM SIGMM Strategic Workshop and Summer School will be held in Stellenbosch, South Africa, in the beginning of July 2020.

Rooiplein

First ACM Multimedia Strategic Workshop

The first Multimedia Strategic Workshop follows the successful series of workshops in areas such as information retrieval. The field of multimedia has continued to evolve and develop: collections of images, sounds and videos have become larger, computers have become more powerful, broadband and mobile Internet are widely supported, complex interactive searches can be done on personal computers or mobile devices, and soon. In addition, as large business enterprises find new ways to leverage the data they collect from users, the gap between the types of research conducted in industry and academics has widened, creating tensions over “repeatability” and “public data” in publications. These changes in environment and attitude mean that the time has come for the field to reassess its assumptions, goals, objectives and methodologies. The goal is to bring together researchers in the field to discuss long-term challenges and opportunities within the field. 

The participants of Multimedia Strategic Workshop will be active researchers in the field of Multimedia. The strategic workshop will give these researchers the opportunity to explore long-term issues in the multimedia field, to recognise the challenges on the horizon, to reach consensus on key issues and to describe them in the resulting report that will be made available to the multimedia research community. The report will stimulate debate, provide research directions to both researchers and graduate students, and also provide funding agencies with data that can be used coordinate the support for research.

The workshop will be held at the Wallenberg Research Centre at the Stellenbosch Institute for Advanced Study (STIAS). STIAS provides  provides venues and state-of-the art equipment for up to 300 conference guests at a time as well as breakaway rooms. 

The First ACM Multimedia Summer School on Multimedia

The motivation of the proposed summer school is to build on the success of the Deep Learning Indaba, but to focus on the application of machine learning to the field of Multimedia. We want delegates to be exposed to current research challenges in Multimedia. A secondary goal is to establish and grow the community of African researchers in the field of Multimedia; and to stimulate scientific research and collaboration between African researchers and the international community. The exact topics covered during the summer school will decided later together with the instructors but will reflect the current research trends in Multimedia.

The Strategic Workshop will be followed by the Summer School on Multimedia. Having the first summer school co-located with the Strategic Workshop will help to recruit the best possible instructors for the summer school. 

The Multimedia Summer School on Multimedia will be held at the Faculty of Engineering at Stellenbosch University, which is one of South Africa’s major producers of top quality engineers. The faculty was established in 1944 and is housed in a large complex of buildings with modern facilities, including lectures halls and electronic classrooms.

Stellenbosch is a university town in South Africa’s Western Cape province. It’s surrounded by the vineyards of the Cape Winelands and the mountainous nature reserves of Jonkershoek and Simonsberg. The town’s oak-shaded streets are lined with cafes, boutiques and art galleries. Cape Dutch architecture gives a sense of South Africa’s Dutch colonial history, as do the Village Museum’s period houses and gardens.

For more information about both events, please refer to the events’ web site (africanmultimedia.acm.org) or contact the organizers:

Report from ACM ICMR 2018 – by Cathal Gurrin

 

Multimedia computing, indexing, and retrieval continue to be one of the most exciting and fastest-growing research areas in the field of multimedia technology. ACM ICMR is the premier international conference that brings together experts and practitioners in the field for an annual conference. The eighth ACM International Conference on Multimedia Retrieval (ACM ICMR 2018) took place from June 11th to 14th, 2018 in Yokohama, Japan’s second most populous city. ACM ICMR 2018 featured a diverse range of activities including: Keynote talks, Demonstrations, Special Sessions and related Workshops, a Panel, a Doctoral Symposium, Industrial Talks, Tutorials, alongside regular conference papers in oral and poster session. The full ICMR2018 schedule can be found on the ICMR 2018 website <http://www.icmr2018.org/>. The organisers of ACM ICMR 2018 placed a large emphasis on generating a high-quality programme and in 2018; ICMR received 179 submissions to the main conference, with 21 accepted for oral presentation and 23 for poster presentation. A number of key themes emerged from the published papers at the conference: deep neural networks for content annotation; multimodal event detection and summarisation; novel multimedia applications; multimodal indexing and retrieval; and video retrieval from regular & social media sources. In addition, a strong emphasis on the user (in terms of end-user applications and user-predictive models) was noticeable throughout the ICMR 2018 programme. Indeed, the user theme was central to many of the components of the conference, from the panel discussion to the keynotes, workshops and special sessions. One of the most memorable elements of ICMR 2018 was a panel discussion on the ‘Top Five Problems in Multimedia Retrieval’ http://www.icmr2018.org/program_panel.html. The panel was composed of leading figures in the multimedia retrieval space: Tat-Seng Chua (National University of Singapore); Michael Houle (National Institute of Informatics); Ramesh Jain (University of California, Irvine); Nicu Sebe (University of Trento) and Rainer Lienhart (University of Augsburg). An engaging panel discussion was facilitated by Chong-Wah Ngo (City University of Hong Kong) and Vincent Oria (New Jersey Institute of Technology). The common theme was that multimedia retrieval is a hard challenge and that there are a number of fundamental topics that we need to make progress in, including bridging the semantic and user gaps, improving approaches to multimodal content fusion, neural network learning, addressing the challenge of processing at scale and the so called “curse of dimensionality”. ICMR2018 included two excellent keynote talks <http://www.icmr2018.org/program_keynote.html>. Firstly, Kohji Mitani, the Deputy Director of Science & Technology Research Laboratories NHK (Japan Broadcasting Corporation) explained about the ongoing evolution of broadcast technology and the efforts underway to create new (connected) broadcast services that can provide viewing experiences never before imagined and user experiences more attuned to daily life. The second keynote from Shunji Yamanaka, from The University of Tokyo discussed his experience of prototyping new user technologies and highlighted the importance of prototyping as a process that bridges an ever increasing gap between advanced technological solutions and societal users. During this entertaining and inspiring talk many prototypes developed in Yamanaka’s lab were introduced and the related vision explained to an eager audience. Three workshops were accepted for ACM ICMR 2018, covering the fields of lifelogging, art and real-estate technologies. Interestingly, all three workshops focused on domain specific applications in three emerging fields for multimedia analytics, all related to users and the user experience. The “LSC2018 – Lifelog Search Challenge”< http://lsc.dcu.ie/2018/> workshop was a novel and highly entertaining workshop modelled on the successful Video Browser Showdown series of participation workshops at the annual MMM conference. LSC was a participation workshop, which means that the participants wrote a paper describing a prototype interactive retrieval system for multimodal lifelog data. It was then evaluated during a live interactive search challenge during the workshop. Six prototype systems took part in the search challenge in front of an audience that reached fifty conference attendees. This was a popular and exciting workshop and could become a regular feature at future ICMR conferences. The second workshop was the MM-Art & ACM workshop <http://www.attractiveness-computing.org/mmart_acm2018/index.html>, which was a joint workshop that merged two existing workshops, the International Workshop on Multimedia Artworks Analysis (MMArt) and the International Workshop on Attractiveness Computing in Multimedia (ACM). The aim of the joint workshop was to enlarge the scope of discussion issues and inspire more works in related fields. The papers at the workshop focused on the creation, editing and retrieval of art-related multimedia content. The third workshop was RETech 2018 <https://sites.google.com/view/multimedia-for-retech/>, the first international workshop on multimedia for real estate tech. In recent years there has been a huge uptake of multimedia processing and retrieval technologies in the domain, but there are still a lot of challenges remaining, such as quality, cost, sensitivity, diversity, and attractiveness to users of content. In addition, ICMR 2018 included three tutorials <http://www.icmr2018.org/program_tutorial.html> on topical areas for the multimedia retrieval communities. The first was ‘Objects, Relationships and Context in Visual Data’ by Hanwang Zhang and Qianru Sun. The second was ‘Recommendation Technologies for Multimedia Content’ by Xiangnan He, Hanwang Zhang and Tat-Seng Chua and the final tutorial was ‘Multimedia Content Understanding, my Learning from very few Examples’ by Guo-Jun Qi. All tutorials were well received and feedback was very good. Other aspects of note from ICMR2018 were a doctoral symposium that attracted five authors and a dedicated industrial session that had four industrial talks highlighting the multimedia retrieval challenges faced by industry. It was interesting from the industrial talks to hear how the analytics and retrieval technologies developed over years and presented at venues such as ICMR were actually being deployed in real-world user applications by large organisations such as NEC and Hitachi. It is always a good idea to listen to the real-world applications of the research carried out by our community. The best paper session at ICMR 2018 had four top ranked works covering multimodal, audio and text retrieval. The best paper award went to ‘Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval’, by Niluthpol Mithun, Juncheng Li, Florian Metze and Amit Roy-Chowdhury. The best Multi-Modal Paper Award winner was ‘Cross-Modal Retrieval Using Deep De-correlated Subspace Ranking Hashing’ by Kevin Joslyn, Kai Li and Kien Hua. In addition, there were awards for best poster ‘PatternNet: Visual Pattern Mining with Deep Neural Network’ by Hongzhi Li, Joseph Ellis, Lei Zhang and Shih-Fu Chang, and best demo ‘Dynamic construction and manipulation of hierarchical quartic image graphs’ by Nico Hezel and Kai Uwe Barthel. Finally, although often overlooked, there were six reviewers commended for their outstanding reviews; Liqiang Nie, John Kender, Yasushi Makihara, Pascal Mettes, Jianquan Liu, and Yusuke Matsui. As with some other ACM sponsored conferences, ACM ICMR 2018 included an award for the most active social media commentator, which is how I ended up writing this report. There were a number of active social media commentators at ICMR 2018 each of which provided a valuable commentary on the proceedings and added to the historical archive.
fig1

Of course, the social side of a conference can be as important as the science. ICMR 2018 included two main social events, a welcome reception and the conference banquet. The welcome reception took place at the Fisherman’s Market, an Asian and ethnic dining experience with a wide selection of Japanese food available. The Conference Banquet took place in the Hotel New Grand, which was built in 1927 and has a long history of attracting famous guests. The venue is famed for the quality of the food and the spectacular panoramic views of the port of Yokohama. As with the rest of the conference, the banquet food was top-class with more than one of the attendees commenting that the Japanese beef on offer was the best they had ever tasted.

ICMR 2018 was an exciting and excellently organised conference and it is important to acknowledge the efforts of the general co-chairs: Kiyoharu Aizawa (The Univ. Of Tokyo), Michael Lew (Leiden Univ.) and Shin’ichi Satoh (National Inst. Of Informatics). They were ably assisted by the TPC co-chairs, Benoit Huet (Eurecom), Qi Tian (Univ. Of Texas at San Antonio) and Keiji Yanai (The Univ. Of Electro-Comm), who coordinated the reviews from a 111 person program committee in a double-blind manner, with an average of 3.8 reviews being prepared for every paper. ICMR 2019 will take place in Ottawa, Canada in June 2019 and ICMR 2020 will take place in Dublin, Ireland in June 2020. I hope to see you all there and continuing the tradition of excellent ICMR conferences.

The Lifelog Search Challenge Workshop attracted six teams for a real-time public interactive search competition.

The Lifelog Search Challenge Workshop attracted six teams for a real-time public interactive search competition.

The Lifelog Search Challenge Workshop attracted six teams for a real-time public interactive search competition.

The Lifelog Search Challenge Workshop attracted six teams for a real-time public interactive search competition.

Shunji Yamanaka about to begin his keynote talk on Prototyping

Shunji Yamanaka about to begin his keynote talk on Prototyping

Kiyoharu Aizawa and Shin'ichi Satoh, two of the ICMR 2018 General co-Chairs welcoming attendees to the ICMR 2018 Banquet at the historical Hotel New Grand.

Kiyoharu Aizawa and Shin’ichi Satoh, two of the ICMR 2018 General co-Chairs welcoming attendees to the ICMR 2018 Banquet at the historical Hotel New Grand.

SISAP 2018: 11th International Conference on Similarity Search and Applications

The International Conference on Similarity Search and Applications (SISAP) is an annual forum for researchers and application developers in the area of similarity data management. It aims at the technological problems shared by numerous application domains, such as data mining, information retrieval, multimedia, computer vision, pattern recognition, computational biology, geography, biometrics, machine learning, and many others that make use of similarity search as a necessary supporting service.

From its roots as a regional workshop in metric indexing, SISAP has expanded to become the only international conference entirely devoted to the issues surrounding the theory, design, analysis, practice, and application of content-based and feature-based similarity search. The SISAP initiative has also created a repository serving the similarity search community, for the exchange of examples of real-world applications, the source code for similarity indexes, and experimental testbeds and benchmark data sets (http://www.sisap.org). The proceedings of SISAP are published by Springer as a volume in the Lecture Notes in Computer Science (LNCS) series.

The 2018 edition of SISAP was held at the Universidad de Ingeniería y Tecnología (UTEC) in one of the oldest neighborhoods of Lima, in a modern building just recently inaugurated. The conference was held back-to-back, with a shared session, with the International Symposium on String Processing and Information Retrieval (SPIRE), an independent symposium with some intersection with SISAP. The organization was smooth and with a strong technical program assembled by two co-chairs and sixty program committee members. Each paper was reviewed by at least three referees. The program was completed with three invited speakers of high caliber.

During this 11th edition of SISAP, the first invited speaker was Hanan Samet (http://www.cs.umd.edu/~hjs/) from the University of Maryland, a pioneer in the similarity search field, with several books published on the subject. Professor Samet presented a state of the art system for news search based on the geographical location of the user to get more accurate results. The second invited speaker was Alistair Moffat (https://people.eng.unimelb.edu.au/ammoffat/) from the University of Melbourne, who delivered a talk about a novel technique for building compressed indexes using Asymmetric Numeral Systems (ANS). The ANS is a curious case of a scientific breakthrough not published in a peer-reviewed journal. Although it is available only as an arXiv technical, it is widely used in the industry – from Google and Facebook to Amazon, the adoption has been widespread. The third keynote talk was delivered in the shared session with SPIRE by Moshe Vardi (https://www.cs.rice.edu/~vardi/) of Rice University, a most celebrated editor of Communications of the ACM. Professor Vardi’s talk was an eye-opening discussion of jobs conquered by machines and the perspectives in accepting technological changes in everyday life. In the same shared session, a keynote presentation of SPIRE was given by Nataša Przulj (http://www0.cs.ucl.ac.uk/staff/natasa/) of University College London, concerning molecular networks and the challenges researchers face in developing a better understanding of them. It is worth noting that roughly 10% of the SPIRE participants were inspired to attend the SISAP technical program.

As it is usually the case, SISAP 2018 included a program with papers exploring various similarity-aware data analysis and processing problems from multiple perspectives. The papers presented at the conference in 2018 studied the role of similarity processing in the context of metric search, visual search, nearest neighbor queries, clustering, outlier detection, and graph analysis. Some of the papers had a theoretical emphasis, while others had a systems perspective, presenting experimental evaluations comparing against state-of-the-art methods. An interesting event at the 2018 conference, as well as the two previous editions, was a poster session that included all accepted papers. This component of the conference generated many lively interactions between presenters and attendees, to not only learn more about the presented techniques but also to identify potential topics for future collaboration.

A shortlist for the Best Paper Award was created from those conference papers nominated by at least one of their 3 reviewers. An award committee of 3 researchers ranked the shortlisted papers, from which a final ranking was decided using Borda count. The Best Paper Award was presented during the Conference Dinner. In a tradition that began with the 2009 conference in Prague, extended versions of the top-ranked papers were invited for a Special Issue of the Information Systems journal.

The venue and the location of SISAP 2018 deserve a special mention. In addition to the excellent conference facilities at UTEC, we had many student volunteers who were ready to help ensure that the logistical aspects of the conference ran smoothly. Lima was a superb location for the conference. Our conference dinner was held at the Huaca Pucllana Restaurant, located on the site of amazing archaeological remains within the city itself. We also had many opportunities to enjoy excellently-prepared traditional Peruvian food and drink. Before and after the conference, many participants chose to visit Machu Picchu, voted as one of the New Seven Wonders of the World.

SISAP 2018 demonstrated that the SISAP community has a strong stable kernel of researchers, active in the field of similarity search and to fostering the growth of the community. Organizing SISAP is a smooth experience thanks to the support of the Steering Committee and dedicated participants.

SISAP 2019 will be organized in Newark (NJ, USA) by Professor Vincent Oria (NJIT). This attractive location in the New York City metropolitan area will allow for easy and convenient travel to and from the conference. One of the major challenges of the SISAP conference series is to continue to raise its profile in the landscape of scientific events related to information indexing, database and search systems.

Figure 1. The conference dinner at Pachacamac ruins

Figure 1. The conference dinner at Pachacamac ruins

Figure 2. After the very interesting technical sessions, we ended the conference with an excursion to Lima downtown

Figure 2. After the very interesting technical sessions, we ended the conference with an excursion to Lima downtown

Figure 3. Keynote by Vardi

Figure 3. Keynote by Vardi