Author Archives: Mario Montagud

SIGMM Test of Time Paper Award, SIGMM Funding for Special Initiatives 2020 and SIGMM-sponsored Conference Fee Reduction

By | | ,
Leave a comment

In this note I provide an update on some recent SIGMM funding initiatives we are putting in place in 2020. These come about based on feedback from you, our members, on what you believe to be important and what you would like your SIGMM Executive Committee to work on. The specific topics covered here are the new SIGMM test of Time Paper Award, the various projects to be funded as a result of our call for funding applications for special initiatives some of which are further support for student travel, and the SIGMM sponsorship of conference fee reduction.

The SIGMM Test of Time Paper Award

A new award has just been formally approved by the ACM Awards Committee called the SIGMM Test of Time Paper Award with details available here.  To have an award formally approved by ACM the proposal has to be approved by a SIG Executive Committee, then approved by ACM headquarters, then approved by the ACM SIG Governing Board and then approved by the ACM Awards Committee. This ensures that ACM-approved awards are highly prestigious and rigorous in the way they select their winners.

SIGMM has been operational for 26 years and in that time has sponsored or co-sponsored more than 100 conferences and workshops, which have collectively published more than 15,574 individual papers.  5,742 of those papers have been published 10 or more years ago and the SIGMM Executive believes it is time to recognise the most significant and impactful from among those 5,742 papers.

The new award will be presented every year, starting this year, to the authors of a paper published either 10, 11 or 12 years previously at an SIGMM sponsored or co-sponsored conference. Thus the 2020 award will be for papers presented at a 2008, 2009 or 2010 SIGMM conference or workshop and will recognise the paper that has had the most impact and influence on the field of Multimedia in terms of research, development, product or ideas.  The paper may include theoretical advances, techniques and/or software tools that have been widely used, and/or innovative applications that have had impact on multimedia computing.

The award-winning paper will be selected by a 5-person selection committee consisting of 2 members of the organising committee for the MULTIMEDIA Conference in that year plus 3 established and respected members of our community who have no conflict of interest with the nominated papers.  The nominated papers are those top-ranked based on citation count from the ACM Digital Library, though the selection committee can add others if they wish.

Faced with the issue of recognising papers published prior to the 10, 11 or 12 year window of consideration, in this inaugural year when we announce the inaugural winner from 2008/2009/2010 we will also announce a set of up to 14 papers published at SIGMM conferences prior to 2008 as “honourable mentions” which could have been considered as strong candidates in their year of publication, if there had been an award for that year.  The first SIGMM MULTIMEDIA Conference was held in 1993 but was not sponsored by SIGMM as SIGMM was formed only in 1994, and so these up to 14 honourable mentions will cover the years 1994 to 2007 inclusive.

Selecting these papers from among all these candidates will be a challenging task for the selection committee and we wish them well in their deliberations and look forward to the award announcements at the MULTIMEDIA Conference in Seattle later this year.

SIGMM Funding for Special Initiatives 2020

For the last three years in a row, the SIGMM Executive committee has issued an invitation for applications for funding for new initiatives, which are submitted by SIGMM members. The assessment criteria for these initiatives were that they focus on one, or more, of the following:

– building on SIGMM’s excellence and strengths;
   

– nurturing new talent in the SIGMM community;
    

– addressing weakness(es) in the SIGMM community and in SIGMM activities

In late 2019 we issued our third call for funding and we received our strongest yet response from the SIGMM community. Submissions were evaluated and assessed by the SIGMM Executive and discussed at an Executive Committee meeting and in this short note I outline the funding awards which were made.

Before looking at the awards, it is worth reminding the reader that starting this year, SIGMM is centralising our support for student travel to our SIGMM-supported events, namely ICMR (in Dublin), MMSys (in Istanbul), IMX (in Barcelona), IH&MMSec (in Denver), MULTIMEDIA (in Seattle) and MM Asia (in Singapore).  As part of this scheme, any student member of SIGMM is eligible to apply, however, the students who are the first author of an accepted paper) are particularly encouraged. The value of the award will depend on the travel distance with up to US$2000 for long-haul travel and up to US$1000 for short-haul travel which are defined based on the location of the conference.  Details of this scheme and the link for submitting applications have already started to appear on the websites of some of these conferences.

With the SIGMM scheme supporting travel for student authors as a priority, some of these conferences applied for and have been approved for further funding to support other conference attendees and the IMX Conference in Barcelona, in June 2020 was awarded travel support for under-represented minorities while the MMSys conference in Istanbul in June 2020 was awarded travel support for non-student minorities. In both these cases the conferences themselves will administer selection and awarding of the funding. Student travel support was also awarded to the African Winter School in Multimedia, in Stellenbosch, South Africa in July 2020, an event which SIGMM also sponsors.

A number of other events which are not sponsored by SIGMM but which are closely related to our area also applied for funding to support student travel and the following have also been awarded funding for supporting student travel:



– the Adaptive Streaming Summer School, in Klagenfurt, Austria, July;

– the Content Based Multimedia Information (CBMI) Conference, in Lille, France, September;

– the International Conference on Quality of Multimedia Experience (QoMEX), in Athlone, Ireland, May, for female and under-represented minority students;

– the MediaEval Benchmarking Initiative for Multimedia Evaluation, workshop, late 2020.

All this funding, both the centralised and the special awards above, will help many students to travel to events in multimedia during 2020 and in addition to travel support, SIGMM will fund a number of events at some of our conferences.  These include a women and diversity lunch at CBMI in Lille, a diversity lunch and childcare support at the Information Hiding and Multimedia Security Workshop (IH&MMSec) in Denver, childcare support and a diversity and an inclusion panel discussion at IMX, a multimedia evaluation methodology workshop at the MediaEval workshop,
and childcare support and an N2Women meeting at MMSys.

We are also delighted to announce that SIGMM will also support some other activities besides travel and events and one of these is the costs of software development and presentation for Conflow at ACM Multimedia in Seattle. Conflow, and its predecessor ConfLab is a unique initiative from Hayley Hung and colleagues at TU Delft which encourages people with similar or complementary research interests to find each other at conference and ultimately to help them to connect with potential collaborators. It does this by instrumenting a physical space at an event with environmental sensors and distributing wearable sensors for participants who sign up and agree to have data about their interactions with others, captured, anonymised and used as a dataset for analysis. A pilot version at ACM Multimedia in Nice in 2019 called ConfLab ran with several dozen participants and was built around the notion of meeting the conference Chairs and this will be extended in 2020.

The final element of the SIGMM funding awarded recently was to the ICMR conference in Dublin in June which will be the testbed for calculation of a conference’s carbon footprint. ACM already has some initiatives in this area based on estimating the CO2e cost of air travel of conference attendees to/from the venue and there are software tools to help with this. The SIGMM funding will include this plus estimating the CO2e costs of local transport, food, accommodation, and more, plus it will also raise awareness of individual carbon footprints for delegates. This will be done for ICMR in a way that allows the process of calculating made available for other events.

SIGMM Sponsorship of Conference Fee Reduction

The third initiative which SIGMM is starting sponsorship of in 2020 is a reduction in the registration fees for SIGMM-sponsored conferences and this means for ICMR, MMSys, IMX, IH&MMSec, MULTIMEDIA and MM Asia. This has been a particular bug-bear for many of us so it is good to be able to do something about it.

Starting in 2020, SIGMM will sponsor US$100 toward conference registration fees for SIGMM members only, for the early-bird conference registrations. This will apply to students and non-students, and to ACM members and non-members. It means the conference registration choice may look a bit complicated but basically if you are an ACM member you get a certain reduction, if you are a SIGMM member you also get a reduction (from SIGMM), and if you are a student then you also get a reduction.  The amount of the reduction in the conference fee for being a SIGMM member ($100) is far more than the cost of joining SIGMM (which is either $20 or $15 for a student) thus it makes sense to join SIGMM and get the conference fee reduction and your SIGMM membership is an important thing for us.

The SIGMM Executive Committee believe this fee sponsorship is an appropriate way of giving back to the SIGMM community. Beyond 2020 we have not made a decision on sponsoring conference fee reductions, we will see how it works out in 2020 before deciding.

I’d also like to add one final note about attending our conferences and workshops.  We have a commitment to addressing diversity in our 25 in 25 strategy and we also have “access all areas” policy for our conferences. This means that a single registration fee allows access to all events and activities at our conferences … lunches, refreshments, dinners, etc., all bundled into one fee.  We also support those with special needs such as accessibility or dietary requirements and when these are brought to our attention, typically when an attendee registers, then we can put in place whatever support mechanisms are needed to maximise that attendee’s conference experience.  Our events strive to be harassment-free and pleasant conference experiences for all participants. We do not tolerate harassment of conference attendees and that means all our attendees, speakers and organizers are bound by ACM’s Policy Against Harassment. Participants are asked to confirm their commitment to upholding the policy when registering.

Finally, thank you for your support of SIGMM and our events. If there is one thing you can do to help us to help you, it is joining SIGMM, not just for the reduced conference registration fee but to show your support for what we do.  With a fixed rate of $20 or $15 for a student you’ll find details on the SIGMM Membership tab at http://sigmm.org/

Report from ACM SIG Heritage Workshop

By | | , , , ,
Leave a comment

“What does history mean to computer scientists?” – that was the first question that popped up in my mind when I was to attend the ACM Heritage Workshop at Minneapolis few months back. And needless to say, the follow up question was “what does history mean for a multimedia systems researcher?” As a young graduate student, I had the joy of my life when my first research paper on multimedia authoring (a hot topic those days) was accepted for presentation in the first ACM Multimedia in 1993, and that conference was held along side SIGGRAPH. Thinking about that, it gives multimedia systems researchers about 25 to 30 years of history. But what a flow of topics this area has seen: from authoring to streaming to content-based retrieval to social media and human-centered multimedia, the research area has been hot as ever. So, is it the history of research topics or the researchers or both? Then, how about the venues hosting these conferences, the networking events, or the grueling TPC meetings that prepped the conference actions?

Figure 1. Picture from the venue

With only questions and no clear answers, I decided to attend the workshop with an open mind. Most SIGs (Special Interest Groups) in ACM had representation at this workshop. The workshop itself was organized by the ACM History Committee. I understood this committee, apart from the workshop, organizes several efforts to track, record, and preserve computing efforts across disciplines. This includes identifying distinguished persons (who are retired but made significant contributions to computing), coming up with a customized questionnaire for the persons, training the interviewer, recording the conversations, curating them, archiving, and providing them for public consumption. Efforts at most SIGs were mostly based on the website. They were talking about how they try to preserve conference materials such as paper proceedings (when only paper proceedings were published), meeting notes, pictures, and videos. For instance, some SIGs were talking about how they tracked and preserved ACM’s approval letter for the SIG! 

It was very interesting – and touching – to see some attendees (senior Professors) coming to the workshop with boxes of materials – papers, reports, books, etc. They were either downsizing their offices or clearing out, and did not feel like throwing the material in recycling bins! These materials were given to ACM and Babbage Institute (at University of Minnesota, Minneapolis) for possible curation and storage.

Figure 2. Galleries with collected material

ACM History committee members talked about how they can fund (at a small level) projects that target specific activities for preserving and archiving computing events and materials. ACM History Committee agreed that ACM should take more responsibility in providing technical support to web hosting – obviously, not sure whether anything tangible would result.

Over the two days at the workshop, I was getting answers to my questions: History can mean pictures and videos taken at earlier MM conferences, TPC meetings, SIGMM sponsored events and retreats. Perhaps, the earlier paper proceedings that have some additional information than what is found in the corresponding ACM Digital Library version. Interviews with different research leaders that built and promoted SIGMM.

It was clear that history meant different things to different SIGs, and as SIGMM community, we would have to arrive at our own interpretation, collect and preserve that. And that made me understand the most obvious and perhaps, the most important thing: today’s events become tomorrow’s history! No brainer, right? Preserving today’s SIGMM events will give us a richer, colorful, and more complete SIGMM history for the future generations!

For the curious ones:

ACM Heritage Workshop website is at: https://acmsigheritage.dash.umn.ed

Some of the workshop presentation materials are available at: https://acmsigheritage.dash.umn.edu/uncategorized/class-material-posted/

Introducing the new role of the Director of Diversity and Outreach

By | | ,
1 Comment

sigmm-logo2

Over the last few decades SIGMM has grown with regard to the number and size of conferences and workshops we organize and sponsor, and we have grown with regard to our international outreach. Researchers from all over the world now participate in SIGMM and its many activities. In the same way in which we grow internationally with regard to members, with regard to the participants attending our conferences and their different backgrounds, the diversity of SIGMM is also growing. However, we can observe that diversity and all the aspects it brings to a society is not necessarily “just something” but needs to be supported and embraced by a cultural change of the organization and all its members.

Introducing the new role of SIGMM Director of Diversity and Outreach

In 2019, SIGMM created the new role of SIGMM Director of Diversity and Outreach with a variety of roles and responsibilities, for an initial 3-year period. Creation of this position is a sign and an action to establish future activities and an invitation on a more formal level to move our work in this area beyond anecdotal activities and personal engagement. The Director of Diversity and Outreach will be a voting member of the SIGMM Executive Committee. The EC Chair has drafted and circulated a role specification for this and sent a call to the community for expressions of interest in the role in Spring 2019. The confirmation of an appointment was made by the EC in May 2019. For the inaugural appointment 2019-2021, Susanne Boll has has been elected unanimously for this role. With this new director of diversity and outreach, SIGMM is supporting and developing diversity on an institutional level as a voting member of the SIGMM Executive Committee.

First Initiative

As a first initiative, the SIGMM EC has decided on a “25 in 25’’ strategy to strategically increase the participation of women in SIGMM and all its activities. This strategy aims at increasing the participation of women in all activities and committees of SIGMM to at least 25% by 2025. 

It can be observed that female participation in SIGMM has been low over many years. Even though there were good initiatives over the last decades, we have failed to include a proportionate number of women researchers into the SIG and into our executive structures and event organization. As we observe that about 25% of all CS degrees in computer science are held by women, we may well expect that ACM find these numbers reflected in the number of women active within their Special Interest Groups –  which is not the case in SIGMM. We strongly believe that it will only change if we as SIGMM take action. This action will take place on three levels. 

With the SIGMM Executive Actions we aim at an obligatory inclusion of women in the steering committees of SIGMM. For the coming elections in 2021, we will implement a voting scheme by which the two leading chair positions, SIGMM Chair and SIGMM Vice Chair, will be filled by a man and a woman. For the forthcoming SIGMM officer elections, SIGMM will also fill other candidate roles with two individuals, one man and one woman to ensure gender equality on the level of the different roles. 

With the SIGMM Conference Steering Actions for all forthcoming appointments to the individual Steering Committees, the Steering Committees will invite female candidates in order to reach at least a 25% share of their memberships. All Steering Committees will have their members online and maintain a history of their SC and the different positions on the organizing committee of their related conferences online.

With the SIGMM Conference Actions we request that all SIGMM-sponsored conferences have at least 25% representation of women in all roles of their organizing committee which will be observed for all forthcoming bids for conferences.  We aim at organising committees in which the many volunteer roles for our conferences, such as general chair, workshop chair, tutorial chair, panel chair, web chair, local chair, or proceedings chair could be filled by two individuals, one woman and one man.  

The SIGMM Director of Diversity and Outreach will observe the implementation of these rules and report on the state and progress annually within the EC, at the annual SIGMM business meeting at ACM Multimedia and publish a report in SIGMM Records.

What’s next?

The creation of the role of the  SIGMM Director of Diversity and Outreach was a first step. The initiative “25 in 25” is the first set of initiatives and further initiatives will follow. Currently, we are already in discussion about actions across SIGMM events such as travel support, childcare, mentoring support, support for speakers and targeted meetings. We will regularly inform you through our regular newsletter, website, and meetings. 

SIGMM understands the new role as actively pushing and developing diversity and outreach within SIGMM. The new director is here to listen and to act for a better diversity of our Special Interest Group MM, our activities and our outreach to the multimedia community.  All SIGMM members are strongly invited to support the activities of the director of outreach and the different initiatives. The director will also seek and actively exchange with and learn from other Special Interest Groups within ACM and other societies. If you want to get involved please join us (contact: Susanne Boll boll@acm.org).

Report from MMSYS 2019 – by Alia Sheikh

By | | , , ,
Leave a comment

Alia Sheikh (@alteralias) is researching immersive and interactive content. At present she is interested in the narrative language of immersive environments and how stories can best be choreographed within them.

Being part of an international academic research community and actually meeting said international research community are not exactly the same thing it turns out. After attending the 2019 ACM MMSys conference this year, I have decided that leaving the office and actually meeting the people behind the research is very worth doing.

This year I was invited to give an overview presentation at ACM MMSys ’19, which was being hosted at the University of Massachusetts. The MMSys, NOSSDAV and MMVE (International Workshop on Immersive Mixed and Virtual Environment Systems) conferences happen back to back, in a different location each year. I was asked to talk about some of our team’s experiments in immersive storytelling at MMVE. This included our current work on lightfields and my work on directing attention in, and the cinematography of, immersive environments.

To be honest it wasn’t the most convenient time to decide to catch a plane to New York and then a train to Boston for a multi-day conference, but it felt like the right time to take a break from the office and find out what the rest of the community had been working on.

Fig.1: A picturesque scene from the wonderful University of Massachussetts Amherst campus

Fig.1: A picturesque scene from the wonderful University of Massachussetts Amherst campus

I arrived at Amherst the day before the conference and (along with another delegate who had taken the same bus) wandered the tranquil university grounds slightly lost before being rescued by the ever calm and cheerful Michael Zink. Michael is the chair of the MMSys organising committee and someone who later spent much of the conference introducing people with shared interests to each other – he appeared to know every delegate by name.

Once installed in my UMass hotel room, I proceeded to spend the evening on my usual pre-conference ritual: entirely rewriting my presentation.

As the timetable would have it, I was going to be the first speaker.

Fig 2: Attendees at MMSys 2019 taking their seats

Fig. 2: Attendees at MMSys 2019 taking their seats

Fig 3: Alia in full flow during our talk on day 1

Fig. 3: Alia in full flow during our talk on day 1

I don’t actually know why I do this to myself, but there is something about turning up to the event proper that gives you a sense of what will work for that particular audience, and Michael had given me a brilliantly concise snapshot of the type of delegate that MMSys attracts – highly motivated, expert on the nuts and bolts of how to get data to where it needs to be and likely to be interested in a big picture overview of how these systems can be used to create a meaningful human connection.

Using selected examples from our research, I put together a talk on how the experience of stories in high tech immersive environments differs from more traditional formats, but, once the language of immersive cinematography is properly understood, we find that we are able to create new narrative experiences that are both meaningful and emotionally rich.

The next morning I walked into an auditorium full of strangers filing in, gave my talk (I thought it went well?) and then sank happily into a plush red flip-seat chair safe in the knowledge that I was free to enjoy the rest of the event.

The next item was the keynote and easily one of the best talks I have ever experienced at a conference. Presented by Professor Nimesha Ranasinghe it was a masterclass in taking an interesting problem (how do we transmit a full sensory experience over a network?) And presenting it in such a way as to neatly break down and explain the science (we can electrically stimulate the tongue to recreate a taste!) while never losing sight of the inherent joy in working on the kind of science you dream of as a child (therefore electrified cutlery!).

Fig. 4: Professor Nimesha Ranasinghe during his talk on Multisensory experiences

Fig. 4: Professor Nimesha Ranasinghe during his talk on Multisensory experiences

Fig 5: Multisensory enhanced multimedia - experiences of the future ?

Fig. 5: Multisensory enhanced multimedia – experiences of the future ?

Fig6: Networking and some delicious lunch

Fig. 6: Networking and some delicious lunch

At lunch I discovered the benefit of having presented my talk early – I made a lot of friends with people who had specific questions about our work, and got a useful heads up on work they were presenting either in the afternoon’s long papers session or the poster session.

We all spent the evening at the welcome reception on the top floor of UMass Hotel, where we ate a huge variety of tiny, delicious cakes and got to know each other better. It was obvious that in some cases, researchers that might collaborate remotely all year, were able to use MMSys as an excellent opportunity to catch up. As a newcomer to this ACM conference however, I have to say that I found it a very welcoming event, and I met a lot of very friendly people many of them working on research that was entirely different to my own, but which seemed to offer an interesting insight or area of overlap.

I wasn’t surprised that I really enjoyed MMVE – virtual environments are very much my topic of interest right now. But I was delighted by how much of MMSys was entirely up my street. ACM MMSys provides a forum for researchers to present and share their latest research findings in multimedia systems, and the conference cuts across all media/data types to showcase the intersections and the interplay of approaches and solutions developed for different domains. This year, the work presented on how to best encode and transport mixed reality content, as well as predict head motion to better encode and deliver the part of a spherical panorama a viewer was likely to be looking at, was particularly interesting to me. I wondered whether comparing the predicted path of user attention to the desired path of user attention, would teach us how to better control a users attention within a panoramic scene, or whether peoples viewing patterns were simply too variable. In the Open Datasets & Software track, I was fascinated by one particular dataset: “ A Dataset of Eye Movements for the Children with Autism Spectrum Disorder”. This was a timely reminder for me that diversity within the audience needed to be catered for when designing multimedia systems, to avoid consigning sections of our audience to a substandard experience.

Of the demos, there were too many interesting ones to list, but I was hugely impressed by the demo for Multi-Sensor Capture and Network Processing for Virtual Reality Conferencing. This used cameras and Kinects to turn me into a point cloud and put a live 3D representation of my own physical body in a virtual space.A brilliantly simple and incredibly effective idea and I found myself sitting next to the people responsible for it at a talk later that day and discussing ways to optimise their data compression.

Despite wearing a headset that allowed me to see the other participants, I was still able to see and therefore use my own hands in the real world – even extending to picking up and using my phone.

Fig7: Trying out some cool demos during a bustling demo session

Fig. 7: Trying out some cool demos during a bustling demo session

Fig. 8: An example of the social media interaction from my "tweeting"

Fig. 8: An example of the social media interaction from my “tweeting”

Amusingly, I found that I was (virtually) sat next to a point-cloud of TNO researcher Omar Niamut which led to my favourite twitter exchange of the whole conference. I knew Omar from online, but we had never actually managed to meet in real life. Still, this was the most life-like digital incarnation yet!

I really should mention the Women’s and Diversity lunch event which (pleasingly) was attended by both men and women and offered some absolutely fascinating insights.

These included: the value of mentors over the course of a successful academic life, how a gender pay-gap is inextricably related to work family policies and steps that have successfully been taken by some countries and organisations to improve work-life balance for all genders.

It was incredibly refreshing to see these topics being discussed both scientifically and openly. The conversations I had with people afterwards as they opened up about their own experiences of work and parenthood, were among the most interesting I have ever had on the topic.

Another nice surprise – MMSys offers childcare grants available for conference attendees who are bringing small children to the conference and require on-site childcare or who incur extra expenses in leaving their children at home. It was very cheering to see that the Inclusion Policy did not stop at simply providing interesting talks, but also translated into specific inclusive action.

Fig. 9: Women’s and Diversity lunch! What a wonderful initiative - well done MMSys and SIGMM

Fig. 9: Women’s and Diversity lunch! What a wonderful initiative – well done MMSys and SIGMM

I am delighted that I made the decision to attend MMSys. I had not realised that I was feeling somewhat detached from my peers and the academic research community in general, until I was put in an environment which contained a concentrated amount of interesting research, interesting researchers and an air of collaboration and sheer good will. It is easy to get tunnel vision when you are focused on your own little area of work, but every conversation I had at the conference reminded me that research does not happen in a vacuum.

Fig. 10: A fascinating talk at the Women’s and Diversity lunch - it initiated great post event discussions!

Fig. 10: A fascinating talk at the Women’s and Diversity lunch – it initiated great post event discussions!

Fig. 11: The food truck experience - one of many wonderful social aspects to MMSys 2019

Fig. 11: The food truck experience – one of many wonderful social aspects to MMSys 2019

I could write a thousand more words about every interesting thing I saw or person I met at MMSys, but that would only give you my own specific experience of the conference. (I did live tweet* a lot of the talks and demos just for my own records and that can all be found here: https://twitter.com/Alteralias/status/1148546945859952640?s=20)

Fig. 12: Receiving the SIGMM Social Media Reporter Award for MMSys 2019!

Fig. 12: Receiving the SIGMM Social Media Reporter Award for MMSys 2019!

Whether you were someone I was sitting next to at a paper session, a person I spoke to standing next to in line at the food truck (one of the many sociable meal events) or someone who demoed their PhD work to me, thank you so much for sharing this event with me.

Maybe I will see you at MMSys 2020.

* p.s it turns out that if you live-tweet an entire conference, Niall gives you a Social Media Reporter award.

An Interview with Professor Susanne Boll

By | | , ,
Leave a comment

Describe your journey into research from your youth up to the present. What foundational lessons did you learn from this journey? 

My journey into research started with my interest in computers and computer science at school while I was still in my early years at that time. I liked all the STEM subjects and was very good at these in school. I got in touch with programming and the first Mac in high school when my physics teacher started the first basic programming course. After highschool, I continued on this journey and became a Mathematical-Technical1 Assistant and continued studying CS and went on to do a PhD, always driven by the desire that I could learn more, could explore and understand more of this field.

Why were you initially attracted to Multimedia? 

Susanne at

Susanne Boll at the beginning of her research career in 2001

I was initially attracted by multimedia when information systems started to look at novel methods of integrating large amounts of unstructured multimedia and different media types into structured database systems. I joined the GMD Institute for Integrated Publication and Information Systems who were working on multimedia database systems. My PhD was on multimedia document models for representing and replaying multimedia presentations in the context of multimedia information systems. One of the most inspiring early events was a small but very nice IFIP working conference on Database Semantics – Semantic Issues in Multimedia Systems in New Zealand 1999 where I met many researchers from the multimedia community some of whom I still consider my research friends today. I stayed in the field of multimedia but as my work was always relating to the applications of multimedia and the interaction with the user it was not surprising that I moved into the field of Human Computer Interaction and SIGCHI in which I am an active member also today. Over the last three decades I have worked in the field of interactive multimedia and human computer interaction – in different application domains from personal media to health, from mobility to industry 4.0. To cite a much valued friend of mine whom I just met again – “I enjoy when my research makes me smile”, when I can see how research can be translated in applications for a better use.

Why did I volunteer for the role of the director for diversity and outreach? 

Professor Susanne Boll in 2019

Professor Susanne Boll in 2019

Over more than three decades now I was supporting gender equality as a mentor, in different roles, in committees and institutions, by speaking up and by driving actions. Within the multimedia community I observed that there are many individuals supporting and acting for a better gender equality, however, it remained efforts of individuals and we as a community were not able to turn this into a collective understanding. 

There were actually a few recent events related to SIGMM that made truly sad and consider if I should leave this community which I at the same time consider my home community. Some years ago I was observing in a panel in which only men were discussing the future and challenges in multimedia. Observing this was painful for me. I knew and met with each of them individually over the years and they were interesting researchers and great mentors. But that panel it made again obvious that we as a community failed to be inclusive also with regard to the women. Why would there be not an excellent woman would have her say in that panel? Why would not someone organizing the panel consider to be inclusive with regard to gender? Why would not the panelists, when they are invited, ask who else would be on the panel and encourage this?

When I talk about gender equality in these days I almost immediately get the reaction that gender is not diversity. People say that looking at gender equality would be too short sighted and that I should care more about diversity and not gender alone. So let me clearly say that I am well aware that diversity is not gender it is much more than that.  But, don’t let the perfect be the enemy of the good. My personal story starts with gender equality in STEM fields. Looking at women participation in SIGMM, I decided that the actions described in the “25 in 25’’ strategy would be a good starting point for my new role – it is just the beginning.

What are my plans serving in this position?

Within SIGMM, we need to understand and fully embrace the different dimensions of diversity. We should not use the term in the sense of an easy cover-up of a multitude of aspects in which the individual needs get blurred. I sometimes have the feeling as if one aspect of diversity could be traded for another one, and the term was used as if there was a measure that there is “sufficient” diversity in some setting. 

As a  director for diversity and outreach I will be caring about the richness of diversity.  I want to bring the different dimensions of diversity into the multimedia community and make us understand, embrace listen and take action for better diversity and outreach of SIGMM.

1Mathematical-Technical Assistant (MaTA, MA or MTA for short; also: mathematical-technical software developer) is the occupational title of a recognised training occupation according to the Vocational Training Act in Germany, which has existed since the mid-1960s. It is the first non-academic training occupation in data processing.

Bios

Prof. Susanne Boll: 

Susanne Boll is a full professor for Media Informatics and Multimedia Systems at the University of Oldenburg and a member of the board of the OFFIS-Institute for Information Technology. OFFIS belongs to the top 5% research institutes among the non-university institutes in computer science in Germany. Over the last two decades, she has consistently achieved highly competitive research results in the field of multimedia and human–computer interaction. She has actively been driving these fields of research by many scientific research projects and organization of highly visible events in the field. Her scientific results have been published in competitive peer-reviewed international conferences such as Multimedia, CHI, MobileHCI, AutomotiveUI, DIS, and IDC, as well as internationally recognized journals. Her research makes competitive contributions to the field of human computer interaction and ubiquitous computing. Her research projects also have a strong connection to industry partners and application partners and addresses highly relevant challenges in the applications field of automation in transportation systems as well as health care technologies. I am an active member of the scientific community and have co-chaired and organized many international events in my field. Her teaching follows combination of theoretical foundations with team-oriented and research-oriented practical assignments.  She currently leads a highly visible international team of researchers (PhD students, research associates, post docs, senior principal scientists).

The V3C1 Dataset: Advancing the State of the Art in Video Retrieval

By | | , ,
Leave a comment

Download

In order to download the video dataset as well as its provided analysis data, please follow the instructions described here:

https://github.com/klschoef/V3C1Analysis/blob/master/README.md

Introduction

Standardized datasets are of vital importance in multimedia research, as they form the basis for reproducible experiments and evaluations. In the area of video retrieval, widely used datasets such as the IACC [5], which has formed the basis for the TRECVID Ad-Hoc Video Search Task and other retrieval-related challenges, have started to show their age. For example, IACC is no longer representative of video content as it is found in the wild [7]. This is illustrated by the figures below, showing the distribution of video age and duration across various datasets in comparison with a sample drawn from Vimeo and Youtube.

datasets1

 

datasets2

Its recently released spiritual successor, the Vimeo Creative Commons Collection (V3C) [3], aims to remedy this discrepancy by offering a collection of freely reusable content sourced from the video hosting platform Vimeo (https://vimeo.com). The figures below show the age and duration distributions of the Vimeo sample from [7] in comparison with the properties of the V3C.datasets3

datasets4

The V3C is comprised of three shards, consisting of 1000h, 1200h and 1500h of video content respectively. It consists not only of the original videos themselves, but also comes with video shot-boundary annotations, as well as representative key-frames and thumbnail images for every such video shot. In addition, all the technical and semantic video metadata that was available on Vimeo is provided as well. The V3C has already been used in the 2019 edition of the Video Browser Showdown [2] and will also be used for the TRECVID AVS Tasks (https://www-nlpir.nist.gov/projects/tv2019/) starting 2019 with a plan for future usage in the coming several years. This video provides an overview of the type of content found within the dataset

Dataset & Collections

The three shards of V3C (V3C1, V3C2, and V3C3) contain Creative Commons videos sourced from video hosting platform Vimeo. For this reason, the elements of the dataset may be freely used and publicly shared. The following table presents the composition of the dataset and the characteristics of its shards, as well as the information on the dataset as a whole.

Partition V3C1 V3C2 V3C3 Total
File Size (videos) 1.3TB 1.6TB 1.8TB 4.8TB
File Size (total) 2.4TB 3.0TB 3.3TB 8.7TB
Number of Videos 7’475 9’760 11’215 28’450
Combined

Video Duration

 1’000 hours,

23 minutes,

50 seconds

 1’300 hours,

52 minutes,

48 seconds

 1’500 hours,

8 minutes,

57 seconds

 3801 hours,

25 minutes,

35 seconds
Mean Video Duration 8 minutes,

2 seconds

 7 minutes,

59 seconds

 8 minutes,

1 seconds

 8 minutes,

1 seconds
Number of Segments 1’082’659 1’425’454 1’635’580 4’143’693

Similarly to IACC, V3C contains a master shot reference, which segments every video into non-overlapping shots based on the visual content of the videos. For every single shot, a representative keyframe is included, as well as the thumbnail version of that keyframe. Furthermore, for each video, identified by a unique ID, a metadata file is available that contains both technical as well as semantic information, such as the categories. Vimeo categorizes every video into categories and subcategories. Some of the categories were determined to be non-relevant for visual based multimedia retrieval and analytical tasks, and were dropped during the sourcing process of V3C. For simplicity reasons, subcategories were generalized into their parent categories and are, for this reason, not included. The remaining Vimeo categories are:

  • Arts & Design
  • Cameras & Techniques
  • Comedy
  • Fashion
  • Food
  • Instructionals
  • Music
  • Narrative
  • Reporting & Journals

Ground Truth and Analysis Data

As described above, the ground truth of the dataset consists of (deliberately over-segmented) shot boundaries as well as keyframes. Additionally, for the first shard of the V3C, the V3C1, we have already performed several analyses of the video content and metadata in order to provide an overview of the dataset [1]. 

In particular, we have analyzed specific content characteristics of the dataset, such as:

  • Bitrate distribution of the videos
  • Resolution distribution of the videos
  • Duration of shots
  • Dominant color of the keyframes
  • Similarity of the keyframes in terms of color layout, edge histogram, and deep features (weights extracted from the last fully-connected layer of GoogLeNet).
  • Confidence range distribution of the best class for shots detected by NasNet (using the best result out of the 1000 ImageNet classes) 
  • Number of different classes for a video detected by NasNet (using the best result out of the 1000 ImageNet classes)
  • Number of shots/keyframes for a specific content class
  • Number of shots/keyframes for a specific number of detected faces

This additional analysis data is available via GitHub, so that other researchers can take advantage of it. For example, one could use a specific subset of the dataset (only shots with blue keyframes, only videos with a specific bitrate or resolution, etc.) for performing further evaluations (e.g., for multimedia streaming, video coding, but also for image and video retrieval, of course). Additionally, due the public dataset and the analysis data, one could easily create an image and video retrieval system and use it either for participation in competitions like the Video Browser Showdown [2], or for submitting other evaluation runs (TRECVID Ad-hoc Video Search Task).

Conclusion

In the broad field of multimedia retrieval and analytics, one of the key components of research is having useful and appropriate datasets in place to evaluate multimedia systems’ performance and benchmark their quality. The usage of standard and open datasets enables researchers to reproduce analytical experiments based on these datasets and thus validate their results. In this context, the V3C dataset proves to be very diverse in several useful aspects (upload time, visual concepts, resolutions, colors, etc.). Also it has no dominating characteristics and provides a low self-similarity (i.e., few near duplicates) [3].

Further, the richness of V3C in terms of content diversity and content attributes enables benchmarking multimedia systems in close-to-reality test environments. In contrast to other video datasets (cf. YouTube-8M [4] and IACC [5]), V3C also provides a vast number of different video encodings and bitrates per second, so that it enables research focusing on video retrieval and analytical tasks regarding those attributes. The large number of different video resolutions (and to a lesser extent frame-rates) makes this dataset interesting for video transport and storage applications such as the development of novel encoding schemes, streaming mechanisms or error-correction techniques. Finally, in contrast to many current datasets, V3C also provides support for creating queries for evaluation competitions, such as VBS and TRECVID [6].

References

[1] Fabian Berns, Luca Rossetto, Klaus Schoeffmann, Christian Beecks, and George Awad. 2019. V3C1 Dataset: An Evaluation of Content Characteristics. In Proceedings of the 2019 on International Conference on Multimedia Retrieval (ICMR ’19). ACM, New York, NY, USA, 334-338.

[2] Jakub Lokoč, Gregor Kovalčík, Bernd Münzer, Klaus Schöffmann, Werner Bailer, Ralph Gasser, Stefanos Vrochidis, Phuong Anh Nguyen, Sitapa Rujikietgumjorn, and Kai Uwe Barthel. 2019. Interactive Search or Sequential Browsing? A Detailed Analysis of the Video Browser Showdown 2018. ACM Trans. Multimedia Comput. Commun. Appl. 15, 1, Article 29 (February 2019), 18 pages.

[3] Rossetto, L., Schuldt, H., Awad, G., & Butt, A. A. (2019). V3C–A Research Video Collection. In International Conference on Multimedia Modeling (pp. 349-360). Springer, Cham.

[4] Abu-El-Haija, S., Kothari, N., Lee, J., Natsev, P., Toderici, G., Varadarajan, B., & Vijayanarasimhan, S. (2016). Youtube-8m: A large-scale video classification benchmark. arXiv preprint arXiv:1609.08675.

[5] Paul Over, George Awad, Alan F. Smeaton, Colum Foley, and James Lanagan. 2009. Creating a web-scale video collection for research. In Proceedings of the 1st workshop on Web-scale multimedia corpus (WSMC ’09). ACM, New York, NY, USA, 25-32. 

[6] Smeaton, A. F., Over, P., and Kraaij, W. 2006. Evaluation campaigns and TRECVid. In Proceedings of the 8th ACM International Workshop on Multimedia Information Retrieval (Santa Barbara, California, USA, October 26 – 27, 2006). MIR ’06. ACM Press, New York, NY, 321-330.

[7] Luca Rossetto & Heiko Schuldt (2017). Web video in numbers-an analysis of web-video metadata. arXiv preprint arXiv:1707.01340.

Report from ACM MM 2018 – by Ana García del Molino

By | | , ,
Leave a comment

Seoul, what a beautiful place to host the premier conference on multimedia! Living in never-ending summer Singapore, I fell in love with the autumn colours of this city. The 26th edition of the ACM International Conference on Multimedia was held on October 22-26 of 2018 at the Lotte Hotel in Seoul, South Korea. It packed a full program including a very diverse range of workshops and tutorials, oral and poster presentations, art exhibits, interactive demos, competitions, industrial booths, and plenty of networking opportunities.

For me, this edition was a special one. About to graduate, with my thesis half written, I was presenting two papers. So of course, I was both nervous and excited. I had to fly to Seoul a few days ahead just to prepare myself! I was so motivated, I somehow managed to get myself a Best Social Media Reporter Award (who would have said… Me! A reporter!).

So, enough with the intro. Let’s get to the juice. What happened in Seoul between the 22nd and 26th of October 2018?

The first and last day of the conference were dedicated to Workshops and Tutorials. Those were a mix between Deep Learning themed and social applications of multimedia. The sessions included tutorials like “Interactive Video Search: Where is the User in the Age of Deep Learning?” that discussed the importance of the user in the collection of datasets, evaluation, and also interactive search, as opposed to using deep learning to solve challenges with big labelled datasets. In “Deep Learning Interpretation” Jitao Sang presented the main multimedia problems that can’t be addressed using deep learning. On the other hand, new and important trends related to social media (analysis of information diffusion and contagion, user activities and networking, prediction of real-world events, etc) were discussed in the tutorial “Social and Political Event Analysis using Rich Media”. The workshops were mainly user-centred, with special interest in affective computing and emotion analysis and use for multimedia (EE-USAD, ASMMC – MMAC 2018, AVEC 2018).

The conference kick-started with a wonderful keynote by Marianna Obrist. With “Don’t just Look – Smell, Taste, and Feel the Interaction” she showed us how to bring art into 4D by using technology, driving us through a full sensory experience that let us see, hear, and almost touch and smell. Ernest Edmonds also delved into how to mix art and multimedia in “What has art got to do with it?” but this time the other way around: what can multimedia research learn from the artists? Three industry speakers completed the keynote program. Xian-Sheng Hua from Alibaba Group shared their efforts towards visual Intelligence in “Challenges and Practices of Large-Scale Visual Intelligence in the Real-World”. Gary Geunbae Lee shared Samsung’s AI user experience strategy in “Living with Artificial Intelligence Technology in Connected Devices around Us.” And Bowen Zhou presented JD.com’s brand-new concept of Retail as a Service in “Transforming Retailing Experiences with Artificial Intelligence”.

This year’s program included 209 full papers, from a total of 757 submissions. 64 papers were allocated 15-minute oral presentations, while the others got a 90-second spotlight slot in the fast-forward sessions.  The poster sessions and the oral sessions run at the same time. While this was an inconvenience for poster presenters having to leave the poster to attend the oral sessions or miss them, the coffee breaks took place at the same location as the posters, so that was a win-win: chit-chat while having cookies and fruits? I’m in! In terms of content, half of the submissions were to only two areas: Multimedia and Vision and Deep Learning for Multimedia. But who am I to judge, when I had two of those myself! Many members of the community noted that the conference is becoming more and more deep learning, and less multimodal. To compensate, the workshops, tutorials and demos were mostly pure multimedia.

The challenges, competitions, art exhibits and demos happened in the afternoons, so at times it was hard to choose where to head to. So many interesting things happening all around the place! The art exhibit had some really cool interactive art installations, such as “Cellular Music”, that created music from visual motion. Among the demos, I found particularly interesting AniDance, an LSTM-based algorithm that made 3D models dance to the given music; SoniControl, an ultrasonic firewall for NFC protection; MusicMapp, a platform to augment how we experience music; and The Influence Map project, to explore who has influenced each scientist, and who did they most influence through their career.

Regarding diversity, I feel there is still a long way to go. Being in Asia, it makes sense that almost half of the attendees came from China. However, the submission numbers speak by themselves: less than 20% of submissions came from out of Asia, with just one submission from Africa (that’s a 0.13%!) Diversity is not only about gender, folks! I feel like more efforts are needed to facilitate the integration of more collectives in the multimedia community. One step at a time.

The next edition will take place at the NICE ACROPOLIS Convention Center in Nice, France from 21-25 October 2019. The ACM reproducibility badge system will be implemented for the first time at this 27th edition, so we may be seeing many more open-sourced projects. I am so looking forward to this!

Opinion Column: Evolution of Topics in the Multimedia Community

By | | , ,
Leave a comment

For this edition of the SIGMM Opinion Column, we asked members of the Multimedia community to share their impressions about the shift of scientific topics in the community over the years, namely the evolution of “traditional” and “emerging” Multimedia topics. 

This subject has emerged in several conversations over the 2 years of history of the SIGMM Opinion Column, and we report here a summary of recent and old discussions, happened over different channels – our Facebook group, the SIGMM Linkedin group, and in-person conversations between the column editors and MM researchers – with hopes, fears and opinions around this problem. We want to thank all participants for their precious contribution to these discussions.

Historical Perspective of Topics in ACM MM

opinion11_2_1This year, ACM Multimedia turns 27. Today, MM is a large premium conference with hundreds of paper submissions every year, spanning 12 different thematic areas spanning across the wide spectrum of multimedia topics. But back at the beginning of MM’s history, the scale of the topic range was very different.

In the first editions of the conference, a general call for papers encouraged submissions about “technology, tools and techniques for the construction and delivery of high quality, innovative multimedia systems and interfaces”. Already in its 3rd edition, MM featured an Arts and Multimedia program. Starting from 2004, the conference offered three tracks for paper submissions: content (Multimedia analysis, processing, and retrieval), Systems (Multimedia networking and system support), and Applications (Multimedia tools, end-systems, and applications), plus a “Brave New Topics” track for work-in-progress submissions. Later on, the Human-Centered Multimedia track was included in the projects. In 2011, after a conference review, the ACM MM program went beyond the notion of “tracks”, and the concept of areas was introduced to allow the community to “solicit papers from a wide range of timely multimedia-related topics” (see the ACMM11 website). In 2014, the areas became 14, including, among others, Music, Speech and Audio Processing in Multimedia, and Social Media and Collective Online Presence. After a retreat in 2014, starting from 2015, areas are grouped in larger “Themes”, the core thematic areas of ACM Multimedia. After the last retreat in 2014, no major changes were introduced in the thematic structure of the conference.

Dynamics of Evolution Emerging Topics

Emerging topics and less mature works are generally welcome at conferences’ workshops. In our discussions, most members of the community agree that “you’ll see great work there, and very fruitful discussions due to the common focus on the workshop theme”. When emerging topics become more popular, they can be promoted to conference areas, as it happened for the “music, speech and audio” theme. 

It was observed in our community conversations that, while this upgrade to the main conference is great for visibility, being a separate, relatively novel area could lead to isolation: the workload for reviewers specialized on emerging topics could become too high, given that they are assigned to works in other areas; and the flat acceptance rate across all conference themes could mean that even accepting 2 submissions from an emerging topic area would give ‘unreasonably’ high acceptance rate, thus leading to many good papers (even with 3 accepts) having to be rejected. Participants to our forums noticed that these dynamics are somehow “counteracted the ‘Multimedia’ and multidisciplinary nature of the field”, they prevent conferences from growing and eventually hurt emerging topics. One solution proposed to balance this effect is to “maintain a solid specialized reviewer pool (where needed managed by someone from the field), which however would be distributed over relevant MM areas”, rather than forming a new area.

It was also noted that some emerging topics in their early stage would most likely not have an appropriate workshop. Therefore, it is important for the main conference to have places to accept such early works, thus making tracks such as the short paper tracks or the brave new idea track absolutely crucial for the development of  novel topics.

The Near-Future of Multimedia

In multiple occasions, MM community members shared their thoughts about how they would like to see the Multimedia community evolve around new topics.

There are a few topics that emerged in the past and that the community wishes they continued growing, and these include interactive Multimedia applications, as well as music-related Multimedia technology, Multimedia in cooking spaces, and arts and Multimedia. It was also pointed out that, although very important for Multimedia applications, topics around compression technology are also often given low weight in Multimedia spaces, and that MM should encourage submissions in the domain of machine learning concepts applied to compression.

There are also a few areas that are emerging across different sub-communities in computer science, and that, according to our community members, we should be encouraging to grow within the Multimedia field as well. These include works in digital health exploring the power of Multimedia for health care and monitoring, research around applications of Multimedia for good, understanding how the technologies we develop can help having a real impact on society, and discussions around the ethics and responsibility of Multimedia technologies, encouraging fair, transparent, inclusive and accountable Multimedia tools.

The Future of Multimedia

The future of MM according to the participants of the discussion goes beyond the forms we know today, as new technologies could significantly broaden and shake the current applicative paradigm of Multimedia. 

The upcoming 5G technology will enable a plethora of applications that are now extremely limited by the lack of bandwidth. This could go from mobile virtual reality, to interconnection with objects and, of course, smart cities. To extract meaningful information to be presented to the user, various and highly diverse data streams will need to be treated consistently. And Multimedia researchers will develop methods, applications, systems and models to understand how to properly develop and impact this field. Likewise, this technology will push the limits of what is currently possible in terms of content demand and interaction with connected objects. We will see technologies for hyper-personalization, dynamic user interaction and real-time video personalization. These technologies will be enabled by the study of how film grammar and storytelling works for novel content types like AR, VR, panoramic and 360° video, by research around novel immersive media experiences, and by the design of new media formats, with novel consumption paradigms.

Multimedia has a bright future, with new, exciting emerging topics to be discussed and encouraged. Perhaps time for a new retreat or for a conference review?

Report from ACM ICMR 2018 – by Cathal Gurrin

By | | , ,
Leave a comment

 

Multimedia computing, indexing, and retrieval continue to be one of the most exciting and fastest-growing research areas in the field of multimedia technology. ACM ICMR is the premier international conference that brings together experts and practitioners in the field for an annual conference. The eighth ACM International Conference on Multimedia Retrieval (ACM ICMR 2018) took place from June 11th to 14th, 2018 in Yokohama, Japan’s second most populous city. ACM ICMR 2018 featured a diverse range of activities including: Keynote talks, Demonstrations, Special Sessions and related Workshops, a Panel, a Doctoral Symposium, Industrial Talks, Tutorials, alongside regular conference papers in oral and poster session. The full ICMR2018 schedule can be found on the ICMR 2018 website <http://www.icmr2018.org/>. The organisers of ACM ICMR 2018 placed a large emphasis on generating a high-quality programme and in 2018; ICMR received 179 submissions to the main conference, with 21 accepted for oral presentation and 23 for poster presentation. A number of key themes emerged from the published papers at the conference: deep neural networks for content annotation; multimodal event detection and summarisation; novel multimedia applications; multimodal indexing and retrieval; and video retrieval from regular & social media sources. In addition, a strong emphasis on the user (in terms of end-user applications and user-predictive models) was noticeable throughout the ICMR 2018 programme. Indeed, the user theme was central to many of the components of the conference, from the panel discussion to the keynotes, workshops and special sessions. One of the most memorable elements of ICMR 2018 was a panel discussion on the ‘Top Five Problems in Multimedia Retrieval’ http://www.icmr2018.org/program_panel.html. The panel was composed of leading figures in the multimedia retrieval space: Tat-Seng Chua (National University of Singapore); Michael Houle (National Institute of Informatics); Ramesh Jain (University of California, Irvine); Nicu Sebe (University of Trento) and Rainer Lienhart (University of Augsburg). An engaging panel discussion was facilitated by Chong-Wah Ngo (City University of Hong Kong) and Vincent Oria (New Jersey Institute of Technology). The common theme was that multimedia retrieval is a hard challenge and that there are a number of fundamental topics that we need to make progress in, including bridging the semantic and user gaps, improving approaches to multimodal content fusion, neural network learning, addressing the challenge of processing at scale and the so called “curse of dimensionality”. ICMR2018 included two excellent keynote talks <http://www.icmr2018.org/program_keynote.html>. Firstly, Kohji Mitani, the Deputy Director of Science & Technology Research Laboratories NHK (Japan Broadcasting Corporation) explained about the ongoing evolution of broadcast technology and the efforts underway to create new (connected) broadcast services that can provide viewing experiences never before imagined and user experiences more attuned to daily life. The second keynote from Shunji Yamanaka, from The University of Tokyo discussed his experience of prototyping new user technologies and highlighted the importance of prototyping as a process that bridges an ever increasing gap between advanced technological solutions and societal users. During this entertaining and inspiring talk many prototypes developed in Yamanaka’s lab were introduced and the related vision explained to an eager audience. Three workshops were accepted for ACM ICMR 2018, covering the fields of lifelogging, art and real-estate technologies. Interestingly, all three workshops focused on domain specific applications in three emerging fields for multimedia analytics, all related to users and the user experience. The “LSC2018 – Lifelog Search Challenge”< http://lsc.dcu.ie/2018/> workshop was a novel and highly entertaining workshop modelled on the successful Video Browser Showdown series of participation workshops at the annual MMM conference. LSC was a participation workshop, which means that the participants wrote a paper describing a prototype interactive retrieval system for multimodal lifelog data. It was then evaluated during a live interactive search challenge during the workshop. Six prototype systems took part in the search challenge in front of an audience that reached fifty conference attendees. This was a popular and exciting workshop and could become a regular feature at future ICMR conferences. The second workshop was the MM-Art & ACM workshop <http://www.attractiveness-computing.org/mmart_acm2018/index.html>, which was a joint workshop that merged two existing workshops, the International Workshop on Multimedia Artworks Analysis (MMArt) and the International Workshop on Attractiveness Computing in Multimedia (ACM). The aim of the joint workshop was to enlarge the scope of discussion issues and inspire more works in related fields. The papers at the workshop focused on the creation, editing and retrieval of art-related multimedia content. The third workshop was RETech 2018 <https://sites.google.com/view/multimedia-for-retech/>, the first international workshop on multimedia for real estate tech. In recent years there has been a huge uptake of multimedia processing and retrieval technologies in the domain, but there are still a lot of challenges remaining, such as quality, cost, sensitivity, diversity, and attractiveness to users of content. In addition, ICMR 2018 included three tutorials <http://www.icmr2018.org/program_tutorial.html> on topical areas for the multimedia retrieval communities. The first was ‘Objects, Relationships and Context in Visual Data’ by Hanwang Zhang and Qianru Sun. The second was ‘Recommendation Technologies for Multimedia Content’ by Xiangnan He, Hanwang Zhang and Tat-Seng Chua and the final tutorial was ‘Multimedia Content Understanding, my Learning from very few Examples’ by Guo-Jun Qi. All tutorials were well received and feedback was very good. Other aspects of note from ICMR2018 were a doctoral symposium that attracted five authors and a dedicated industrial session that had four industrial talks highlighting the multimedia retrieval challenges faced by industry. It was interesting from the industrial talks to hear how the analytics and retrieval technologies developed over years and presented at venues such as ICMR were actually being deployed in real-world user applications by large organisations such as NEC and Hitachi. It is always a good idea to listen to the real-world applications of the research carried out by our community. The best paper session at ICMR 2018 had four top ranked works covering multimodal, audio and text retrieval. The best paper award went to ‘Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval’, by Niluthpol Mithun, Juncheng Li, Florian Metze and Amit Roy-Chowdhury. The best Multi-Modal Paper Award winner was ‘Cross-Modal Retrieval Using Deep De-correlated Subspace Ranking Hashing’ by Kevin Joslyn, Kai Li and Kien Hua. In addition, there were awards for best poster ‘PatternNet: Visual Pattern Mining with Deep Neural Network’ by Hongzhi Li, Joseph Ellis, Lei Zhang and Shih-Fu Chang, and best demo ‘Dynamic construction and manipulation of hierarchical quartic image graphs’ by Nico Hezel and Kai Uwe Barthel. Finally, although often overlooked, there were six reviewers commended for their outstanding reviews; Liqiang Nie, John Kender, Yasushi Makihara, Pascal Mettes, Jianquan Liu, and Yusuke Matsui. As with some other ACM sponsored conferences, ACM ICMR 2018 included an award for the most active social media commentator, which is how I ended up writing this report. There were a number of active social media commentators at ICMR 2018 each of which provided a valuable commentary on the proceedings and added to the historical archive.
fig1

Of course, the social side of a conference can be as important as the science. ICMR 2018 included two main social events, a welcome reception and the conference banquet. The welcome reception took place at the Fisherman’s Market, an Asian and ethnic dining experience with a wide selection of Japanese food available. The Conference Banquet took place in the Hotel New Grand, which was built in 1927 and has a long history of attracting famous guests. The venue is famed for the quality of the food and the spectacular panoramic views of the port of Yokohama. As with the rest of the conference, the banquet food was top-class with more than one of the attendees commenting that the Japanese beef on offer was the best they had ever tasted.

ICMR 2018 was an exciting and excellently organised conference and it is important to acknowledge the efforts of the general co-chairs: Kiyoharu Aizawa (The Univ. Of Tokyo), Michael Lew (Leiden Univ.) and Shin’ichi Satoh (National Inst. Of Informatics). They were ably assisted by the TPC co-chairs, Benoit Huet (Eurecom), Qi Tian (Univ. Of Texas at San Antonio) and Keiji Yanai (The Univ. Of Electro-Comm), who coordinated the reviews from a 111 person program committee in a double-blind manner, with an average of 3.8 reviews being prepared for every paper. ICMR 2019 will take place in Ottawa, Canada in June 2019 and ICMR 2020 will take place in Dublin, Ireland in June 2020. I hope to see you all there and continuing the tradition of excellent ICMR conferences.

The Lifelog Search Challenge Workshop attracted six teams for a real-time public interactive search competition.

The Lifelog Search Challenge Workshop attracted six teams for a real-time public interactive search competition.

The Lifelog Search Challenge Workshop attracted six teams for a real-time public interactive search competition.

The Lifelog Search Challenge Workshop attracted six teams for a real-time public interactive search competition.

Shunji Yamanaka about to begin his keynote talk on Prototyping

Shunji Yamanaka about to begin his keynote talk on Prototyping

Kiyoharu Aizawa and Shin'ichi Satoh, two of the ICMR 2018 General co-Chairs welcoming attendees to the ICMR 2018 Banquet at the historical Hotel New Grand.

Kiyoharu Aizawa and Shin’ichi Satoh, two of the ICMR 2018 General co-Chairs welcoming attendees to the ICMR 2018 Banquet at the historical Hotel New Grand.

ACM Multimedia 2019 and Reproducibility in Multimedia Research

By | | , ,
Leave a comment

The first months of the new calendar year, multimedia researchers traditionally are hard at work on their ACM Multimedia submissions. (This year the submission deadline is 1 April.) Questions of reproducibility, including those of data set availability and release, are at the forefront of everyone’s mind. In this edition of SIGMM Records, the editors of the “Data Sets and Benchmarks” column have teamed up with two intersecting groups, the Reproducibility Chairs and the General Chairs of ACM Multimedia 2019, to bring you a column about reproducibility in multimedia research and the connection between reproducible research and publicly available data sets. The column highlights the activities of SIGMM towards implementing ACM paper badging. ACM MMSys has pushed our community forward on reproducibility and pioneered the use of ACM badging [1]. We are proud that in 2019 the newly established Reproducibility track will introduce badging at ACM Multimedia.

Complete information on Reproducibility at ACM Multimedia is available at:  https://project.inria.fr/acmmmreproducibility/

The importance of reproducibility

Researchers intuitively understand the importance of reproducibility. Too often, however, it is explained superficially, with statements such as, “If you don’t pay attention to reproducibility, your paper will be rejected”. The essence of the matter lies deeper: reproducibility is important because of its role in making scientific progress possible.

What is this role exactly? The reason that we do research is to contribute to the totality of knowledge at the disposal of humankind. If we think of this knowledge as a building, i.e. a sort of edifice, the role of reproducibility is to provide the strength and stability that makes it possible to build continually upwards. Without reproducibility, there would simply be no way of creating new knowledge.

ACM provides a helpful characterization of reproducibility: “An experimental result is not fully established unless it can be independently reproduced” [2]. In short, a result that is obtainable only once is not actually a result.

Reproducibility and scientific rigor are often mentioned in the same breath. Rigorous research provides systematic and sufficient evidence for its contributions. For example, in an experimental paper, the experiments must be properly designed and the conclusions of the paper must be directly supported by the experimental findings. Rigor involves careful analysis, interpretation, and reporting of the research results. Attention to reproducibility can be considered a part of rigor.

When we commit ourselves to reproducible research, we also commit ourselves to making sure that the research community has what it needs to reproduce our work. This means releasing the data that we use, and also releasing implementations of our algorithms. Devoting time and effort to reproducible research is an important way in which we support Open Science, the movement to make research resources and research results openly accessible to society.

Repeatability vs. Replicability vs. Reproducibility

We frequently use the word “reproducibility” in an informal way that includes three individual concepts, which actually have distinct formal uses: “repeatability”, “replicability” and “reproducibility”. Again, we can turn to ACM for definitions [2]. All three concepts express the idea that research results must be invariant with respect to changes in the conditions under which they were obtained.

Specifically, “repeatability” means that the same research team can achieve the same result using the same setup and resources. “Replicability” means that that team can pass the setup and resources to a different research team, and that that team can also achieve the same result. “Reproducibility” (here, used in the formal sense) means that a different team can achieve the same result using a different setup and different resources. Note the connection to scientific rigor: obtaining the same result multiple times via a process that lacks rigor is meaningless.

When we write a research paper paying attention to reproducibility, it means that we are confident we would obtain the same results again within our own research team, that the paper includes a detailed description of how we achieved the result (and is accompanied by code or other resources), and that we are convinced that other researchers would reach the same conclusions using a comparable, but not identical, set up and resources.

Reproducibility at ACM Multimedia 2019

ACM Multimedia 2019 promotes reproducibility in two ways: First, as usual, reproducibility is one of the review criteria considered by the reviewers (https://www.acmmm.org/2019/reviewer-guidelines/). It is critical that authors describe their approach clearly and completely, and do not omit any details of their implementation or evaluation. Authors should release their data and also provide experimental results on publicly available data. Finally, increasingly, we are seeing authors who include a link to their code or other resources associated with the paper. Releasing resources should be considered a best practice.

The second way that ACM Multimedia 2019 promotes reproducibility is the new Reproducibility Track. Full information is available on the ACM Multimedia Reproducibility website [3]. The purpose of the track is to ensure that authors receive recognition for the effort they have dedicated to making their research reproducible, and also to assign ACM badges to their papers. Next, we summarize the concept of ACM badges, then we will return to discuss the Reproducibility Track in more detail.

ACM Paper badging

Here, we provide a short summary of the information on badging available on the ACM website at [2]. ACM introduced a system of badges in order to help push forward the processes by which papers are reviewed. The goal is to move the attention given to reproducibility to a new level, beyond the level achieved during traditional reviews. Badges seek to motivate authors to use practices leading to better replicability, with the idea that replicability will in turn lead to reproducibility.

In order to understand the badge system, it is helpful to know that ACM badges are divided into two categories. “Artifacts Evaluated” and “Results Evaluated”. ACM defines artifacts as digital objects that are created for the purpose of, or as a result of, carrying out research. Artifacts include implementation code as well as scripts used to run experiments, analyze results, or generate plots. Critically, they also include the data sets that were used in the experiment. The different “Artifacts Evaluated” badges reflect the level of care that authors put into making the artifacts available including how far do they go beyond the minimal functionality necessary and how well are the artifacts are documented.  

There are two “Results Evaluated” badges. The “Results Replicated” badge, which results from a replicability review, and a “Results Reproduced” badge, which results from a full reproducibility review, in which the referees have succeeded in reproducing the results of the paper with only the descriptions of the authors, and without any of the authors’ artifacts. ACM Multimedia adopts the ACM idea that replicability leads to full reproducibility, and for this reason choses to focus in its first year on the “Results replicated” badge. Next we turn to a discussion of the ACM Multimedia 2019 Reproducibility Track and how it implements the “Results Replicated” badge.

Badging ACM MM 2019

Authors of main-conference papers appearing at ACM Multimedia 2018 or 2017 are eligible to make a submission to the Reproducibility Track of ACM Multimedia 2019. The submission has two components: An archive containing the resources needed to replicate the paper, and a short companion paper that contains a description of the experiments that were carried out in the original paper and implemented in the archive. The submissions undergo a formal reproducibility review, and submissions that pass receive a “Results Replicated” badge, which  is added to the original paper in the ACM Digital Library. The companion paper appears in the proceedings of ACM Multimedia 2019 (also with a badge) and is presented at the conference as a poster.

ACM defines the badges, but the choice of which badges to award, and how to implement the review process that leads to the badge, is left to the individual conferences. The consequence is that the design and implementation of the ACM Multimedia Reproducibility Track requires a number of important decisions as well as careful implementation.

A key consideration when designing the ACM Multimedia Reproducibility Track was the work of the reproducibility reviewers. These reviewers carry out tasks that go beyond those of main-conference reviewers, since they must use the authors’ artifacts to replicate their results. The track is designed such that the reproducibility reviewers are deeply involved in the process. Because the companion paper is submitted a year after the original paper, reproducibility reviewers have plenty of time to dive into the code and work together with the authors. During this intensive process, the reviewers extend the originally submitted companion paper with a description of the review process and become authors on the final version of the companion paper.

The ACM Multimedia Reproducibility Track is expected to run similarly in years beyond 2019. The experience gained in 2019 will allow future years to tweak the process in small ways if it proves necessary, and also to expand to other ACM badges.

The visibility of badged papers is important for ACM Multimedia. Visibility incentivizes the authors who submit work to the conference to apply best practices in reproducibility. Practically, the visibility of badges also allows researchers to quickly identify work that they can build on. If a paper presenting new research results has a badge, researchers can immediately understand that this paper would be straightforward to use as a baseline, or that they can build confidently on the paper results without encountering ambiguities, technical issues, or other time-consuming frustrations.

The link between reproducibility and multimedia data sets

The link between Reproducibility and Multimedia Data Sets has been pointed out before, for example, in the theme chosen by the ACM Multimedia 2016 MMCommons workshop, “Datasets, Evaluation, and Reproducibility” [4]. One of the goals of this workshop was to discuss how data challenges and benchmarking tasks can catalyze the reproducibility of algorithms and methods.

Researchers who dedicate time and effort to creating and publishing data sets are making a valuable contribution to research. In order to compare the effectiveness of two algorithms, all other aspects of the evaluation must be controlled, including the data set that is used. Making data sets publicly available supports the systematic comparison of algorithms that is necessary to demonstrate that new algorithms are capable of outperforming the state of the art.

Considering the definitions of “replicability” and “reproducibility” introduced above, additional observations can be made about the importance of multimedia data sets. Creating and publishing data sets supports replicability. In order to replicate a research result, the same resources as used in the original experiments, including the data set, must be available to research teams beyond the one who originally carried out the research.

Creating and publishing data sets also supports reproducibility (in the formal sense of the word defined above). In order to reproduce research results, however, it is necessary that there is more than one data set available that is suitable for carrying out evaluation of a particular approach or algorithm. Critically, the definition of reproducibility involves using different resources than were used in the original work. As the multimedia community continues to move from replication to reproduction, it is essential that a large number of data sets are created and published, in order to ensure that multiple data sets are available to assess the reproducibility of research results.

Acknowledgements

Thank you to people whose hard work is making reproducibility at ACM Multimedia happen: This includes the 2019 TPC Chairs, main-conference ACs and reviewers, as well as the Reproducibility reviewers. If you would like to volunteer to be a reproducibility committee member in this or future years, please contact the Reproducibility Chairs at MM19-Repro@sigmm.org

[1] Simon, Gwendal. Reproducibility in ACM MMSys Conference. Blogpost, 9 May 2017 http://peerdal.blogspot.com/2017/05/reproducibility-in-acm-mmsys-conference.html Accessed 9 March 2019.

[2] ACM, Artifact Review and Badging, Reviewed April 2018,  https://www.acm.org/publications/policies/artifact-review-badging Accessed 9 March 2019.

[3] ACM MM Reproducibility: Information on Reproducibility at ACM Multimedia https://project.inria.fr/acmmmreproducibility/ Accessed 9 March 2019.

[4] Bart Thomee, Damian Borth, and Julia Bernd. 2016. Multimedia COMMONS Workshop 2016 (MMCommons’16): Datasets, Evaluation, and Reproducibility. In Proceedings of the 24th ACM international conference on Multimedia (MM ’16). ACM, New York, NY, USA, 1485-1486.