Open Source Column: GPAC

GPAC, Toolbox for Interactive Multimedia Packaging, Delivery and Playback

Introduction

GPAC was born 10 years ago from the need of a lighter and more robust implementation of the MPEG-4 Systems standard [1], compared to the official reference software. It has since then evolved into a much wider project, covering many tools required when exploring new research topics in multimedia, while keeping a strong focus on international standard coming from organization such as W3C, ISO, ETSI or IETF. The goal of the project is to provide the tools needed to setup test beds and experiments for interactive multimedia applications, in any of the various environments used to deliver content in modern systems: broadcast, multicast, unicast unreliable streaming, HTTP-based streaming and file-based delivery. The project is used in many contexts, from commercial video services production chains to academic research projects, teaching and lab sessions; it allows fast designing of multimedia prototypes, from content production (creation and delivery) to interactive playback (such as augmented reality, video streaming, 2D or 3D scalable drawing) – more details are available in our paper [6].

Architecture and tools

Most of GPAC functionalities are gathered in a core library called libgpac, allowing building a wide range of applications on top of GPAC without depending on the media player or on the media packagers. Figure 1 shows various GPAC tools and their relationships with other OSS multimedia softwares. All multimedia capabilities needed by the player (access to video display, cameras, sensors such as accelerometers) are abstracted into platform-specific modules.

The core library is written in extremely portable C code, and all modules required by the player have been ported to a wide variety of platforms, from desktop OS (Windows, Linux, OSX) to mobile devices (iOS, Android, Windows CE) or embedded systems (various SH4 and ARM boards, set-top boxes …), as shown in Figure 6. Under constant construction, GPAC is build and bug-checked on a daily basis, thanks to the Buildbot project.

Packaging of Interactive Multimedia Content

The first set of tools present in GPAC deal with multimedia content packaging. These tools are unique in the open-source world, both in terms of supported features and of compliance to industry standards. The most known and used tool is called MP4Box, and has a variety of applicative usages (as illustrated in Figure 2): it can be used to package raw media data such as video or audio streams or subtitles into files conforming to the ISO base media file specification [2], as used by most smartphone and tablet devices. MP4Box can work on ISOBMF files on a variety of ways, adding or removing tracks, extracting media data, analyzing files, splitting in smaller files, concatenating files, adding meta data and tags, embedding extra files (such as HTML/CSS/JavaScript) into the file to simplify the distribution process. MP4Box also features tools for encoding and decoding MPEG scene description streams (MPEG-4 BIFS and LASeR), encryption and decryption of media streams, and is capable of preparing the content for various kind of delivery:

• Altering the data layout to enable progressive download and play of the file,
• Streaming a file over RTP [3] unicast or multicast,
• Creating hint tracks, thereby allowing streaming servers such as the open-source Darwin Streaming Serverto stream the file over RTP without knowledge of the media format,
• Creating and optionally streaming an MPEG-2 Transport Stream [4] from a set of ISOBMF files.

Figure 3 shows one usage of GPAC tools to generate an MPEG-2 Transport Stream containing interactive BIFS scenes generated live, compatible with the T-DMB standard; the resulting multiplex is streamed over RTP or using HTTP Live Streaming format, and is compatible with traditional MPEG-2 TS player without T-DMB support such as VLC. Recent additions to MP4Box concern the packaging of media content for Adaptive HTTP Streaming, such as HLS, Smooth Streaming or HDS, with a special focus on the new MPEG-DASH standard: different bitrate/resolution versions of the content can be processed together by MP4Box to generate a set of “segmented” files used by these streaming protocols, as explained in [5]. The generated files have been successfully tested with existing MPEG-DASH and other HTTP adaptive streaming players, as well as with the experimental implementation of Media Source Extension (Chrome browser). The segment generation process can be automated in MP4Box to simulate an adaptive streaming live session.

Playback and Rendering of Interactive Multimedia Content

The second set of tools in GPAC deal with content playback and interactive application rendering. The software includes a player, referred to as Osmo4 or MP4Client (command-line version), whose main design is to present media data together with rasterizing vector-based media descriptions, in a synchronized and interactive way. The player may use hardware acceleration such as OpenGL whenever possible (for video composition, 2D or 3D image synthesis).

The player supports many audio and video formats, mainly relying on third party open source software such as ffmpeg, OpenSVC, XVID. It also supports native processing and rendering of scene description formats such as X3D/VRML, SVG and BIFS, thereby enabling a wide range of applications: cartoons, interactive video portals, immersive 3D worlds, augmented reality, “follow-me” applications, etc … It can also perform multi-stereoscopic rendering for many 3D displays types, from old-style anaglyphs to side-by-side or interlaced multiview (as illustrated in Figure 4, and can easily be configured for new displays using OpenGL shaders.

The player can be extended easily through flexible plugin architecture. These plugins can either implement new video decoder, new protocol parsers, new JavaScript support, and new rendering methods. For instance, the UPnP plugin allow fast prototyping of networked multimedia applications through UPnP / DLNA (GPAC can act as a media server, media renderer and media controller). This plugin has been used to demonstrate application discovery and cross-browser communications using UPnP based on the MPEG-U standard, enabling secondary screen applications by relocating part of the content, whether synthetic (game, UI) or natural (video or audio streams).

The player has also been used to showcase adaptive video playback through scalable video decoding, highly immersive worlds on 3D displays, augmented reality using embedded camera, streaming media portals (RTP/RTSP, web radios, progressive download, MPEG-2 TS over IP or DVB4Linux, HTTP Adaptive Streaming). Client-side scripting is possible in ECMAScript through Mozilla SpiderMonkey engine, and some usual bindings are provided (DOM subset, XMLHttpRequest support). Plugins can also expose their functionalities through scripting (for example UPnP/DLNA stack can be completely controlled from script).

Future work

Our future works will focus on more flexibility in the various tools present in GPAC to simplify creation of streaming media applications by third-party developers, as well as new tools to help the research community in areas such as hybrid networking, ultra-high definition displays and immersive environments.

Acknowledgements

Being an open-source project, GPAC would not be GPAC without its user community, which we would like to thank for their support, suggestions, bug spotting and fixing, and positive feedback during these years! Especially, many thanks to all software developers who joined our team at Telecom ParisTech and all our students.

References

 

1. ISO/IEC 14496-1: MPEG-4 Systems
2. ISO/IEC 14496-12: ISO Base Media File Format
3. RTP: A transport protocol for real-time applications
4. ISO/IEC 13818-1: MPEG-2 Systems
5. Stockhammer, T. 2011, “Dynamic Adaptive Streaming over HTTP: Design Principles and Standards”, In Proceedings of the second annual ACM conference on Multimedia systems (MMSys11), ACM, New York, NY, USA
6. J. Le Feuvre, C. Concolato, J.C. Moissinac. “GPAC: open source multimedia framework”. In Proceedings of the 15th international Conference on Multimedia, Augsburg, Germany, September 25 – 29, 2007. MULTIMEDIA ’07. ACM, New York, NY, 1009-1012. DOI= 10.1145/1291233.1291452