Category Archives: Feature

Six years ago, we created a new open source P2P file sharing program called Tribler. During this time over one million users have used it, and three generations of Ph.D. students tested their algorithms in the real world. Tribler is built around BitTorrent. Introduced in 2001, BitTorrent revolutionized the P2P world because of its unprecedented efficiency. However, some problems are not properly addressed in BitTorrent. First, it does not specify how to search the network, relying instead on central websites. These websites allow users to find and download small metadata files called torrents. A torrent describes the content and is required for downloading to start. Second, BitTorrent’s unique method for downloading files is incompatible for streaming. This is due to the fact that it is optimized for speedy and reliable downloading, not providing a method for quick buffering. Tribler is the first client which continuously tries to improve upon the basic BitTorrent implementation by addressing some of the flaws described above. It implements, amongst others, remote search, streaming, channels and reputation-management. All these features are implemented in a completely distributed manner, not relying on any centralized component. Still, Tribler manages to remain fully backwards compatible with BitTorrent. Work on Tribler was initiated in 2005 and has been supported by multiple European grants. In order to maximize the resource contribution of peers (other computers downloading/uploading the same file), BitTorrent splits a file into small pieces. This way, downloaders (called leechers) can upload completed pieces to other leechers, without the need to have the complete file first. Furthermore, uploading is encouraged by the tit-for-tat incentive mechanism: a leecher will rank its connected peers by their upload speed and will upload only to the fastest uploaders. Peers which have the complete file can help others by sending them pieces for free, these peers are called seeders. Before being able to download a file, a peer first has to obtain a torrent. This torrent describes the content of the file and includes the SHA-1 cryptographic hash per piece. This mechanism protects against transfer errors and malicious modifications of the file.

Tribler Design and Features

Tribler component overview

A basic overview of Tribler is shown in the figure on the right, it consists of four distinct components.

• GUI: build using wxWidgets in order to be platform independent
• BTengine: a BitTorrent engine, which has been altered to allow for our customizations
• BuddyCast: our custom BitTorrent overlay, which is slowly being phased out
• Dispersy: our new custom protocol build with NAT traversal and distributed permissions

Tribler is built around PermIds, permanent identities that allow us to identify the actions of users. PermIds are stored as a public/private keypair and are used in Tribler to sign every message. Communication between peers is established by using a ‘special’ BitTorrent swarm. All Tribler peers connect to this swarm and communicate with each other using the BuddyCast protocol. Peers are discovered by connecting to SuperPeers. Identical to ‘normal’ peers, but are considered to always be online. BuddyCast connects to a new peer every 15s and will exchange its preference list. This list contains the last downloads of a peer. By collecting them, we can calculate which peers are most similar to a given user. Those peers, taste buddies, are then used during search. While performing the BuddyCast handshake the user will exchange his current connections as well, allowing it to hillclimb towards finding his most similar peers and discovering new peers.  

Search

Tribler search results

Performing remote search in a decentralized manner has been a problem for many years. An early P2P protocol, Gnutella, used to send a message to all of its neighbors, which was then forwarded until a TTL of 7 was reached. Such an implementation is called flooding, as it causes a search-query to be sent to almost all peers in the network. Flooding a network is very quick, but it consumes huge amounts of bandwidth.

In contrast, Tribler uses a TTL of 1 (i.e., it only uses its neighbors to perform a remote search). Using the connections of the user to similar peers, we can obtain good results. Using only taste buddies, hitrates over 60% are possible. This figure is further improved by local caches deployed at every peer. The caches contain information for up to 50 000 torrents, which are used for improving search. Tribler connects to up to 10 taste buddies and to 10 random peers, thus allowing the user to search within up to 1050000 torrents. A torrent is collected when our algorithms deem it interesting for a peer. This is calculated by using the same user-item matrix that is used to find similar peers. The user-item matrix is constructed by storing the BuddyCast preferences. Collaborative filtering allows us to ‘recommend’ torrents to be collected. Collected torrents are thus tailored for every user, resulting in quicker search results, as we can display in the GUI the locally cached results before receiving any response from a peer. More details are available in our papers [1,2].

Streaming

Tribler VOD streaming

Tribler supports two distinct types of streaming: Video-On-Demand and Live-Steaming. Both of them extend BitTorrent, by replacing one aspect of it. VOD requires a different approach for downloading pieces. The default policy of BitTorrent is to download the rarest piece first, ensuring the health of all pieces in the swarm. In contrast, in VOD we want to download the first few pieces as soon as possible to commence playback as soon as possible.

To allow for this we have defined three priorities (high, mid and low). Priorities are assigned to pieces, based on the current playback position. High priority pieces are downloaded in-order, thus allowing to start playback quickly. After all high priority pieces are downloaded, we start downloading mid priority pieces. Those are downloaded rarest-first, this to ensure we maintain the overall health of the swarm. Because mid priority pieces are only a subset of all available pieces, we still ensure that the playback-buffer remains stable. After downloading all high and mid priority pieces, we start downloading the low priority pieces, rarest-first. Because the playback position is moving forward, the priority of the pieces will be continuously modified. Furthermore, we replaced the default BitTorrent incentive mechanism (tit-for-tat) with Give-to-Get. This incentive mechanism will rank peers according to their forwarding rank. A metric describing how well a peer is sending pieces to other peers. Full details are available in our paper [5].

For live streaming we had to modify the actual torrent file. Because in live streaming pieces are not know beforehand, we cannot include their SHA-1 hashes. We replaced the verification scheme by specifying the public key of the original source in the torrent file. Using the public key, every peer can then verify the validity of the pieces. Because live streaming may have an indefinite duration, we keep pieces that are at most 15 minutes old relative to our playback position.

Channels

Channel overview: listing all torrents

Channel comments: listing latest received comments for this Channel

Channel activity: listing latest activity for this Channel

Since December 2011 we are evaluating, in the wild, the performance of our new transport protocol, Dispersy. Dispersy is the successor of BuddyCast, and it is focused on NAT-traversal. Instead of using TCP, Dispersy uses UDP. Furthermore, while BuddyCast implemented one global overlay to connect all Tribler-peers using a ‘special’ BitTorrent swarm, Dispersy creates a separate overlay per protocol.

Using Dispersy we implemented Channels. These are created by users and consist of a list of torrents they like. Channels are implemented as separate Dispersy overlays and are discovered through one special overlay to which all peers connect. Channels have evolved from a simple list of torrents to a community in which users can comment on torrents, modify their name and description and organize playlists. Modifications are publicly visible, in a system that resembles Wikipedia. By allowing everyone to edit/improve the metadata of torrents, we hope to get a similar quality of experience as in Wikipedia. If a channel owner (the user creating the channel) does not want other users to interfere with his channel, he can limit which messages other users are able to send. This is enabled by using the decentralized permission system build into Dispersy, and it allows for a flexible configuration of channels. By voting on a Channel, Dispersy will start to collect its contents. Before voting on a Channel only a snapshot of its content is available. More details regarding the voting is described in our paper [6]. Currently, popular channels have well over 30 000 torrents. Furthermore, our users have currently casted over 60 000 votes.

Reputation

BarterCast graph: showing data transfers between peers

A feature lacking in BitTorrent is the cross-swarm identification of peers. While downloading a file, peers have an incentive to upload to others due to tit-for-tat. But after completing a download, no incentives are in place to motivate a peer to keep uploading a file.

In order to address this, Tribler uses its PermIds to identify Tribler peers in other swarms. Additionally, we employ a mechanism called BarterCast, which builds a history of upload and download traffic between Tribler peers. We can then build a graph consisting of the download behavior of peers, scoring them accordingly. A peer which has shown to upload more than others is rewarded by being able to download at a faster rate, while lacking peers can be prevented from downloading at all. More details are available in our papers [3,4].

Acknowledgments

Since the start of the project in 2005 many many people have contributed to the project. Amongst others are A. Bakker, J.J.D. Mol, J. Yang, L. d’Acunto, J.A. Pouwelse, J. Wang, P. Garbacki, A. Iosup, J. Doumen, J. Roozenburg, Y. Yuan, M. ten Brinke, L. Musat, F. Zindel, F. van der Werf, M. Meulpolder, J. Taal, R. Rahman, B. Schoon and N.S.M. Zeilemaker. Tribler is a project which continues to evolve with the help of its community. Currently we have an active userbase who are commenting and suggesting features in the forums and continues to innovate together with our european partners.

If you are interested by the text above and want to try out Tribler, you can download it from our website http://www.tribler.org. Furthermore, the website has even more documentation of feature Tribler has and had, feel free to look around and leave a comment in the forums.

References

1. Pouwelse JA, Garbacki P, Wang J, et al. Tribler: A social-based peer-to-peer system. Concurrency and Computation: Practice and Experience 2008. Available at: http://www3.interscience.wiley.com/journal/114219988/abstract
2. Zeilemaker N, Capota M, Bakker A. Tribler: P2P media search and sharing. Proceedings of the 19th ACM international conference on Multimedia (ACM MM) 2011. Available at: http://dl.acm.org/citation.cfm?id=2072433
3. Meulpolder M, Pouwelse J. Bartercast: A practical approach to prevent lazy freeriding in p2p networks. Sixth Int’l Workshop on Hot Topics in P2P Systems (HoT-P2P) 2009. Available at: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5160954
4. Delaviz R, Andrade N, Pouwelse JA. Improving accuracy and coverage in an internet-deployed reputation mechanism. IEEE Tenth International Conference on Peer-to-Peer Computing (IEEE P2P) 2010. Available at: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5569965
5. Mol J, Pouwelse J, Meulpolder M, Epema D, Sips H. Give-to-get: Free-riding-resilient video-on-demand in p2p systems. Fifteenth Annual Multimedia Computing and Networking (MMCN) 2008. Available at: http://www.pds.ewi.tudelft.nl/pubs/papers/mmcn2008.pdf
6. Rahman R, Hales D, Meulpolder M, Heinink V, Pouwelse JA and Sips H. Robust vote sampling in a P2P media distribution system. In: Proceedings of IPDPS 2009. Available at: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5160946
7. Tribler Protocol Specification. Available at: http://svn.tribler.org/bt2-design/proto-spec-unified/trunk/proto-spec-current.pdf

Mozilla Popcorn: Web Video Interaction Using Client-Side Javascript

Context and history

Popcorn is an HTML5 media project from Mozilla, the non-profit organization that makes the Firefox web browser. It makes media-oriented web development easy through shared development, open source libraries and tools.

Popcorn.js: a Javascript library for interactions between video and the web

Popcorn.js makes web media more connected by providing an event-driven API to hook <video> and <audio> content into the of the capibilities of the web platform (developer.mozilla.org). Prior to HTML5, web video lived exclusively inside browser plug-ins like Flash and VLC, which put it outside the reach of JavaScript, CSS, and other techniques for interacting with the rest of the surrounding HTML document. Popcorn.js turns media into fully-interactive JavaScript objects, so that media objects can both trigger and listen for events. It enables developers to cue events along a media timeline using a simple Javascript syntax:

var pop = Popcorn("#my-video");
pop.text({
start: 1.38,
end: 5.12,
text: "Hello World",
target: "my-div"
});
pop.play();

Live source available at http://jsfiddle.net/p8Kbs/80/. Additionally, media playback is accessible via popcorn.play(), popcorn.pause(), and popcorn.currentTime(seconds), which allows you to jump to any point in the timeline of the referenced media. As a nod toward the expectations of media producers and videographers, Popcorn.js also provides methods like popcorn.cue(), which simply ties actions to specific times. Aside from simple time-based triggers, you can use popcorn.listen(event,callback_function) to bind the callback function to a specified event. Built-in events are provided to handle typical HTML5 web video playback scenarios, such as “play,” “pause,” “loadstart,” “seeked,” “volumechange,” and so on. However, you can define custom events and trigger them directly by using popcorn.trigger(event[,data]), where the data parameter is an optional data object to send to listeners.

Extensibility

By design, Popcorn is extensible. Mozilla supports about 20 plugins that come packaged with the library, ranging from simple HTML element insertion, to complex data retrieval and aggregation. Examples include a subtitle plugin, a GoogleMaps plugin, a Twitter plugin, a Facebook plugin, and a JavaScript code plugin. If some desired functionality doesn’t yet exist in the library, Popcorn.js has a well-documented plugin architecture: http://popcornjs.org/popcorn-docs/addon-development/. Popcorn works best with HTML5 media, but also has wrappers for arbitary objects (through the “baseplayer”) and Flash players, like YouTube, Vimeo, Flowplayer, and Soundcloud. It’s easy to write a wrapper for any web-oriented video player. Popcorn also includes a set of parsers for reading common data files (SRT, TTML, XML, etc). Of course, as with other parts of Popcorn, it is easy to create a custom data parser. Tested thoroughly, Popcorn.js supports all modern browsers and IE8. Currently, it’s stable at version 1.2. You can download the Popcorn source or use a web-based build tool to wrap a custom, compressed version.

Potential Applications

Popcorn is in use by a range of publishers, service providers, creative coders and individuals to mash video with the rest of the web. RAMP, a content optimization company, uses an automated process to display time-coded metadata about significant people, places and things whenever they are mentioned in a video. Using Popcorn, RAMP can support a range of player types and contexts (web, mobile, headless) by developing against the common Popcorn API (http://www.ramp.com/solutions/optimized-video/metaplayer/popcorn/). The Dutch multimedia archive Beeld en Geluid has used Popcorn to create a “living archive,” connecting cultural archival material with a range of semantic metadata (http://www.openimages.eu/blog/2012/01/13/open-images-videos-enriched-with-open-data/). Popcorn has also been used to create hyperlinked transcripts that use text as an interface for traversing and editing long media assets (http://yoyodyne.cc/h/) Aside from scale applications, Popcorn is also supported by a burgeoning creative community (In fact, the project was started and is run by Brett Gaylor–a filmmaker!). For instance, documentary producer Kat Cizek uses Popcorn to create web based interactive films. In “1 Millionth Tower,” the web browser creates a navigable 3D space that simulates high rises in major cities around the world. Popcorn is used to turn the camera at key moments, spawn visual effects, and to download live weather data from web APIs. If it’s raining in Toronto, it’s also raining in virtual Toronto (http://highrise.nfb.ca/onemillionthtower/1mt_webgl.php).

Popcorn Maker

A key goal of the Popcorn project is to enable more connected web video on a mass scale, and to open creative possibilities to individual media-makers. Popcorn Maker is a user-facing web application used to create interactive media. It requires no code knowledge. Users pick a video from YouTube or the wider web, open the media object in a prepared HTML template, customize the project, and publish. Popcorn Maker can be used to create pop-up videos, multimedia reports, guided web tours and more. Project composition happens live in the browser. Users can drag and drop events onto a timeline interface, position objects on the page, and watch a live preview of the project be constructed. Popcorn Maker projects are entirely human-readable HTML, CSS and Javascript. For the time being, Popcorn Maker does not support media editing and sequencing. Users must come prepared with an edited video file. For this reason, Popcorn Maker is not a web-based video editor–rather, it’s a video-based web editor. We may revisit this decision later when web browsers handle media playback and synchronization more precisely. Like the rest of the Popcorn project, Popcorn Maker is 100% free and open source. Developers of time-based multimedia apps are encouraged to build on the Butter SDK (source code available at http://github.com/mozilla/butter) and contribute back to the project. At the time of this writing, Popcorn Maker is in active development at version 0.5, and is scheduled for a 1.0 release in late 2012. http://mozillapopcorn.org.

Credits

The Popcorn project and its constituents are lovingly crafted by Ben Moskowitz, Bobby Richter, Brett Gaylor, David Seifried, Christopher De Cairos, Matthew Schranz, Jon Buckley, Scott Downe, Mohammed Buttu, Kate Hudson, David Humphrey, Jeremy Banks, Brian Chirls, James Burke, Robert Stanica, Anna Sobiepanek, Rick Waldron, Nick Cammarata, Daniel Hodgin, Daniel Brooks, Boaz Sender, Dan Ventura, Brad Chen, Minoo Ziaei, Cesar Gomes, Steven Weerdenburg, Cole Gillespie, and Nick Doiron.