A couple of weeks ago I interviewed Michael Stapelberg, head developer of the DustFS project. You can listen to a recording of this interview here. DustFS is an encrypted, distributed file system based on the BitTorrent protocol. DustFS’s major features are full encryption, authentication using certificates and usage of BitTorrent for fast distribution of files (accelerated by using so-called cache servers).

Michael is quite active in the Open Source community having contributed to a number of projects such as Enlightenment (e17), collectd and Bacula. Apart from the above Michael is also the author of mxallowd an anti-spam daemon for Linux/*BSD and is about to enter a Computer Science program at the University of Heidelberg in southern Germany.

According to Michael, the idea for DustFS came from a local Chaos Computer Club (CCC) meeting. In Germany, most of the DSL providers offer high downstream but low upstream bandwidth Internet connections. If you are a DSL customer in Germany and you want to be able to conveniently share large files with your friends, your options are essentially limited to the existing, well-known protocols such as FTP, NFS, Samba and AFS. The problem however is that all of these protocols require a central server – this factor combined with the limited upstream bandwidth in Germany conspire to make this sub-optimal for distributing large files amongst a group of friends. A distributed, P2P protocol has the advantage that one can download simultaneously from multiple peers, therefore effectively combining their upstream bandwidth, rather than being limited by the pipe of the central server – resulting in greatly improved download speed. Another issue to note is that most DSL connections in Germany do not have a static IPv4 address, therefore one must additionally deal with the problem of co-ordinating server addresses with your friends. P2P networks, on the other hand, are inherently distributed and self-configuring.

One solution to the above problem would be to build a torrent file for the data you want to share, set up a BitTorrent tracker and attempt to keep it private – however this is a lot of manual set up overhead and would be quite annoying to do every time you wanted to share a single file. The main aim of DustFS is to make sharing a file among a small group of friends in a secure and trustworthy fashion easy and painless.

As I mentioned already, DustFS is an encrypted FUSE file system built on top of the P2P Research Institute’s BitTorrent implementation Unworkable. Originally DustFS used Libtransmission as its BiTorrent implementation. However despite it’s name, Libtransmission is not in fact just a library.  Libtransmission provides a number of features not required by DustFS and at 17,000 lines of source code is quite large. Unworkable, on the other hand, is closer to 4,000 lines of code. To illustrate the difference this makes, the DustFS binary built with Libtransmission is 1.7 megabytes in size, as opposed to 700 kilobytes when built with Unworkable.

In Germany it is also quite cheap to rent servers which have reasonable upstream bandwidth allowances, a fact that DustFS plans to leverage to accelerate transfers. These rented servers could be used as intermediate cache servers for the DustFS network. The idea is that when you transfer a file over DustFS, a cache server kicks in and mirrors this file. If yet another user requests the same file, they download using the copy stored on the cache server – at a vastly greater rate then from a DSL peer. A cache server is stripped down DustFS node.  A DustFS cache node doesn’t need FUSE, and generally has a lower set of dependencies than a normal node.

While DustFS is still in early stages of development, it looks extremely promising. We think that Michael has struck on a very good idea. If his project can gain enough momentum to solve the non-trivial engineering problems, DustFS could revolutionise file-sharing among groups of friends. Imagine a world where everyone has an embedded, DustFS-capable device in their homes. Family videos, photos and so on would be easily shareable between trusted parties in a secure manner. Musicians could collaborate on tracks effortlessly without worry of their intellectual property being stolen. The potential is huge. We wish Michael and DustFS the best of luck and look forward to collaborating with him in the future.

Four employees from BitTorrent, Inc were present. It was wonderful to have the chance to talk to some of the pioneers in this technology. They seemed very interested in the work we are doing and invited us to call in to their offices in downtown SF some time.

Overall, it was wonderful to have so much interest in our research and to be able to spread some information about BitTorrent and p2p to a wider audience. I believe the talk was recorded on video and should be made available by Google on YouTube or Google Video, however I don’t have the URL just yet. I did the slides in S5 though, and you can view the slides for the p2p research talk at Google here in a regular web browser.

I will post my slides online and I believe there will be a good-quality recording of the talk made available on Youtube or Google Video.

Why so little porn?

While it is true that certain torrent aggregators/trackers specialise in a specific category, sites included in the survey such as Mininova and TPB have a very wide variety of content types. Given the apparent prevalence of porn on the Internet as a whole, I’m surprised that the numbers are so low for it. It is possible that much of this material was mis-categorised, or lumped in with another category such as movies or pictures.

More music than TV?

Another surprise, especially considering the huge amount of talk about rampant sharing of TV shows. Also, I would posit that there may be more TV episodes released per annum than music albums. However, perhaps there is a tendency – as with DVD releases of the same shows – to group individual TV shows into one torrent per season. On the other hand, the same pattern exists with music, where an artist’s entire discography is released in a single torrent.

If I was willing to tolerate adverts and poor quality, AOL TV seemed like a reasonable choice. Amazon Unbox was another tempting “now” option, but what if I wanted some sweet MacGyver lovin’ on the bus home from work with my iPod? Unbox’s DRM does not support the iPod so I’d be SOL on that count. Besides these restrictions Unbox wanted to charge me $1.99 an episode which at roughly 24 episodes per season was quite a bit more than I was willing to pay!

How about NetFlix? Well I usually don’t watch that many movies a month so a NetFlix account would stay dormant at least half the time and its digital delivery services suffers from the same DRM problems that afflict Unbox. A quick search of all the main torrent trackers yielded nothing promising (a seedless Love Boat crossover and a French language version of MacGyver series 2) so even if I wanted to sneak into the show for free I’d have to wait an undetermined amount of time for something to appear. Taking all of the above into consideration and considering the totally reasonable price I decided to do it the old fashioned way and buy DVD.

This whole process of evaluation took more than an hour to complete and was way more trouble than your average consumer would be willing to put up with. With all of these catches it’s no wonder that piracy is so prevalent, if a good MacGyver torrent was readily available at the time I was about to click buy, maybe I would have thought twice.

BitTorrent is really good at catering to the latest trends, where it falls down is in catering towards the more obscure pieces of media. In my opinion it is this inconsistent nature which is the entertainment industry’s biggest opportunity to stem the tide in its losing battle against piracy.

Consider what I was doing in my survey of the media options available to me. I was looking for the best bang for my buck. If I went to Swedish BitTorrent site The Pirate Bay and tried searching for MacGyver and the only promising thing that came up was an advertisement for NetFlix or Amazon and the price was right I might have clicked buy. Certainly I was more likely to click on that on an advertisement for pheromones.

By not embracing BitTorrent aggregator sites the entertainment industry is missing out on a valuable opportunity to extract revenue from potential customers. Picture a service that trawled all of the major aggregators and provided an up to the minute search able index of every torrent available on the Internet. A Google for BitTorrent if you will. Now picture text advertisements on this service presenting you with the relative costs of all media available – free with advertisements, cheap DVD postal subscription, DRM streamed immediate download. In my opinion, such a service would be an excellent way to regain lost revenue from piracy.

Distribution of media in peer to peer networks tends to mirror the mainstream media. A new cinematic release will suddenly become the hottest thing to download on BitTorrent. But who releases the films, and how? Usually there are many different individual releases for a given film, and the cycle typically starts with initial low-quality offerings, with higher-quality copies being published as time goes on. This article is an in-depth look at the BitTorrent release progression of one major 2007 Hollywood film, 300.

Time line

• March 9th 2007, blockbuster film 300 opens in movie theatres across the United States. While movie-goers crowd out cinemas on release day, Internet piracy gangs spring into action, working secretly beneath the radar of copyright enforcers in shady arrangements with theatre staff.
• March 10th 2007, so-called “cam” releases – crudely videotaped reproductions – of the hit film appear on the BitTorrent P2P network.
• March 12th 2007, a mere three days after the film has opened to the public, highly sophisticated “telesync” reproductions – which practically necessitate insider help to produce – flood onto BitTorrent.
• March 20th 2007, less than two weeks since the film’s opening, BitTorrent is inundated with leaked “workprint” copies of the flick – very high quality digital reproductions.
• March 21st 2007, piracy groups begin circulating misappropriated DVD “Screeners” – advance retail-quality copies of the film intended for critics and distributors – on BitTorrent.
• April 11th, 2007, while continuous streams of pirated copies have poured onto P2P networks since the film’s release – including versions in French, Spanish and German along with versions with fan-made “alternative soundtracks” – the first authentic “DVDRip” copies appear on BitTorrent, barely more than a month after the initial release date.
• July 21st, 2007, the first in a string of ultra high quality HDTV duplications are released by pirates.
• By the end of 2007, more than 332 different pirate versions of the film have been released on BitTorrent.
• A whopping 25% of all pirated versions of 300 were distributed through infamous Swedish BitTorrent site, The Pirate Bay.

Analysis

The time line above makes it clear that despite huge investments made by the entertainment industry in piracy prevention, piracy is rampant. Indeed, it is easier now than ever before to download a pirated movie. What is unclear, however, is the impact of various offerings, such as Amazon’s online movie store, or Netflix’s, are having on piracy. At what point will people switch over to simply paying for a download rather than firing up their torrent client? What needs to change in terms of cost, software and licensing? These are some of the questions we are really interested in, and will continue to research.