A couple of weeks ago I interviewed Michael Stapelberg, head developer of the DustFS project. You can listen to a recording of this interview here. DustFS is an encrypted, distributed file system based on the BitTorrent protocol. DustFS’s major features are full encryption, authentication using certificates and usage of BitTorrent for fast distribution of files (accelerated by using so-called cache servers).

Michael is quite active in the Open Source community having contributed to a number of projects such as Enlightenment (e17), collectd and Bacula. Apart from the above Michael is also the author of mxallowd an anti-spam daemon for Linux/*BSD and is about to enter a Computer Science program at the University of Heidelberg in southern Germany.

According to Michael, the idea for DustFS came from a local Chaos Computer Club (CCC) meeting. In Germany, most of the DSL providers offer high downstream but low upstream bandwidth Internet connections. If you are a DSL customer in Germany and you want to be able to conveniently share large files with your friends, your options are essentially limited to the existing, well-known protocols such as FTP, NFS, Samba and AFS. The problem however is that all of these protocols require a central server – this factor combined with the limited upstream bandwidth in Germany conspire to make this sub-optimal for distributing large files amongst a group of friends. A distributed, P2P protocol has the advantage that one can download simultaneously from multiple peers, therefore effectively combining their upstream bandwidth, rather than being limited by the pipe of the central server – resulting in greatly improved download speed. Another issue to note is that most DSL connections in Germany do not have a static IPv4 address, therefore one must additionally deal with the problem of co-ordinating server addresses with your friends. P2P networks, on the other hand, are inherently distributed and self-configuring.

One solution to the above problem would be to build a torrent file for the data you want to share, set up a BitTorrent tracker and attempt to keep it private – however this is a lot of manual set up overhead and would be quite annoying to do every time you wanted to share a single file. The main aim of DustFS is to make sharing a file among a small group of friends in a secure and trustworthy fashion easy and painless.

As I mentioned already, DustFS is an encrypted FUSE file system built on top of the P2P Research Institute’s BitTorrent implementation Unworkable. Originally DustFS used Libtransmission as its BiTorrent implementation. However despite it’s name, Libtransmission is not in fact just a library.  Libtransmission provides a number of features not required by DustFS and at 17,000 lines of source code is quite large. Unworkable, on the other hand, is closer to 4,000 lines of code. To illustrate the difference this makes, the DustFS binary built with Libtransmission is 1.7 megabytes in size, as opposed to 700 kilobytes when built with Unworkable.

In Germany it is also quite cheap to rent servers which have reasonable upstream bandwidth allowances, a fact that DustFS plans to leverage to accelerate transfers. These rented servers could be used as intermediate cache servers for the DustFS network. The idea is that when you transfer a file over DustFS, a cache server kicks in and mirrors this file. If yet another user requests the same file, they download using the copy stored on the cache server – at a vastly greater rate then from a DSL peer. A cache server is stripped down DustFS node.  A DustFS cache node doesn’t need FUSE, and generally has a lower set of dependencies than a normal node.

While DustFS is still in early stages of development, it looks extremely promising. We think that Michael has struck on a very good idea. If his project can gain enough momentum to solve the non-trivial engineering problems, DustFS could revolutionise file-sharing among groups of friends. Imagine a world where everyone has an embedded, DustFS-capable device in their homes. Family videos, photos and so on would be easily shareable between trusted parties in a secure manner. Musicians could collaborate on tracks effortlessly without worry of their intellectual property being stolen. The potential is huge. We wish Michael and DustFS the best of luck and look forward to collaborating with him in the future.

Direct download link to the tarball

Download MP3 audio of interview, part one.

Download MP3 audio of interview, part two.

Last week I had the chance to interview Jeremy Blackburn, a graduate student at the University of South Florida. Jeremy’s research interests are in the energy efficiency of computer networks. I spoke to Jeremy concerning his work to improve the power management of the BitTorrent P2P protocol. The  interview is available in two parts (part one and part two).

Jeremy’s main argument is that data centers are very expensive.  In the US alone, they consume about 1.2% of all electricity, costing approximately $3bn a year.  Worldwide, data centers consume roughly $7bn worth of electricity. Jeremy argues that this high cost is a strong incentive to offload some of the hosting traditionally done in the data center, onto the consumer, through P2P technologies.  In other words, the person downloading a file from you can contribute some of their own computing and bandwidth resources to the hosting of that file. Thus reducing data center expenditure.  However, he goes on to say that while this does reduce the energy cost for the content providers, it will probably increase overall energy consumption – through losses in efficiency – and will certainly increase the energy consumption for the users themselves.

A problem with the above premise is that it assumes that the world’s data centers are mainly used for file sharing or other services which are easily distributed across a P2P network. This is clearly not the case. I would argue that most of the computing resources in data centers are dedicated to database-related work, as opposed to activities easily distributed through P2P technologies.  One is unlikely to see eCommerce transaction processing being carried out by consumer computers participating in a P2P network any time soon.  Surely this kind of activity makes up a large chunk of that 1.2% figure.  Furthermore, when I asked Jeremy for evidence of this being a major industry trend, the only research study he could cite was about a P2P set-top box released by Telefonica, a Spanish ISP.

Having said all this, no one questions the fact that BitTorrent comprises an estimated 18-35% of traffic on the Internet (according to CacheLabs and CacheLogic) or the fact that large businesses such as Blizzard Entertainment are using it to reduce their software distribution costs.  Also consider that the Kyoto protocol only requires that on average countries reduce their emissions 5.2% below their 1990 baseline by 2010 – even a reduction of a fraction of a percent in data center energy consumption would be significant toward this goal. Indeed, how many people leave their computers on all day to download files? With all of this in mind surely a more energy efficient BitTorrent protocol is absolutely worth pursuing.

The main focus of Jeremy’s investigation is to see what can be done to the BitTorrent protocol to reduce energy consumption. Jeremy has been studying the protocol by running simulations using an implementation of a BitTorrent client within an open source network simulator called NS-2. These simulations consist of a 51 peer swarm with varying arrival times and one seed awake at all times for file availability. The files used in these simulations are between 10 to 100 mb (Jeremy intends to use files larger than a gigabyte in later simulations).

Jeremy introduces the concept of putting a node to “sleep”.  This would involve having the computer enter a low-power mode, akin to putting one’s laptop to sleep.  According to Jeremy’s simulations, theoretically by putting “seeds” (seeds are peers with all data in the torrent) to sleep 50% of the time you can halve the energy cost but only increase download times by about 10%. This saving comes from the fact that every peer in a swarm keeps a list of all other peers it is connected to in a peer table. Normally what would happen is that if a peer went to sleep then the TCP connection between it and all of the other peers it was connected to would be dropped and those connected peers would forget that the sleeping peer ever existed. The only way this peer’s resources could be made available again is if it decided to reconnect to the swarm. In the modified BitTorrrent client used within Jeremy’s simulation, when a seed is not uploading, it goes to sleep and all of the peers it is connected to do not forget about it. When those peers need that seeds resources again they can wake it up.

However, the idea of putting one’s personal computer to sleep based on network activity is not likely to be practical nor of interest to typical computer users.  Although wake-on-LAN technology exists and indeed is bundled with most modern computers, a more likely application of this sleeping technique would be in set-top boxes.  A set-top box is not a general-purpose computer.  It is inherently a single-purpose device, dedicated to an individual task such as downloading video.  Such set-top boxes generally consume a lot less power than general purpose computers, and furthermore the user is less concerned should the device suspend itself.

At some point in the near future Jeremy intends to apply his research to a real-world BitTorrent client. At the P2P Research Institute we have developed our own highly optimized BitTorrent client, Unworkable. Jeremy mentioned our client as a prime candidate for testing out these changes. Unworkable is written in C and uses a high-performance, single-threaded, asynchronous networking communication model. Unworkable has been measured downloading torrents on a ten-year-old 270mhz UltraSparc II machine, using less CPU time than the top(1) process used to monitor it.

All of the above of course begs the question, what other protocols could be modified for greater energy efficiency?  We look forward to seeing what Jeremy comes up with next in his quest to reduce the energy consumption of computer networks.  To read more about Jeremy’s research in this area, download his poster [Adobe PDF] [Microsoft Power Point].

After more than six months of working on our data-mining and statistical analysis software, I’ve finally had some free time to work on our high-performance cross-platform BitTorrent implementation, Unworkable. My interest in Unworkable was greatly renewed by the fact that Michael Stapelberg is using it as a basis for his distributed user-land file system (via FUSE) called DustFS. Michael sent me a bunch of useful patches and fixes for this release. Watch this space for an article and interview with Michael from us, hopefully in a week or two.

Whats new

In any case, Unworkable 0.5 is a great improvement. There were a number of critical bug fixes, eliminating potential crashes and properly handling certain bitwise operations. When taken together, these fixes make Unworkable a much more effective participant in BitTorrent swarms. While the main focus of this release was to improve stability and effectiveness, I also added two nice new features. Firstly, I implemented initial support for the BitTorrent Fast Extension, a.k.a. BEP 6. This extension eliminates some potential race conditions inherent in the choking algorithm, and also has some bandwidth saving messages – for example the peer can easily just say “I have all the data” and “I have none of the data” in a single short message, instead of sending its entire bitfield representation of which pieces it does and does not have. For large torrents, this bitfield can get quite long. There is some quite in-depth discussion about the Fast Extension / BEP 6 from BitTorrent, Inc developers David Harrison and Bram Cohen on the BitTorrent.org forum if you would like more info.

I also implemented – at long last – a fast resume mechanism. For large torrents, and especially on slower computers, the initial hash check for a torrent can take a long time. With the fast resume support in Unworkable 0.5, how much data you have is recorded in a small on-disk file. If this file is present, the hash check can be skipped. In practice, this means you can quit torrents and resume them almost instantaneously.

As usual for this release, I have made some portability improvements. Somehow I neglected in the past to build Unworkable with 64-bit file offsets under Linux, which was a major oversight on my part. Also as usual, I have been developing Unworkable on both fast (2Ghz amd64) and slow (270Mhz UltraSparc II) machines. My 270 Mhz Sun Microsystems Ultra 5 with 192M ram can happily download large Linux distro DVD images at speeds saturating my DSL connection!

Whats next

Now that Unworkable is pretty stable, I would like to work more on adding features such as native rate-limiting, multi-torrent support, and improvements to the control protocol. Once these features are complete, work on the GUI can be resumed and we should have quite a compelling BitTorrent client!

Download

Download the sources for Unworkable 0.5. Builds and runs on OpenBSD, FreeBSD 6.2, Ubuntu Linux 8.04, Centos 5, Fedora 7 (on EC 2), Gentoo Linux, Arch Linux, Mac OS X, Solaris 10 and Windows XP (cygwin).

The Holy Grail

Streaming video is one of the largest consumers of bandwidth today. Many estimates put it ahead of P2P in terms of gross bandwidth consumption. Sites like YouTube and Google Video attract vast numbers of viewers. Various online video streaming services are starting up, offered by companies like Amazon and NetFlix. It seems that streaming video, and the sales and advertising opportunities which come along with it, represent an irresistible revenue source for large companies.

However, streaming video also has large bandwidth costs associated with it. According to the Wikipedia page on streaming video, to stream a standard video to 1,000 viewers would require 300 Mbit/sec of bandwidth using a traditional unicast approach. All this bandwidth is expensive. How can streaming providers spread this cost out? With P2P. Make your consumers pitch in to host the video. As the number of viewers increases, so should the amount of peers which can serve up the data, allowing you to scale up the number of participants without proportionate increases in your own bandwidth.

Why not BitTorrent

As we have written about before on this blog, BitTorrent is not good for streaming video due to its rarest-first download ordering policy. In order to stream video, or music, or whatever – you want it to arrive in a predictable order. Typically that order is linear, starting at the start. This way data arrives in the order of consumption. But BitTorrent does not provide this. In fact, it almost explicitly guarantees that it will not order data in a linear fashion. BitTorrent trades predictable ordering for replication increases. Under BitTorrent, the rarest pieces of data will be replicated the most, and so become less rare.

So what, then?

Companies are instead developing their own protocols. The EU has given 19 million euro to one P2P group which is modifying the BitTorrent protocol to support streaming – presumably by doing away with the rarest-first policy.

China has a number of well-funded start ups developing their own P2P video streaming technologies. Blin.cn claims to be 50x faster than BitTorrent for video streaming. Google is an investor in Chinese streaming company Xunlei.

And of course BitTorrent, Inc have been working to develop their own video streaming version of their protocol.

Who will come out on top remains to be seen.

Four employees from BitTorrent, Inc were present. It was wonderful to have the chance to talk to some of the pioneers in this technology. They seemed very interested in the work we are doing and invited us to call in to their offices in downtown SF some time.

Overall, it was wonderful to have so much interest in our research and to be able to spread some information about BitTorrent and p2p to a wider audience. I believe the talk was recorded on video and should be made available by Google on YouTube or Google Video, however I don’t have the URL just yet. I did the slides in S5 though, and you can view the slides for the p2p research talk at Google here in a regular web browser.

Another important thing to understand is that BitTorrent peers use a “tit-for-tat” transfer scheduling algorithm. That is, peers will be rewarded by other peers for uploading data to them. It is therefore important that a bootstrapping peer have at least a small number of complete pieces as soon as possible, in order to share them, and hence be rewarded by the “tit-for-tat” algorithm. The bootstrapping peer wants to get at least a few pieces to share as soon as possible.

Additionally, pieces are usually downloaded in rarest-first order. This ensures that rare pieces are replicated sufficiently so that holes do not appear in the torrent as peers leave the swarm. One of the design considerations of BitTorrent – vs other distributed data mechanisms – is to have as reliable replication as possible. It is highly undesirable in bit torrent for even a single piece to disappear, as this could render the final file(s) completely unusable. This contrasts heavily with a file distribution system such as Amazon’s Dynamo, where it may be quite acceptable for data to disappear – is it really a huge deal if a user loses the contents of their shopping cart in an outage?

In any case, the desire to ensure replication of the rarest pieces in a torrent is overridden by the need to get a small number of pieces as a peer is bootstrapping. To this end, the initial pieces are downloaded in random order as opposed to rarest-first order. In practice this means the bootstrapping peer should get its initial data quite a bit faster. The number of pieces to download before switching to rarest-first order is suggested to be four.

As a general point, the peer will aim to complete a piece fully before requesting blocks belonging to another piece. Alternately put, if the peer has a few blocks in one piece, it will concentrate on downloading the remaining blocks belonging to that piece before starting on another piece.

The End Game is the name for the final download strategy – there is a tendency for the last few pieces of a torrent to download quite slowly. To avoid this, many BitTorrent implementations issue requests for the same remaining blocks to all its peers. When a block comes in from one peer, you send CANCEL messages to all the other peers requested from, in order to save bandwidth. Its cheaper to send a CANCEL message than to receive the full block and just discard it.

However, there is no formal definition of when to enter End Game Mode. I found two popular definitions:

1. All blocks have been requested
2. Number of blocks in transit is greater than number of blocks left, and no more than 20

The official BitTorrent DHT specification states that the protocol is based on Kademilia. In BitTorrent, DHT is mostly separated from the original protocol. Peers listen on an additional port, using a UDP protocol, to issue network searches and so forth. The DHT protocol is known as KRPC and consists of three message types – query (”q”), response (”r”) and error (”e”). There are four queries:

• PING
• FIND_NODE
• GET_PEERS
• ANNOUNCE_PEER

Each KRPC message is formatted in B-Encode, with various key-value pairs encoded in a dictionary structure. Each node participating in the DHT has its own routing table containing contact information for nodes “near” to it (according to a mathematical ‘distance function’). This routing table is used to produce responses to queries. We will go more into details of how this works in the next article.

The TCP BitTorrent peer protocol itself has only been very slightly extended in order to take advantage of this new DHT functionality. During the handshake, peers may indicate that they support DHT. If the receiver also supports DHT, it should send a new message called PORT (id is 0×09) with the port number of its UDP DHT service, encoded as a 16-bit value. The receiver of this message will then try to send a PING message to the peer on this port, and if it gets a reply, routing table adjustments etc should take place.

Those are the very basics of BitTorrent DHT from a network perspective.