More good stuff from MIT: http://www.youos.com
http://www.slate.com/id/2144896/

http://www.gpbullhound.com/research/documents/SectorReportP2PJuly2006.pdf

This report looks at the data explosion and provides a little data and some estimates. Scary stuff.

Nic – good point on the 50% vs 2%. But do you mean that in the same sense as eg “The top 2% of the sites contribute 50% of the volume”? If so then I think we all mean the same thing, though I would personally hazard a guess that for downloadable movies it might be 1% contributing more than 75% of volume. Don’t know the numbers though.

I think you are also right that ISPs will most likely be active participants in the solution to the load problem. That said it *is* possible to combine the benefits of encryption with caching. In several different ways, in fact.

The other problem with caching this stuff is recognising it. You can see a future where P2P traffic is encrypted so ISPs can’t recongnise it and throttle it back. If it’s encrypted it can’t be cached.

If you haven’t yet seen it you may enjoy:
http://www.amazon.com/gp/product/0195056442/002-0429603-4396814?v=glance&n=283155

Dream the Cartesian dream if you like ;-)

Possibly the courtroom of public opinion demands more though?

Talking of peers… Of course, there is a sense in which an extra peer on Bittorrent is just another (active) cache, and any active cache is a peer in a multimaster (asynchronously) replicated environment.

If you want to restrict it to time-shiftable stuff then that’s a good assumption IMHO.

So the question is: what is the impact of adding new peers to a content delivery network? Then, new peers decrease load on the old peers (the servers), but only decrease network traffic if getting the data from the new peer costs less than getting it from the old peer. In other words: if the new peer is near in the network topology, than the old peer. I.e. peer = cache.

For guys like Warner, reducing server load makes sense of course:
http://gigaom.com/2006/05/08/bittorrent-snags-warner-brothers/

For anyone shovelling traffic around, adding a new peer may not help unless the traffic routes elsewhere. Moreover in this case time-shifting does not help except in (possibly) suggesting some traffic shaping heuristics.

Net net: the benefits of caching/peering etc do not look like providing a solution, though clearly they are a necessary step towards one. There is a report on this which has some useful data in it. I am going to ask the owner if they can put it on the web and provide a link.

Finally, re your 2% short fat head number. Fair assumption but the distribution is surely *so* aggressively Zipf (eg http://www.useit.com/alertbox/zipf.html) that the short fat head constitutes most of the volume? Consider eg the release of a new ‘mega’ movie. How much traffic do we expect in week 1?

What I’m saying is that the long tail may contributing relatively little to the ‘caching’ problem.

Lots of assumptions. All I’m trying to do is take a no-taboos view, no anchors or frames.

This works because the economics/maths of the long tail is (surely) that the integral of the tail in the ‘attention curve’ is less than that in the head, while, for the profit curve, it can be the other way around.

Donald Knuth took the idea further, with the quater-imaginary system http://en.wikipedia.org/wiki/Quater-imaginary_base with base 2i. Every complex number can be represented with just the digits 0, 1, 2, 3 without a sign.

For even more esoteric discussions to take to the pub, what is God’s own numbering system? It must be able to hold from the Planck length (1.616 x 10^-35 m) to the diameter of the universe (perhaps 1.5 x 10^27 m), a ratio of 2.4 x 10^62 times. That requires 208 bits of precision or 131 trits or 104 quadrits or … And when wrap-around occurs we have the cyclic universe theory :-)

Users are already disgruntled with the concept of ‘install’, rebelling against the idea of ‘implement’.

With the user base suffering from a combination of Customer Attention Deficit Disorder and the need for instant gratification, how does the Enterprise Software market (big code) respond?