Counting what counts: Musing about Wikipedia and Drosophila

I feel at ease. For once I am not confused. At least I am less confused than I was earlier.
You remember the sequence of posts I wrote about opensource and gatekeepers? [Those new to the conversation can find them here, here, here, here and here, in chronological order. Alternatively you can enter “gatekeeper” into the search box in the sidebar and get the same results. But it’s not important.]

I’ve been tracking what Aaron Swartz has been saying about Wikipedia in Raw Thought, his blog. Now why would I do this? Simple. In a strange kind of way I think that Wikipedia is the Drosophila of social software, much like a recent Scientific American article suggested that chess could be the Drosophila of cognitive science. You can measure things in Wikipedia, conduct experiments, analyse what’s happening, predict results, compare actuals against predictions. Wikipedia provides the nearest thing to a live laboratory for the study of social software. For now, anyway.

In a recent post, headlined Who Writes Wikipedia, Swartz has some interesting analysis of what happens there. I quote from the post:

  • Wales decided to run a simple study to find out: he counted who made the most edits to the site. “I expected to find something like an 80-20 rule: 80% of the work being done by 20% of the users, just because that seems to come up a lot. But it’s actually much, much tighter than that: it turns out over 50% of all the edits are done by just .7% of the users … 524 people. … And in fact the most active 2%, which is 1400 people, have done 73.4% of all the edits.” The remaining 25% of edits, he said, were from “people who [are] contributing … a minor change of a fact or a minor spelling fix … or something like that.”

They expected to find an extreme Pareto Law in operation. They looked for it. And they were not disappointed. They found it.

There were critics of the findings, who felt that counting the raw number of edits was not a useful indicator, that there should be a measure related to that which was created, the body text. [Notice I didn’t say Content. Bad word.] And Wales indicated this would happen over time.

When I first heard this, I couldn’t get my head around the Pareto-versus-Long-Tail tension. Surely Wikipedia should behave Long-Tail rather than Pareto? And Wisdom-Of-Crowds Emergence surely cannot depend on a small gatekeeper fraternity, because that is hard to scale? But I’m used to the gatekeeper mentality, so I just stayed quiet. And confused.

Swartz was curious, and did a little bit of homegrown investigation. Here are his early observations:

  • Wales seems to think that the vast majority of users are just doing the first two (vandalizing or contributing small fixes) while the core group of Wikipedians writes the actual bulk of the article. But that’s not at all what I found. Almost every time I saw a substantive edit, I found the user who had contributed it was not an active user of the site. They generally had made less than 50 edits (typically around 10), usually on related pages. Most never even bothered to create an account.

So there was some reason to look at what the critics (of the original findings) were saying. Maybe analysis at number-of-edits level was not particularly useful; maybe the amount of original text created would be a better yardstick.

Here’s what Swartz found, admittedly in a simple “homegrown” experiment:

  • To investigate more formally, I purchased some time on a computer cluster and downloaded a copy of the Wikipedia archives. I wrote a little program to go through each edit and count how much of it remained in the latest version. Instead of counting edits, as Wales did, I counted the number of letters a user actually contributed to the present article.
  • If you just count edits, it appears the biggest contributors to the Alan Alda article (7 of the top 10) are registered users who (all but 2) have made thousands of edits to the site. Indeed, #4 has made over 7,000 edits while #7 has over 25,000. In other words, if you use Wales’s methods, you get Wales’s results: most of the content seems to be written by heavy editors.
  • But when you count letters, the picture dramatically changes: few of the contributors (2 out of the top 10) are even registered and most (6 out of the top 10) have made less than 25 edits to the entire site. In fact, #9 has made exactly one edit — this one! With the more reasonable metric — indeed, the one Wales himself said he planned to use in the next revision of his study — the result completely reverses.

Read the article for yourself. I have no idea who Swartz is, never met him, never even heard of him until I decided (following the gatekeeper debate) to try and understand what makes Wikipedia tick. That’s all.
One swallow does not make a summer. Swartz’s findings are based on a relatively small sample; and whenever you study anything remotely connected with society, there is a high risk that you bring in cultural bias and what Stephen Jay Gould called Reification (in The Mismeasure of Man). I am not, therefore, suddenly extrapolating Swartz’s findings into a resolution of the MidEast crisis.
What I am doing is sharing those findings with you. They support Wisdom of Crowds. They support Emergence. They support micromarkets and microbrands. They support Long Tail rather than Hit Culture. They support markets being conversations. All these are about people and relationships and access and empowerment, but in different fields and with different perspectives.
And most importantly, they suggest that the success of social software is based on everyone doing something about whatever it is they are passionate about. Everyone. Passion.

There’s an important principle in there somewhere, which (I hope) will be borne out by more intensive and formalised study:

Given enough eyeballs, all bugs are indeed shallow….provided there is code to look at. The value starts with the created output, not with the bug.
Gatekeepers can and should be eyeballs that help us all remove bugs and improve the pool of ….code….information….ideas….whatever.

What worried me about gatekeepers, something I could not articulate well before this, was the ability of the gatekeepers to restrict creativity. Particularly when we are yet to grow out of the Blame Culture and Hierarchy and Command and Control.

Without creativity there is nothing, no shallow bugs, no bugs, no nothing. Nothing.

Let’s not forget that as we experiment with, build, adapt and evolve governance models for social software.

To put it in Swartz’s words:

  • Even if all the formatters quit the project tomorrow, Wikipedia would still be immensely valuable. For the most part, people read Wikipedia because it has the information they need, not because it has a consistent look. It certainly wouldn’t be as nice without one, but the people who (like me) care about such things would probably step up to take the place of those who had left. The formatters aid the contributors, not the other way around.

The formatters aid the contributors, not the other way round. Great stuff.

7 thoughts on “Counting what counts: Musing about Wikipedia and Drosophila”

  1. It’s an interesting study for sure, but implicitly equating the size of the edits with their relative importance seems to me to be quite an assumption. Those small edits could be totally changing the tone of the text and alleged facts and arguably be making a bigger contribution to the “quality” of the entry than someone adding a larger number of words. Unless I’ve missed something in his methodology, it seems to me that the most active editors are not merely ensuring the format.

  2. Let’s make the big assumption that both studies are correct The results are not mutually exclusive. And if this is true, it offers a perfectly believable “old-economy” encyclo-editing model.

    There is a “desk” of 500-odd people who eat /sleep. breathe Wiki and format away to their hearts’ content.

    There are uncounted hordes of domain experts who contribute to their specific areas of interest.

    Isn’t this just a scaled-up version of the Brittanica model? That has a full time desk of 40-50 odd and many contributors. Multiply by the number of entries and the open nature of wikification and you’d get the wiki.

  3. I think there a number of distinctions between the old and the new models, which I will try and draw out in a post tonight.

  4. Britannica has authors that could be considered as reliable sources on the subject, i.e. that either like Einstein who actually submitted his theory to Britannica I believe, or the people are specialists in their fields (like Gordon Leslie Squires editor of the Uncertainty Principle of Heisenberg, Lecturer in Physics, University of Cambridge; Fellow of Trinity College, Cambridge. Author of Problems in Quantum Mechanics with Solutions and others). With WikiPedia, I fear that the common denominator of mediocrity will prevail, with a pareto equation of 80% ill-informed content. Will the Wikipedian principle of “No original Research” suffocate the creative or authentic content of its socially applied software, or not? To publish is the prerogative of the educated with access to the publishing media, a (optimistic guestimate) pareto 20% of the world’s population. Does this exclusivity guarantee the integrity of the content? Content analysis as above could or could not put light on the subject. It concerns the questionable applicability in that it makes an interpretation irrespective of the population definition, and also for the level of content that can not be measured in classical analyses of content. Content analysis rely primarily on the researcher’s subjective inferences about the messages within the texts and resulting interpretation of the content.


Let me know what you think