I’d never heard of Sharon Weinberger or of Jack Sarfatti until a few days ago, when Chris Locke pointed me at a microwar brewing in the wikipedia space. [Thanks, Chris!]
Sharon makes some interesting observations in a post headlined Jimmy versus Jack. Examples:
- Wikipedia is a great resource, but I have been even more fascinated by a model of dealing with knowledge that could dispense with the elitism inherent in peer review. Wikipedia’s open editing model sounds so wonderfully subversive. But now that Wikipedia has a dominant web presence, it’s finding that allowing the masses to have free reign over knowledge has its downsides.
- I have argued […….] that peer review, though an imperfect system, may be the best system we have for dealing with science–at least as it pertains to science funded by government. I’ve often doubted my own conviction about this argument, and had secretly hoped that Wikipedia offered some alternative–if not for funding science–then at least for propagating science that might be unfairly quashed by peer review.
- In the final analysis, my issue […..] isn’t even whether […..] belongs to […….], but who gets to determine that classification. Wikipedians, [……], Jimmy Wales, or perhaps peer review?
- I don’t have the answer, and neither does Wikipedia,……
My italics. My elisions. My emboldening. And, if necessary, my bad.
Peer reviews can often be like benchmarking. Everyone accepts the intrinsic value, and yet everyone tends to use the process only when they can be sure that the answer is one that suits them.
I’m interested in Sharon’s observation that there is an elitism inherent in peer reviewing, particularly in scientific academia. More than just interested, intrigued. I sense that all her statements are in the same ball-park as my concerns with gatekeepers.
Humour me as I go for a wander. Take the origin of the word “bankrupt”. Wikipedia has this to say:
- The word bankruptcy is formed from the ancient Latin bancus (a bench or table), and ruptus (broken). A “bank” originally referred to a bench, which the first bankers had in the public places, in markets, fairs, etc. on which they tolled their money, wrote their bills of exchange, etc. Hence, when a banker failed, he broke his bank, to advertise to the public that the person to whom the bank belonged was no longer in a condition to continue his business. As this practice was very frequent in Italy, it is said the term bankrupt is derived from the Italian banco rotto, broken bench (see e.g. Ponte Vecchio). Others rather choose to deduce the word from the French banque, table, and route, vestigium, trace, by metaphor from the sign left in the ground, of a table once fastened to it and now gone. On this principle they trace the origin of bankrupts from the ancient Roman mensarii or argentarii, who had their tabernae or mensae in certain public places; and who, when they fled, or made off with the money that had been entrusted to them, left only the sign or shadow of their former station behind them.
I’ve looked at the Shorter Oxford and at Skeat, and they both endorse what I have always thought. The term is understood to have derived from the Italian banca rotta around the middle of the 16th century. Which is consistent with imagery of the merchants of Lombardy breaking the moneychanging bench of one of their peers when he let them down, so that he could not transact any business in the marketplace.
Peer review. Of a sort. Peer-driven action as a result of some community value or more or principle being broken. [I don’t buy the argument, suggested in the Wikipedia entry, amongst others, that the failing banker broke his own bank.]
Let’s move from bankruptcy to Speakers Corner. While there may be many such instances, the one I am most familiar with is the one at Hyde Park.
The rules appear to be simple and consistent. Anyone is allowed to speak. But you cannot trash the monarchy or seek to overthrow the government. Apparently.
So you can speak about pretty much any subject you like. And there are no bouncers or gatekeepers about. But there are regulars about. Regulars who listen to anyone and everyone, quick to heckle, quick to clap. Who are these regulars? In the context of the All Comers market, their peers.
Destruction of the tools of trade, as in the Lombardy bankrupts. Ejection from the place of trade, as in defenestration. Heckling and jeering, as at Speakers Corner. At some level of abstraction, I guess that even jury trials are a form of peer review.
All of them have their strengths and weaknesses. The bulk of the weakness comes from being able to game the system, which happens as soon as you introduce some formal selection process for the peers, or (much worse) some barriers to selection or entry.
Peer review processes work best when the peers are the men on the Clapham omnibus, or even with Twelve Angry Men.
But we need to keep the selection criteria for peers as open as possible, and the barriers to speaking/publishing/sharing as low as possible.
I need to understand more about how elitism comes into play in scientific reviews. Any offers out there?
I share the concern about elitism, but I also worry about the opposite — what I sometimes playfully call “chronocracy”: rule by those with too much time on their hands. :-) If there are no barriers to entry whatsoever, then those who genuinely care (as demonstrated by having ‘paid their dues’) can often be roughshod by those who have more energy but less sense.
I believe there’s a certain optimal tension between low barriers to entry (for purpose of innovation) but high barriers to exit (for purpose of discipline). Any thoughts on how to achieve that?
JP, it is interesting to see TWELVE ANGRY MEN invoked in a discussion of peer review. I suspect that the “jury of one’s peers” language from Magna Carta constitutes the ur-definition of the peer review process. That language, by the way, does not appear in the United States Constitution. To learn more, check out:
http://www.familyfriendlyjuryduty.org/JuryDutySystem/JuryDutySystem.htm
Nevertheless, the jury for a criminal trial is not a model for scientific peer review. Most importantly, the selection of the jurors involves a lengthy and elaborate screening system in which both the judge and the lawyers for both sides participate. Furthermore, once the jury has been formed, they have very few decisions to make, depending on the number of charges brought against the defendant. This is a far cry from the number of decisions that have to be made in scholarly publication these days, let alone the selection process for the review board!
Finally, as I believe I mentioned in another discussion, taken as a case study, TWELVE ANGRY MEN demonstrates an example of where deliberation leading to consensus trumps decision by majority number of votes. Again, when you consider the volume of content involved in academic publication, there is neither the time nor the inclination to deliberate in search of consensus. My guess is that each review is content to register an opinion and be done with it.
Had we but world enough and time, academic publication would have more to do with EDITING and less with using peer review as a filter. The German Wikipedia has the potential to realize such an editing model. I discussed this in my blog at:
http://blog.360.yahoo.com/blog-Mff23hgidqmHGqbcv.lfskakEtS6qLVHUEMFUG4-?cq=1&bid=82&yy=2006&mm=8&p=81
Whether or not the Germans will really do this will depend on the volume of content they have to process. My guess is that the content will be too much for such a collaborative-editing approach; but we can always hope!
Interesting stuff, thanks.
Doc, your question reminds me of the classic freerider problem for any “commons”, where a person or persons consumes a disproportionate share of a resources in the context of the cost share. But unlike classical economics arguments on freeriders, the question throws up an interesting twist. The resource used is time. “Now” time, as they filibuster their way because they can. The cost underpaid is also time. The time they didn’t invest while the others did.
Needs thinking about, I have no simple answer on the optimisation question. Intuitively I am uncomfortable with banning freeriders unless I can see destruction of value.
And Stephen, I threw in Twelve Angry Men accidentally-on-purpose :-) to see what happens. Thanks for your comments as well.
JP
on the specifics of the elitism of Peer Review. Two items spring to mind. In an experiment a number of year back a number of published papers – published in leading peer reviewed journals – were resubmitted to other leading journals. None of the content was changed. They were rejected. The difference was that that when published originally they were submitted by authors from leading universities (e.g. Harvard etc). When resubmitted they came from researchers no one had heard of before in universities much further down the food chain.
There is also a similar effect commented on (the effect has a name the escapes me at the moment) whereby the most well know author of a group of authors submitted a paper is this author that is best associated with the paper even if they had least to do with it, with other strange side effects.
Then there is the case of Sokal Hoax a few years back which wasn’t elitism per se, but an attempt to ride on the back of scientific credentials which by a publication which derives from a similar psychological perspective.
By the way when trying to come up with the name of the effect again I came back across the research of Willam Starbuck http://pages.stern.nyu.edu/~wstarbuc/ whose work may interest you
Regards
Dermot
Thanks Dermot, I will follow your leads with interest. This is a big subject and one which is at the heart of social software.
Dermot, I think you may have misread the results of the Sokol Hoax. The result had very little to do with credentials, particularly since the author (intentially) lacked any credentials in the journal to which he submitted the bogus article. It was not his scientific credentials that got his article published but the ILLUSION of credentials created by his use of the right scientific jargon. The illusion was deliberate because his use of the jargon was content-free. He wanted to demonstrate that the editorial board was so hung up on jargon that they never bothered to see if the paper he submitted actually said anything. It didn’t, but it still got published. When Sokol revealed what he had done, the editors reacted by saying something like, “The author may have THOUGHT that his paper did not say anything, but we believe that there was still significant meaning embedded in this text!” Make of that what you will!
By the way I discovered Starbuck as part of being caught up in the “knoweledge movement” about ten years ago; and I still read his stuff with great interest!
Compare with the market for fine art, where the authenticity of the signature counts far more than the value or quality of the content.
Couldn’t happen with blogs, could it? :-)
Not sure I can agree with that, Andrew. The first thing that has value attributed to it is the creation, Then, as that value grows, the signature accretes value. But I know of no artist whose value is solely in his signature.
Stephen
yes that was one of the points I was trying to make. The editiors who accepted Sokals paper, didn’t evaluate his arguements on their merits (and couldn’t because they were gibberish) they evaluated it on the basis that it agreed with what they wanted to hear.
Dermot
I am afraid that the concept of “the value of a work of art” will always be a can of worms, simply because there are so many different ways to answer the question “value to whom?” If you were Isabella Stewart Gardner, you knowledge of art was minimal; and, to invoke that old cliche, you probably did not even know what you liked. So you paid an “expert” to “furnish your home with art;” and, in the absence of any more defensible criteria, your expert staked much of his reputation of authenticity. You then “bought into” that particular criterion, thus reinforcing a social consensus that tightly couples value to authenticity.
Now let us consider Marcel Duchamp and his readymades. Here is a description of how Duchamp made one provided by Calvin Tomkins:
He bout in an art-supply store three copies of a cheap chromo-lithograph depicting an insipid winter landscape, added two dots of color, one red and one green (the colors of the bottle often placed in druggists’ windows), and called the result PHARMACY.
If you were Walter Arensberg, you thought this was really cool; and, without any consultation from any experts, you wrote Duchamp a generous check, because you wanted this in your personal collection. Eventually, you donated your collection to the Philadelphia Museum of Art, where all of us can now see it. I for one feel that Arensberg’s check was money well spent; and, as a rule, I take more satisfaction in visiting the Arensberg collection than in worshiping at the temple of Isabella Stewart Gardner (unless I happen to be at that temple on the afternoon of a good concert)!
Back when we had the money to spend on art, my wife and I had very few rules. The most important was that we would not buy anything without first holding a “family conference.” At that conference one question dominated all others: “Do you want to look at this every day of your life for the rest of your life?” If we both agreed on an affirmative answer and we felt that the price was within our means, we went for it. Since a presumably original Renoir or van Gogh was always out of our range, we never really worried very much about authenticity. (We were once very amused to find a piece by one of our favorite sculptors hiding in a corner of a gallery in Sonoma and going at a “bargain basement” price, just because the owner of the gallery did not follow the artist’s career; but that was just a stroke of luck for us.)
So, if I were offered a known forgery of Renoir that had only been exposed after years of meticulous research, would I still pay good money for it? Fortunately, I no longer invest in art! If you want to perform that GEDANKENEXPERIMENT, just ask how comfortable you will be living with your decision!