About six months ago, I published a lengthy post, both on Planned Obsolescence and on if:book, about the future of peer review in electronic scholarly publishing. At least some portion of that post was occasioned by Nature's experiment with an open peer-review system. That experiment was closed earlier this month, and the editors have now analyzed the data resulting from it, and have declared the experiment to have failed, and have announced that “for now at least, we will not implement open peer review.”

The statistics that they cite are indeed indicative of some serious issues in the open system they implemented: only 5% of authors who submitted work during the trial agreed to have their papers opened to public comment; only 54% of those papers (or a 38 of a total of 71) received substantive comments. But I have to wonder whether the experiment wasn't rigged from the beginning, destined for a predictable failure because of the trial's constraints.

First, no real impetus was created for authors to open their papers to public review; as I noted back in June, the open portion of the peer review process was wholly optional, and had no bearing whatsoever on the editors' decision to publish any given paper. And second, no incentive was created for commenters to participate in the process; why go to all the effort of reading and commenting on a paper if your comments serve no identifiable purpose?

What I want to ask at this point is what MediaCommons can learn from the ostensible failures of Nature's experiment. How can we develop a successful open peer-review process with adequate author and reviewer buy-in?

This entry was posted on Friday, December 22nd, 2006 at 2:36 am and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.