MediaCommons

5. Peer Review

¶ 1 Leave a comment on paragraph 1 0 This suggests the most massive potential change that a move of the scholarly monograph into a truly electronic mode of publishing might entail — a vast transformation in both the mechanisms and the purposes of peer-review. Peer review is extremely important — I want to acknowledge that right up front — but it threatens to become the axle around which all conversations about the future of publishing get wrapped, like Isadora Duncan’s scarf, strangling any possible innovations in scholarly communication in the humanities before they can get launched. In order to move forward with any kind of innovative publishing process, we must solve the peer review problem, but in order to do so, we first have to separate the structure of peer review from the purposes it serves — and we need to be a bit brutally honest with ourselves about those purposes, distinguishing between those purposes we’d ideally like peer review to serve and those functions it actually winds up fulfilling.

¶ 2 Leave a comment on paragraph 2 2 The issue of peer review has been taken up by a number of recent publishing experiments, including arXiv’s 2004 implementation of an “endorsement” system that requires scholars to vouch for one another before they are allowed to upload papers. (It appears, however, that this endorsement system is only in effect for scholars without academic affiliations, or with email addresses that do not reveal such affiliations.) The peer review experiment that has gotten the most press of late, however, is that which was undertaken last year by the journal Nature, which was accompanied by a debate about the future of peer review. The experiment was fairly simple: the editors of Nature created an online open review system that ran parallel to its traditional anonymous review process. “From 5 June 2006,” the editors wrote, “authors may opt to have their submitted manuscripts posted publicly for comment. Any scientist may then post comments, provided they identify themselves. Once the usual confidential peer review process is complete, the public ‘open peer review’ process will be closed. Editors will then read all comments on the manuscript and invite authors to respond. At the end of the process, as part of the trial, editors will assess the value of the public comments.”

¶ 3 Leave a comment on paragraph 3 0 That experiment was closed in early December, after which time the editors did analyze the data resulting from it, and, later in the month, declared the experiment to have failed, announcing that “for now at least, we will not implement open peer review.”

¶ 4 Leave a comment on paragraph 4 0 The statistics that they cited are indeed indicative of some serious issues in the open system they implemented: only 5% of authors who submitted work during the trial agreed to have their papers opened to public comment; of those papers, only 54% (or 38 out of a total of 71) received substantive comments. But certain aspects of the experiment beg the question of whether the test wasn’t rigged from the beginning, destined for a predictable failure because of the trial’s constraints.

¶ 5 Leave a comment on paragraph 5 0 First, no real impetus was created for authors to open their papers to public review; in fact, the open portion of the peer review process was wholly optional, and had no bearing whatsoever on the editors’ decision to publish any given paper. And second, no incentive was created for commenters to participate in the process; why go to all the effort of reading and commenting on a paper if your comments serve no identifiable purpose?

¶ 6 Leave a comment on paragraph 6 0 As several entries in the web debate held alongside Nature’s peer review trial made clear, though, the editors had not chosen a groundbreaking model; the editors of several other scientific journals that already use open review systems to varying extents posted brief comments about their processes. Electronic Transactions in Artificial Intelligence, for instance, has a two-stage process, a three-month open review stage, followed by a speedy up-or-down refereeing stage (with some time for revisions, if desired, inbetween). This process, the editors acknowledge, has produced some complications in the notion of “publication,” as the texts in the open review stage are already freely available online; in some sense, the journal itself has become a vehicle for re-publishing selected articles.

¶ 7 Leave a comment on paragraph 7 1 Peer review is, by this model, designed to serve two different purposes—first, fostering discussion and feedback amongst scholars, with the aim of strengthening the work that they produce; second, filtering that work for quality, such that only the best is selected for final “publication.” ETAI’s dual-stage process makes this bifurcation in the purpose of peer review clear, and manages to serve both functions well. Moreover, by foregrounding the open stage of peer review — by considering an article “published” during the three months of its open review, but then only “refereed” once anonymous scientists have held their up-or-down vote, a vote that comes only after the article has been read, discussed, and revised — this kind of process seems to return the center of gravity in peer review to communication amongst peers.

¶ 8 Leave a comment on paragraph 8 0 This process highlights the relatively conservative move that Nature made with its open peer review trial. First, the journal was at great pains to reassure authors and readers that traditional, anonymous peer review would still take place alongside open discussion. Beyond this, however, there was a relative lack of communication between those two forms of review: open review took place at the same time as anonymous review, rather than as a preliminary phase, preventing authors from putting the public comments they received to use in revision; and while the editors “read” all such public comments, it appears that only the anonymous reviews were considered in determining whether any given article was published.

¶ 9 Leave a comment on paragraph 9 0 Was this caution about open review an attempt to avoid throwing out the baby of quality control with the bathwater of anonymity? In fact, the editors of Atmospheric Chemistry and Physics presented evidence (based on their two-stage review process) that open review significantly increases the quality of articles a journal publishes:

¶ 10 Leave a comment on paragraph 10 0 “Our statistics confirm,” they wrote, “that collaborative peer review facilitates and enhances quality assurance. The journal has a relatively low overall rejection rate of less than 20%, but only three years after its launch the ISI journal impact factor ranked Atmospheric Chemistry and Physics twelfth out of 169 journals in ‘Meteorology and Atmospheric Sciences’ and ‘Environmental Sciences’.”

¶ 11 Leave a comment on paragraph 11 0 They continue: “These numbers support the idea that public peer review and interactive discussion deter authors from submitting low-quality manuscripts, and thus relieve editors and reviewers from spending too much time on deficient submissions.”

¶ 12 Leave a comment on paragraph 12 2 So closed, anonymous peer review processes and quality control aren’t all that related. In fact, it’s arguable that the primary result of a closed peer review process is a negative one — scholars are hindered in their ability to learn from the review process, to put comments on their work to use, and to respond to those comments in kind. If anonymous, closed peer review processes don’t facilitate increased and improved scholarly discourse, what purposes do they serve? Gatekeeping, I’d argue, is a primary one; as almost all of the folks I’ve talked with over the last year about the future of scholarly publishing have insisted, peer review is necessary to ensuring that the work published by scholarly outlets is of sufficiently high quality, and anonymity is necessary in order to allow reviewers the freedom to say that an article should not be published.

¶ 13 Leave a comment on paragraph 13 0 In fact, this question of anonymity is quite fraught for most of the academics with whom I’ve spoken; they have repeatedly responded with various degrees of alarm to suggestions that their review comments might in fact be more productive delivered publicly, as part of an ongoing conversation with the author, rather than as a backchannel, one-way communication mediated by an editor. Such a position may be justifiable if, again, the primary purpose of peer review is quality control, and if the process is reliably scrupulous. However, as other discussants in the Nature web debate pointed out, blind peer review is not a perfect process, subject as it is to all kinds of failures and abuses, ranging from flawed articles that nonetheless make it through the system to ideas that are appropriated by unethical reviewers, with all manner of cronyism and professional jealousy inbetween.