NYTimes > Scholars Test Web Alternative to Peer Review

Published in "Scholarship 2.0: An Idea Whose Time Has Come", on August 25, 2010.

"Now some humanities scholars have begun to challenge the monopoly that peer review has on admission to career-making journals and, as a consequence, to the charmed circle of tenured academe. They argue that in an era of digital media there is a better way to assess the quality of work. Instead of relying on a few experts selected by leading publications, they advocate using the Internet to expose scholarly thinking to the swift collective judgment of a much broader interested audience."

How would such a system avoid rating publications according to their writer's popularity and truly rate the publication's merit?

"Mixing traditional and new methods, the journal posted online four essays not yet accepted for publication, and a core group of experts — what Ms. Rowe called “our crowd sourcing” — were invited to post their signed comments on the Web site MediaCommons, a scholarly digital network. Others could add their thoughts as well, after registering with their own names. In the end 41 people made more than 350 comments, many of which elicited responses from the authors. The revised essays were then reviewed by the quarterly’s editors, who made the final decision to include them in the printed journal, due out Sept. 17."

Does this really shift the emphasis from a closed peer review system to open, web based mass peer review?

"Clubby exclusiveness, sloppy editing and fraud have all marred peer review on occasion. Anonymity can help prevent personal bias, but it can also make reviewers less accountable; exclusiveness can help ensure quality control but can also narrow the range of feedback and participants. Open review more closely resembles Wikipedia behind the scenes, where anyone with an interest can post a comment. This open-door policy has made Wikipedia, on balance, a crucial reference resource. "

Why do reviewers need to be more accountable? Their responsibilities lie in judging the merit of an article and no more. Exclusiveness in reviewing is essential in order to make sure the qualified personnel judges the merit of a paper. However, exclusiveness does not limit readership (especially in the case of open access journals). HOw can we make open review into a useful process in peer review?

"Advocates of more open reviewing, like Mr. Cohen at George Mason argue that other important scholarly values besides quality control — for example, generating discussion, improving works in progress and sharing information rapidly — are given short shrift under the current system. "

Generating discussion can be measured using the impact factor- The number of times a paper is cited in other papers, thus indicating the existence of a process of discussion, albeit a very slow one as the papers are published in high frequency.

"To Mr. Cohen, the most pressing intellectual issue in the next decade is this tension between the insular, specialized world of expert scholarship and the open and free-wheeling exchange of information on the Web. “And academia,” he said, “is caught in the middle.”  "

Would expert scholarship still count as expert if so many non-specialists are involved in generating it ?