Leave a comment on paragraph 1 0
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.
¶ 4 Leave a comment on paragraph 4 0 Whatever form it may take, the peer review of manuscripts submitted to scientific journals always has the same purpose: to proffer a totally objective opinion. This involves taking on the role of “modest witness”, that of a man of science unpolluted by his own subjectivity and corporeality, whose historical and political emergence was so well documented by Shapin and Schaffer (1985). In her now famous text, Haraway (1996) reconceptualized this posture by arguing for the elucidation of the material conditions of existence of subjects and objects of knowledge in as much as they are continuously traversed by language, story, corporeality and metaphor. It is in this context that we now examine the forms of arrangement developed between anonymity and objectivity in the evaluation of scientific articles. We will pay particular attention to the way in which the technologies used to decide upon the fate of manuscripts attribute distinct positions of judgement within the decision-making process. And as this story unfolds, we will see that the commitment of researchers (female scholars in particular) who feel themselves to have been hard done by is far from anecdotal.
¶ 5 Leave a comment on paragraph 5 2 Peer review was one of several different technologies to emerge during the 17th century, notably at The Royal Society in London, for the purpose of carrying out experiments, determining the forms of participation by scientists and guaranteeing validation of the facts (Shapin and Schaffer 1985). In addition to the head of Philosophical Transactions, as from 1750 an editorial committee was also tasked with collegially examining and selecting manuscripts (Bazerman 1988, chap. 5). However, since D. Crane’s cross-disciplinary study (1967), we know that the editor-in-chief was the sole decision-maker, especially when it came to refusing to publish certain manuscripts, and for a long time remained in a dominant position in numerous journals. The systematic use of outside reviewers, often named referees, is a more recent practice, essentially to add to the editor-in-chief’s areas of expertise and to cope with the increasing volume of manuscripts (Burnham 1990). The fate of a manuscript nowadays depends on a series of successive judgments, the number and sequence of which varies from one journal to another.
¶ 6 Leave a comment on paragraph 6 0 So what conditions are considered optimal for making a judgment on a manuscript? This is a vast question which cannot be answered in just one short article. We will confine ourselves to examining the single issue of the respective knowledge of authors and reviewers: is it fairer to judge a manuscript in the full light of day, or hidden away from prying eyes? Should one know everything about the authors of a manuscript, or nothing at all? In short, does the anonymity of the reviewers and/or authors guarantee or prevent an objective assessment? We will begin by looking at how these became central issues for scientific journals between 1950 and 1970. We will then examine how, from the 1980s onwards, a certain number of categories became stabilized – such as the “single blind” and “double blind” which lay down the options available to journals and learned societies.
¶ 7 Leave a comment on paragraph 7 2 In sociology journals in the middle of the 20th century, manuscripts were almost exclusively reviewed internally, by editors-in-chief, their assistant editors and their editorial committees. It was within this framework of practices that the issue of author anonymity emerged. In June 1955, the American Sociological Review notified its contributors of a modification to its procedures for reviewing manuscripts:
¶ 8 Leave a comment on paragraph 8 2 As an experiment in the evaluation of articles, all papers will now be circulated to the assistant editors and judged without name or institutional identification. It will be helpful if contributors to the Review will attach a cover page giving the title, author’s name and institutional affiliation. The first page of the paper should bear the title as a means of identification, but not name and institution.
¶ 9 Leave a comment on paragraph 9 0 As far as we are aware, this original method of manuscript submission was the beginning of the anonymization of authors submitting articles to an academic journal. The proposition was relatively simple: by materially separating names and institutional affiliations from the rest of the manuscript, only the editor-in-chief would be in a position to link the submitted texts to a specific name. The following year, this experimental procedure was approved by the American Sociological Association (ASA) and the anonymization of authors was henceforth also applied to editors-in-chief. A journal’s secretariat thus became the only body capable of identifying the authors of manuscripts submitted for review .
¶ 10 Leave a comment on paragraph 10 0 In the 1960s, other American sociology journals made the same recommendations to their authors (American Journal of Sociology in 1962, Social Problems in 1966…) and in 1967 the ASA applied the principle to all of its journals. One central argument supported such a decision: an article should be assessed in terms of its content and not on the basis of the reputation of its authors or of the prestige of their institutional affiliation. It was thus a question of putting the advancement of knowledge at the heart of the evaluation process.
¶ 11 Leave a comment on paragraph 11 0 In the mid-1970s, essentially due to the publication of articles examining the issue of reviewer “bias” (Zuckerman & Merton 1971, Cicchetti & Conn 1976, Mahoney 1977), the anonymization of authors spread to other disciplines related to social sciences and the humanities. This was especially the case of literary, management, economic and psychology journals. But this extension was not driven solely by editors-in-chief. It was also the result of demands from groups of women within American learned societies who pointed out the low acceptance rates of articles written by female scholars (Bendek 1976, Weller, 2001: 225-226). The issue was thus less one of ensuring objective judgments than of repairing injustices in the name of equity between peers. The question of reviewer “bias” thus became one of justice for authors.
¶ 12 Leave a comment on paragraph 12 2 These mobilizations did not lead to researchers criticising the principle of blind reviews. Although some authors already felt that the notion was illusory and that reviewers were able to identify the authors of manuscripts, the majority apparently accepted the idea in silence. Only a few isolated researchers let their discontent be known, writing to the journals to stress the importance of a name as an identifying reference and specifying that they did not wish to be “author X” or that the content of manuscripts could not be separated from the institutional attributes of their authors.
¶ 13 Leave a comment on paragraph 13 2 We frequently forget that a man’s name is important (whether it is widely known or not). It can identify his biases and perspectives (sources of professional training, previous work, occupational, experiences, etc.) and, therefore, can be used as a basis for judging the reliability and relevance of what he says.
¶ 14 Leave a comment on paragraph 14 0 So the anonymization of authors was not considered to be a mere technical gesture. It served a radically different conception of the evaluation of scientific writings: judgments must be based purely upon the content of a text and any information about its authors will taint said judgments. The opposing vision stated that in order to come to an opinion, one needs to be able to link writings to writers.
¶ 15 Leave a comment on paragraph 15 0 When the issue of author anonymity was discussed from the end of the 1960s to the mid-1970s in the experimental sciences (physics, medicine and biology in particular), it was the latter conception of evaluation which was preferred by the vast majority. Several editors-in-chief underlined the inseparability of authors and their texts at three different levels (Ward & Goudsmit 1967, BMJ 1974). Firstly, they believed that materially speaking, the total anonymization of a text was complicated by the self-quotations that authors use on a regular basis to refer to their own previous works. Secondly they said that the credit reviewers gave to the described experiments and results was based to a very large extent on past studies and on the use of specific equipment. Anonymization would thus lead to a cognitive confusion which would hinder evaluation. Finally, they pointed out that if no names were given, reviewers would attempt to attribute texts to given authors and would base their judgments on this attribution, whether their assumption was correct or not. For these three (at least) reasons, at that time many journals, especially in the field of biomedicine, preferred not to accept anonymization.
¶ 16 Leave a comment on paragraph 16 0 In the field of experimental sciences, although recourse to outside reviewers had become common practice, there were nevertheless debates on whether or not their identity should be revealed. These referees had no dialogue with the authors; they formed a judgement which was passed on to the editorial committee or to the editor-in-chief, thus allowing them to remain anonymous . This situation was criticized for the first time by a biochemist in the New Scientist.
¶ 17 Leave a comment on paragraph 17 0 On occasion, the act of submission of a paper can place the author at the mercy of the malignant jealousy of an anonymous rival. Manifestations of antipathy can take many forms, which range from contemptuous mockery (…) to outright theft. To have suffered such an experience generally induces sympathy from colleagues, but condolences are not enough. What is needed is justice; and justice can be secured only by altering the system.
¶ 18 Leave a comment on paragraph 18 0 Jones did not say that referees were by their very nature deleterious to the editorial process, but he did consider that the anonymization procedure allowed them to act without any fear of sanction. He gave the example of how one referee deliberately delayed handing in his report so that his own works could be published first. In extreme cases, a referee might also recommend that an article be refused, in order to plagiarise all or part of the text. Jones therefore suggested that reviewers should sign their reports as was the case with other editorial practices, such as book reviews.
¶ 19 Leave a comment on paragraph 19 0 Following Jones’ article, other researchers criticized reviewer anonymity. Based on their own experiences of the editorial process, they underlined the competition between authors and reviewers which gave the latter the opportunity to write defamatory reviews of the former or, on the contrary, which prevented any verification of the referee’s competency. These criticisms stigmatized the way in which the principle of anonymity conflicted with the status given to public debate, to the very foundation of scientific activity: if authors do not know who is reviewing their work, it is difficult for them to respond to what they consider to be unfounded criticism, with any rational arguments being proscribed by the conditions of assessment. As a result, they reconnect with a conception of the scientific exchange inherited and adapted from legal practices based on contradictory discussion (Shapiro 2000). In this perspective they recommended that just like judges in the legal profession or umpires in sport, the identity of referees be made public and that they sign their evaluations so as to take responsibility for their standpoints and comments.
¶ 20 Leave a comment on paragraph 20 0 Referring to works which underlined “reviewer bias”, critics focused on the repeated divergences between reviewers of the same manuscript, the lack of care given to their reviews (e.g. not meeting deadlines, delegating the evaluation to third parties, etc.), and their lack of awareness of literature which had already been published, thus preventing them from identifying plagiarism or self-plagiarism. Whilst reviewer anonymity was not the only factor to be mentioned, critics stressed the extent to which it contributed towards such bias, the discovery of which had a more wide-ranging consequence: it undermined confidence in the entire evaluation process. The image of an upright reviewer gave way to one of a colleague steeped in self-interest, prejudice and beliefs which formed an integral part of his/her opinion. In other words, reviewers were no longer in the position of “modest witnesses” – or rather, they embodied, from a negative standpoint, the queerization of Haraway’s witness (1996).
¶ 21 Leave a comment on paragraph 21 0 For a long time these critical stances nevertheless remained very marginal compared to the assertion of the need for reviewer anonymity, initially supported by journals. In 1974, Nature responded to Jones’ article in an editorial defending the principle of anonymity for referees. It put forward three arguments. The journal believed that scientific communication should be in written form and that it would be disastrous if referees could be directly contacted by authors who were unhappy that their texts had been refused for publication, thus turning scientific discussion into open disputes. It also held that anonymity allowed room for strong but well-founded criticism, such as the similarity between the manuscript under review and previous publications by the same author(s). Finally, anonymous evaluation was deemed to be an initial and vital stage in communication with peers, dealing not only with a manuscript’s methodological or technical quality, but also with its readability and with its more general utility. From this standpoint, the removal of anonymity would hinder the process of generalization by reducing reviewers’ remarks and criticisms to a simple demonstration of personal opinions.
¶ 22 Leave a comment on paragraph 22 0 To the arguments put forward by Nature, editors-in-chief and researchers gradually added others, also based on their own experiences of the editorial process (DeBakey 1976, Wilson 1978, Morgan, 1984). Far from using their anonymity to settle old scores, referees were described as constructive, as making suggestions to improve manuscripts, and even as recusing themselves prior to any evaluation, because they considered themselves to be too close to the authors, or because they felt they lacked the required competency. Moreover, these supporters of anonymity pointed out that referees only offer opinions, with the final decision on manuscripts depending entirely on the editor-in-chief or the editorial board. They also explained that if the names of referees were revealed, there would be another negative consequence; to avoid the latter feeling that their careers were under threat, thus causing them to produce evaluations which were vapid or overly consensual, it would become necessary to only use tenure academics. Finally, some compared reviewers to electors, thus deeming it important that they vote anonymously.
Objectivity versus publicity
¶ 23 Leave a comment on paragraph 23 0 Within a context of repeated revelations of scientific fraud (Broad and Wade 1982) and of the regular publication of experiments demonstrating the flaws in article evaluation (Peters and Ceci 1982), whether or not to keep the names of authors and reviewers secret was becoming an increasingly pressing issue. To deal with the problem of anonymity, categories describing distinct solutions stabilized during the mid-1980s. Journals were thus placed in a position of choice within a space of possibilities described in table 1.
¶ 26 Leave a comment on paragraph 26 3 It was during this period that the terms “single blind” and “double blind”, borrowed from the methodologies used for controlled clinical trials, were adopted and constituted the two most common options. Blind review did not extend beyond certain social sciences and humanities and was in practice based on evaluations carried out solely by the members of editorial committees. Developed as an alternative to the single blind as from the early 1980s, open review was for a long time limited to just a few journals which encouraged or required their reviewers to sign their reports (Knox 1981).
¶ 27 Leave a comment on paragraph 27 0 Certain journals positioned themselves within this space, stressing the flexibility of their procedures: some left the issue of anonymity to the authors, allowing them to designate their reviewers, or to allow reviewers to choose whether or not to remain anonymous or to sign their reports. These various denominations lasted through to the 1990s, to such an extent that they gradually came to be considered as alternative models. From a disciplinary point of view, and as a continuation of the stances described earlier on, different surveys showed that the majority of social science journals preferred the double blind system, whereas biomedical journals (especially the most prestigious among them) maintained a single blind procedure (Weller 2001). However, this division should not be taken to mean that real concerted policies were at work beyond the range of a given learned society or academic publisher. Moreover, the trajectory of certain journals showed that depending on the period, there was a see-saw effect between single blind and double blind.
¶ 28 Leave a comment on paragraph 28 0 The American Economic Review (AER) is a perfect example, making four procedural changes between 1973 and 2011. Following a motion defended during an American Economic Association (AEA) congress in 1972, calling for the anonymization of authors in order to protect young researchers, AER’s editor-in-chief G. Borts (1974) said that it was necessary to conduct an experiment to verify the validity of the double blind. The conclusions of this experiment, carried out over a period of nine months and covering over 500 articles, were heavily in favour of the double blind procedure, considering that it was fair and that it created no financial or logistic issues for the journal. The double blind was consequently introduced in January 1974 in the form inaugurated by the American Sociological Review (a page separate from the manuscript, showing the title of the text and the name and institutional affiliation of the author) along with a request designed to avoid self-identification of the authors in the manuscript. Seven years later, the new editor-in-chief, R. Clower, significantly revamped the evaluation procedures and decided to return to the single blind method, simply stating that the changes corresponded to his own editorial preferences, which differed from those of his predecessor. In the mid-1980s, the AEA’s Committee on the Status of Women in the Economics Profession expressed concerns about the single blind’s negative effect on the acceptance rate for articles by female authors. In response, the AER’s new editor-in-chief, O. Ashenfelter, asked a female economist to set up a controlled trial to compare the effects of the single blind and double blind. Conducted between 1987 and 1989, and covering almost 1 500 manuscripts, this controlled experiment included a questionnaire for reviewers, to learn their opinions on the double blind and asking them whether or not they had recognized the authors of the article (Blank 1991). Published in December 1991 in the AER, the study showed that half of the reviewers had recognized the authors, that they believed that the double blind procedure did not hinder their evaluation and that it was materially feasible for the journal. For these reasons and despite the fact that the study offered no statistical proof that the single blind discriminated against women, in the same issue the journal announced a significant change in policy: not only did an editorial announcement formalise a shift to the double blind, but instructions to authors once again stipulated the necessity for a separate cover page for each article. Twenty years later, the journal once again returned to the single blind, this time for very different reasons:
¶ 29 Leave a comment on paragraph 29 2 Easy access to search engines increasingly limits the effectiveness of the double-blind process in maintaining anonymity. Further, it increases the administrative cost of the journals and makes it harder for referees to identify an author’s potential conflicts of interest arising, for example, from consulting.
¶ 30 Leave a comment on paragraph 30 0 This announcement represents a double reversal to support an ultimate change in procedure. The technical simplicity of anonymization via the detachable cover page gave way to the ease of identifying authors by using search engines to spot their working papers, thus making the double blind technically unachievable. Suspicion thus shifted from the assumed bias of reviewers towards possible conflicts of interest for authors. A petition was signed by 500 scholars demanding the return of the double blind (deemed necessary due to reviewer bias) along with a commitment from reviewers not to use search engines to identify authors; the petition had no effect.
¶ 31 Leave a comment on paragraph 31 0 During the 2000s, the single blind and double blind were thus maintained as alternatives, but open review was developed to such an extent that it is systematically included in works which bear upon these different options (Lee 2013). A study on the review of articles among more than 5 000 authors, reviewers and editorial board members, funded by British commercial publishers, recently examined their practices, preferences and values in this domain (Mark Ware Consulting 2008). It portrays a divided landscape, with half preferring the double blind, a quarter choosing the single blind and an eighth opting for open review.
¶ 32 Leave a comment on paragraph 32 0 The scientific and moral values attached to each option are also contrasted. Open review is valorized for the responsibility it places upon reviewers signing their reports and for the transparency and dialogue that it encourages. Supporters of the single blind focus on the anonymity of reviewers, putting them in a position where they are sheltered from the consequences of their reviews and where the fact that they know the names of the authors allows them to more accurately place their manuscripts in relation to their previous works or to detect conflicts of interest. It can also be a default choice, some researchers feeling, like the AER, that the double blind is not achievable in practice. Supporters of the double blind, on the other hand, praise the objective nature of the procedure and the equity that it guarantees, particularly with regard to lesser known authors.
¶ 33 Leave a comment on paragraph 33 4 On the matter of inter-knowledge between articles and reviewers, there are thus at least three distinct forms which still endure, supported by different motives relating to evaluation. Far from a linear vision which sees the single blind as a necessarily unfinished version of the double blind, it is in fact the coexistence of a variety of evaluation techniques developed by journals or learned societies which prevails, combined with continuous research into the evaluation of the technologies themselves. Instead of being set in stone once and for all, they are continually reassessed, within a given domain or journal, by actors whose objective is to improve or reform them.
¶ 36 Leave a comment on paragraph 36 0 Blank, R.M., 1991. The effects of double-blind versus single-blind reviewing: Experimental evidence from the American Economic Review. The American Economic Review, 81, 5, 1041-1067.
¶ 39 Leave a comment on paragraph 39 0 Borts, G., 1974. Report of the managing editor.American Economic Review, Papers and Proceedings of the Eighty-sixth Annual Meeting of the American Economic Association. American Economic Review 64, 2, 476-482.
¶ 41 Leave a comment on paragraph 41 0 Cicchetti, D.V., Conn, H.O., 1976. A statistical analysis of reviewer agreement and bias in evaluating medical abstracts. The Yale Journal of Biology and Medicine, 49, 373-383.
¶ 44 Leave a comment on paragraph 44 0 Haraway, D. J., 1996. “Modest Witness: Feminist Diffractions in Science Studies” in Galison P. and Stump D. J. (Eds.), The Disunity of Science: Boundaries, Contexts, and Power, California: Stanford University Press, 428-441.
¶ 46 Leave a comment on paragraph 46 0 Lee, C. J., Sugimoto, C. R., Zhang, G., Cronin, B., 2013. Bias in peer review. Journal of the American Society for Information Science and Technology, 64, 1, 2-17.
¶ 50 Leave a comment on paragraph 50 0 Peters, D.P., Ceci, S.J., 1982. Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences 5, 2, 187-195.
¶ 55 Leave a comment on paragraph 55 0 Wilson, J.D., 1978. Peer review and publication. Presidential address before the 70th annual meeting of the American Society for Clinical Investigation, San Francisco, California, 30 April 1978. Journal of Clinical Investigation 61, 6, 1697-1701.
¶ 56 Leave a comment on paragraph 56 0 Zuckerman, H., & Merton, R. K., 1971. Patterns of evaluation in science: Institutionalisation, structure and functions of the referee system. Minerva, 9, 1, 66-100.