As part of Peer Review Week (September 19-25), Rory McVeigh, co-editor of the American Sociological Review, has written the following blog post. Thank you to our reviewers at all of our journals–your work is essential to the scholarly publishing process!
Frustration with the peer review process among both authors and reviewers has led some sociologists to propose that academic journals should move to an evaluative, rather than developmental, peer review. In an evaluative review, the reviewer’s primary role is to assess the quality of the work and to make a recommendation as to whether it is strong enough for publication. In a developmental review, the reviewer not only evaluates the quality of the work but also offers guidance on how the paper might be improved through revision. At American Sociological Review, we are committed to maintaining the developmental review model, while recognizing that there is room for both models in th
e discipline, given the broad range of publishing outlets available to scholars. Below, I briefly discuss the benefits of the developmental review and also clear up some common misconceptions that, if resolved, could make writing a developmental review more attractive and less burdensome, and could make reviews more beneficial to authors and to journal editors.
Benefits of Developmental Review
Perhaps, most obviously, a developmental review offers benefits for authors. Perhaps especially for young scholars, going through a developmental review process provides a type of feedback that often cannot be easily acquired from peers and mentors who are willing to read a paper prior to submission. A journal’s reviewers can help authors to assess where the bar is set in terms of standards required for publication in aspirational journals. More importantly, the developmental peer review process allows authors to transcend their local environment, giving them access to a broader range of expertise and insight than might be available in their own departments or in their own networks. To the extent that homophily drives our selection of mentors and departmental colleagues, a journal’s reviewers can provide a perspective that draws attention to problems that our well-intentioned mentors and colleagues may miss.
The generous feedback provided by reviewers, coupled with editors’ guidance in navigating feedback from multiple reviewers, can produce papers that are clearer, more impactful, and more effectively executed.
Developmental review also benefits journals by improving the quality of work that is published. With my nine years of experience as a journal editor, I can’t recall a case where a paper that was published was not of higher quality than the one that was initially submitted. The generous feedback provided by reviewers, coupled with editors’ guidance in navigating feedback from multiple reviewers, can produce papers that are clearer, more impactful, and more effectively executed. Readers of the journal, in turn, benefit from the improvements made through review. Finally, the developmental review can help to unclog an entire discipline’s peer-review process in which many scholars feel overburdened in their roles as reviewers and many authors are frustrated by the time it takes for a decision to be made on their submissions. One problem with an evaluative review system is that it essentially “kicks the can down the road.” If authors receive little feedback to help them to improve their papers, they are likely to submit the same paper—with all its problems still in place—to another journal and a new set of reviewers who have to devote valuable time to evaluating things that the author might have fixed if she or he had been given more guidance.
Misconceptions about the Developmental Review
A common complaint about the developmental review—and an excuse used by some for turning down requests from editors—is that writing the review can be too time consuming and the reviewer ends up doing work that should have instead been done by the author prior to submission. This is a perfectly valid complaint. The solution, however, is not to decline requests to perform reviews. Instead, reviewers should expect authors to live up to their end of the bargain and they should hold authors accountable if they do not. Prior to submission, graduate students and younger scholars, in particular, should rely on faculty mentors to help them strengthen their papers and to help them to make decisions about where they should submit the paper. More senior scholars should also secure feedback on papers from colleagues and through conference presentations so that they can submit more polished papers to journals. Authors should not exploit reviewers’ by using the submission process to get “free” feedback on articles that were not ready for a serious evaluation. In cases when authors are sending out work before it is ready, honest (but respectful) feedback to the author about the paper’s general shortcomings can provide a very important developmental function. Authors learn that they need to get serious before submitting their work. It is not the reviewer’s responsibility to correct grammatical errors or sentence structure, and it is not the reviewer’s job to help the author to come up with a research question or a coherent theoretical argument. Journal editors also have a responsibility here, as in some cases papers simply should not be sent out for review because the author did not take the time to learn, and attempt to meet, the journal’s standards.
In cases when authors are sending out work before it is ready, honest (but respectful) feedback to the author about the paper’s general shortcomings can provide a very important developmental function. Authors learn that they need to get serious before submitting their work.
Reviewers, however, can provide a very valuable service to authors and editors by focusing on the big picture. This involves identifying flaws and weaknesses in the arguments and analysis and, where possible, providing constructive feedback that can help the author to address those problems. Effective reviews are those that don’t insist upon a particular framing of the research question that aligns with her or his own preferences, when doing so does not improve the paper. Articles improve the most when reviewers try to put themselves into the mindset of the author, helping to improve the paper on the author’s own terms while helping the author to recognize blind spots that she or he may be displaying, logical flaws, and problems with research design and analysis.
I do not want to create the impression that writing a developmental review requires little time and effort. Instead, I am suggesting that reviewers often invest their energy on the wrong things and as a result feel put upon by the process. Rather than spending time doing work that the author should have done before submitting the paper, the reviewer can instead focus on helping the author to solve problems by sharing expertise and experience needed to produce impactful research. Rather than feeling (understandably) resentful, the reviewer can instead feel the satisfaction that comes from knowing that she or he played an important role in developing not only a strong piece of scholarship, but also in developing the scholar.
Guidelines for Reviewers
In my many years of engaging with the peer review process as an author, a reviewer, and an editor, I have been struck by the extraordinary variation in the content of reviews (in terms of length, style, quality of feedback, etc.). Most of us have received little or no training when it comes to writing reviews and are, therefore, left to figure out editors’ expectations on our own. To help assist reviewers, at American Sociological Review we have developed guidelines for reviewers to help them to write the kinds of reviews that will be most helpful to the editors and authors. We have developed general guidelines, as well as guidelines that are tailored toward evaluating work utilizing different methodological approaches (such as ethnographic, comparative-historical, and policy-oriented papers, and theory papers). We embed links to the guidelines in emails inviting manuscript reviews, but have also made them available on the journal’s website. We invite you to check them out and perhaps share them with graduate students who are entering the pool of reviewers that are vitally important to the success of any journal
Asking reviewers to provide guidance about how a submitted paper might be improved is an excellent idea. It would also be very helpful if authors were required, or at least strongly encouraged, by editors to include similar guidance in their conclusions in published papers. In his 1987 semi-autobiographical contribution to the Annual Review of Sociology, “Three Fragments from a Sociologist’s Notebooks,” Robert Merton highlighted the importance of what he called “specified ignorance.” Authors should conclude their articles, he wrote, not simply by summarizing their findings, but by providing concrete suggestions as to what further steps in research would be most fruitful. Who knows better than the authors of articles what those steps should be? By providing guidance, authors could stimulate others to follow up on their published research, enhancing both the rate of advance and, not incidentally, their own influence in the field.
This may seem obvious, but Merton was focusing on specified ignorance for a reason. In a review of recent journal articles on policy change, I discovered that authors almost never do this. Often, their conclusions simply summarize their findings. When they do suggest next steps, the suggestions are usually generic and not especially helpful–typically, that the same research be repeated for other countries or other time periods.
If reviewers and editors saw to it that authors make serious suggestions for future work in their conclusions, articles would be much more useful to those trying to advance their fields, including the authors themselves.
Bravo! An important message, well said.