Demystifying the Peer Review Process: Advice from Associate Editors

May 1, 2017

Guest post by Ethan R. Van Norman, Georgia State University

In a previous post, the ECF offered guidance on identifying where to submit a manuscript. One piece of advice was to consider the composition of the journal’s editorial board. From that we can infer that the likelihood your paper will be viewed favorably depends in part on the background and theoretical orientation of the individuals that will review your paper. Many of the readers of the ECF serve on those editorial boards. In the frenzy to submit papers and hopefully obtain tenure, it is easy to lose sight of the importance and necessity of the peer review process. In fact, wading through reviewer comments to revise an article (if we are so lucky to be invited to do so) is often viewed as a chore instead of an opportunity to improve our work. Such woes may be self-inflicted (e.g., we did a poor job setting up the rationale for our study or overlooked critical threats to the internal validity of our findings) or, as many of us would rather believe, we are tasked with addressing reviewer comments that appear to be devoid of meaningful content or consist of unfair criticisms without solutions. The purpose of this blog post is to highlight steps that early career scholars can take to ensure that the reviews they provide are of the variety to make the revision process for colleagues as pain free as possible.

In November Michelle Demaray discussed serving on an editorial board pre-tenure. In that post Dr. Demaray cited two important characteristics of a reviewer: (1) promptness and editorial board etiquette as well as (2) writing quality reviews. The purpose of this post is to further discuss what constitutes a high-quality review from the perspective of associate editors. This topic is particularly relevant for early career scholars because many universities expect pre-tenure faculty to serve on editorial boards. Indeed, serving on an editorial board is an effective and meaningful way to engage in service for the field. Beyond early career considerations, peer review is a mechanism to ensure scholarly work of sufficient quality and importance is published. High quality reviews can help authors re-conceptualize or refine the initial premises of the paper, which may ultimately lead to a more meaningful or methodologically sound work.

To further explore what early career faculty can do to complete high quality reviews, we solicited responses to a series of questions from two associate editors of school psychology journals. Given the somewhat sensitive nature of the questions, we elected to withhold the names of the associate editors. The ECF is grateful for the feedback from both contributors.

Advice to Reviewers

Question 1: What types of comments do you seek from reviewers? That is, which type of comments do you find the most useful when rendering your editorial decision? Is there a general “approach” you recommend reviewers take when conducting a review?

AE #1: I find that reviews that not only provide a general perspective on the manuscript, but also provide specific information regarding strengths and questions/concerns to be the most useful. Reviewers who articulate the big ideas from their review (e.g., conceptualization issues, methodological strengths and problems, how findings were reported) up front and then expand by illustrating specific instances that support their perspectives not only help me make sense of their viewpoints, but also can help the authors with any subsequent revisions. In other words, I think a critical discussion of the manuscript typically is more helpful than a list of issues that require the authors and the action editor to construct the reviewer’s message.

AE #2: I find reviews most helpful if they (a) comprehensively evaluate each section of the manuscript, (b) provide an overall evaluation of the manuscript, (c) are constructive, and (d) note positive attributes of the manuscript.

Question 2: Relatedly, to what degree do you consider reviewer feedback when rendering your editorial decision? Can you elaborate on a situation in which you rendered a decision counter to the prevailing opinion of the reviewers – or the circumstances in which that might occur?

AE #1: Reviewer feedback absolutely informs the decisions that I render. I am one person who cannot possibly have a command of all of the issues surrounding a given topic or study. Quality reviews help me identify and think through issues relevant to a manuscript. In many instances, they also can help with articulating the rationale for a decision and issues to be addressed if a revision is invited. I have yet to render a decision for a manuscript that was not consistent with at least one or more of the reviewers; however, when I have rendered different decisions than those recommended by individual reviewers, it typically had to do with a difference in perspective (e.g., the appropriateness of the methods used, the fit of the manuscript for the journal). As much as we often strive to be objective in our field, subjectivity absolutely is a part of the peer review process.

AE #2: I strongly consider reviewer feedback as a critical component of the peer review process. An editorial decision is not, however, a “vote count” or tally. In some instances, I or a reviewer may identify a critical flaw not identified by other reviewers.

Question 3: Inevitably every author will receive an undesirable editorial decision. However, it seems that subjective experience of the peer review process can be highly influenced by the tone of reviewers. Do you have any advice to reviewers to ensure that their comments are as constructive and as respectful as possible?

AE # 1: The peer review process requires a critical review of manuscripts so that scientifically-sound work gets published. However, it can be easy to get on a roll with issues you identify and not remember that other human beings will be on the other side of the decision letter. It helps me to remember that science is a tool invented and used by humans to try and understand their world, which means that no scientific endeavor is perfect. However, all papers I have reviewed or have been the action editor for had strengths in addition to limitations. Additionally, I would encourage reviewers to remember that nobody is correct 100% of the time. It doesn’t hurt to include language such as “my perspective…” or “my thoughts are…” when you don’t agree with a decision that could be defensible.

AE #2: I think it is helpful to make comments that are encouraging, specific, and can be addressed. It is particularly useful when reviewers point to a paper or resource authors can review to improve their work.

Question 4:  If you had to give 1-2 key tips for new reviewers, what would they be?

AE #1: Remember the reviews that you have found the most helpful as an author. I suggest operating from the spirit of pointing out issues and providing suggestions to assist the authors in improving the scholarship represented by the manuscript. It isn’t a reviewer’s job to tell the authors what to do, but providing your perspectives and suggestions can do wonders for authors who ultimately want to see their work published.

AE #2: Accept or decline invitations to review quickly, submit reviews on-time, and create comprehensive and constructive reviews.

Advice to Authors

Question 1: It seems that receiving a decision of “Revise and Resubmit” is often a sigh of relief for new authors. What advice do you have for authors to increase the likelihood that they will receive the coveted “Tentative Accept” or similar decision without going through multiple rounds of reviewer feedback and revisions?

AE #1: I always recommend putting the decision letter down after the initial read and coming back to it once you have had a chance to reflect on how you will respond. Once you come back to it, I strongly recommend that you make sure that you thoroughly and clearly address any direction for revisions provided by the action editor. S/he basically is telling you what s/he expects to see for the paper to be published. I also would write a very clear response letter that articulates exactly how you responded to the requested revisions and provides a strong rationale for any revisions you decided not to make (I would minimize these instances as they will need to be very compelling to convince an editor that his/her initial direction should not have been addressed by the authors). I also would clearly articulate how you responded to any reviewer comments not directly addressed by the editor.

AE #2:
1. Include testable research questions that emanate clearly from the literature review.
2. Make explicit the importance and implications of the study.
3. Clearly link all sections together, based on the research questions.
4. Include only implications that are supported by the study findings.
5. Adhere to the Publication Manual of the American Psychological Association (6th ed.).
6. Attend to details.

Question 2: The time from submitting a manuscript to a journal and receiving an editorial decision can seem excruciatingly long. Given that different journals set different expectations for rendering a decision, do you have any advice as to when an author should follow up with an associate editor regarding a paper? Further, do you have any advice in going about making that request?

AE #1: I definitely would not follow-up before the anticipated decision date. I can’t speak for other action editors, but I would not mind receiving a follow-up email at any point after the communicated timeframe for rendering a decision (2-3 months for my journal). In terms of making the request, I would recommend letting the action editor know that you understand how busy s/he is, but that you were wondering when you might expect to receive a decision. Most of us have been in your shoes as an author and a respectful email may facilitate us moving up on our to-do list rendering a decision on your manuscript.

AE #2: Many journals report average time to decision. I recommend checking the journal website and any associated materials that may identify a timeline. I think it is reasonable to check-in with an Associate Editor or Editor two months after the average time to decision. If a timeline is not reported, I think it is reasonable to check-in after about 9-months.

Conclusion

For more information on the peer review process in School Psychology, Floyd, Cooley, Arnett, Fagan, Mercer, and Hingle (2011) highlight, among other things, what editors consider to be desirable characteristics of reviewers. In line with the comments offered from the current associate editors, editors in that study viewed reviewers that provided constructive feedback with specific strategies to overcome perceived shortcomings as being the most helpful. To that end, early career scholars can seek out references to identify how to write constructive feedback for reviews (e.g., http://www.apa.org/pubs/authors/review-manuscript-ce-video.aspx) as well as handbooks that detail desiderata for different quantitative (e.g., Hancock & Mueller, 2010) and qualitative (e.g., Trainor & Graue, 2013) methods. In conclusion, it’s important to remember that conducting high quality reviews is important for our field. Yes, turning in another review may be another item on your weekly to do list, but the next time you grumble about the amount of time it’s been since you heard back from a journal or the next time your call a colleague to vent about the tone and content of a review you received– take a moment to look back at the most recent review you completed. Did you complete it on time? Was the tone respectful? Did you offer specific rationale for the criticisms you offered and provide specific recommendations to address those criticisms? Were the critiques fair to begin with? Reviewing others how you’d like to be reviewed may go a long way in ensuring the revision process is a worthwhile endeavor.

Do you have different perspectives for how to conduct a high-quality review? Do you use different resources / approaches to review papers? Feel free to share your insights / experiences in the comment section below!

References
Floyd, R. G., Cooley, K. M., Arnett, J. E., Fagan, T. K., Mercer, S. H., Hingle, C. (2011). An overview and analysis of journal operations, journal publication patterns, and journal impact in school psychology and related fields. Journal of School Psychology, 49, 617-647.
Hancock, G. R., & Mueller, R. O. (Eds.). (2010). The reviewer’s guide to quantitative methods. New York, NY: Routledge
Trainor, A., A., & Graue, E. (Eds.). (2013). Reviewing qualitative research in the social sciences. New York, NY: Routledge.

Leave a Reply

Your email address will not be published. Required fields are marked *