Systematic Reviews and Meta-Analyses: Opportunities for Pivoting Research during COVID-19

By Kathrin Maki, University of Florida & Bryn Harris, University of Colorado Denver

Given the ongoing pandemic, many early career researchers may need to pivot their research programs because timelines for tenure and promotion and annual expectations may be unchanged and there may be ongoing disruptions to prior research activities. This leaves researchers looking for new, creative ways to advance their research agendas – a challenging yet exciting process. Completing systematic reviews and meta-analyses may be one method of advancing research and productivity during the pandemic, while also contributing to the literature base and knowledge in the field.

Defining Systematic Reviews and Meta-analyses

The aim of a systematic review is to use systematic methods to summarize empirical evidence to produce reliable findings that inform decision making (Cochrane, 2021). Many literature reviews within articles in peer-reviewed journals differ from systematic reviews because they do not involve a systematic search of the literature, but rather authors select the articles reviewed, which may result in selection bias. Systemic review is an umbrella term for approaches that attempt to reduce this bias to provide a comprehensive synthesis of available evidence on a given topic.

Like all research, systematic reviews vary in quality. Robust, reproducible methods for determining eligibility criteria for studies will result in a higher caliber systemic review. The comprehensiveness of the systematic search is particularly important in systematic reviews. The creation of a comprehensive and well-conceptualized protocol for study inclusion is an essential component of a strong systematic review. This article provides a detailed overview of conducting a strong systematic review in the field of psychology. For example, researchers need to decide on study inclusion criteria such as participant age range, study design, conditions, language of publication, and publication type. Researchers will need to select the type of studies to include such as RCT only, quasi-experimental designs, single-case designs, or qualitative research. Researchers will also need to determine what type of literature to include, for example, the potential inclusion of unpublished studies.

Meta-analysis is a subtype of systematic review. Meta-analyses use statistical methods to summarize the results of the included studies. By quantitatively combining outcomes from all relevant studies, a meta-analysis can provide specific estimates of efficacy that are more precise than single studies because they are computed from a larger body of information (Cochrane, 2021). Like systematic reviews generally, the protocol process for study inclusion is often the most time intensive aspect of meta-analysis. Many published meta-analyses do not include important variables such as race, culture, and indicators of socioeconomic status, thus leading to large gaps in the literature. Examining such variables provides comprehensive information on treatment effectiveness across populations. In systematic reviews generally, and meta-analyses specifically, a discussion of the way in which the authors reduced their risk of bias is also considered best practice (referred to as publication bias; McClain et al., 2021). Although meta-analyses rely on quantitative data, a different form of systematic review called a meta-synthesis summarizes qualitative studies. See Lachal et al. (2017) for an overview of this methodology.

Advantages of Systematic Reviews

 There are several potential advantages of systematic reviews for early career scholars’ research agendas. Many early career scholars experience the pressure of demonstrating research productivity through consistent publication of articles in high quality journals relevant to their field. As part of the tenure and promotion process, early career scholars often need to demonstrate their research productivity (in addition to teaching and service). Systematic reviews contribute to research in school psychology by providing a comprehensive understanding of the state of the evidence for a specific topic. The knowledge gained through systematic reviews can provide synthesized empirical support for implementing certain practices or it can suggest areas in need of further research when results suggest practices may be ineffective or lacking support.

Systematic reviews are also often highly cited, and if conducted rigorously, may be published in strong journals. At some institutions, metrics such as impact factor and citation counts factor into tenure and promotion decisions. Regardless of how closely impact factors and citation counts are examined at your university, you will want to show how your research contributes to the field and such metrics are one way to do so. All school psychology journals welcome systematic reviews and meta-analyses submissions.

Advantages to Conducting Systematic Reviews During the COVID-19 Pandemic

 One year into the pandemic, scholars continue to deal with its effects on their research productivity and teaching. For many school psychology researchers, our work is dependent upon collecting data in K12 schools, but many school districts across the country continue to provide instruction virtually. Even in school districts where face-to-face instruction is provided, students’ educational experiences are anything but traditional. Further, many schools are restricting visitors from school campuses in an effort to minimize opportunities for spreading the virus. Similarly, data collection in other applied settings (e.g., clinics, hospitals) has also been impacted by changes in how services are provided to clients (e.g., telehealth). As a result of these significant changes in systems’ functioning, applied researchers are experiencing significant setbacks in conducting research.

Systematic reviews can offer an alternative research pathway for students, faculty, and other early career scholars who have hit significant roadblocks in conducting research during the pandemic. The most meaningful, and perhaps obvious, advantage to conducting systematic reviews is the elimination of the logistical challenges of conducting traditional research in applied settings during the pandemic. Data collection for systematic reviews is dependent upon accessing published articles (and perhaps dissertations and grey literature) from your university library, which you can comfortably do while you work from home during the pandemic.

Speaking of your university library, your librarian can be a valuable resource if you conduct a systematic review. They are experts in how to navigate the library databases, which databases may be most appropriate for your area of research, how to effectively search the databases, and many other related areas as well. In our experiences, university librarians have been extremely helpful for accessing and learning various resources to support such work.

Challenges and Limitations

For many of us in school psychology, we are all too familiar with the logistical challenges and time intensity of collecting data in complicated systems (e.g., schools, clinics, hospitals). Systematic reviews, therefore, may appear to be the answer you’ve been hoping for to continue your research while alleviating the challenges associated with traditional data collection. And they can be. However, it is also important to be aware of the challenges of conducting systematic reviews before getting started.

Systematic reviews can be time intensive and, like all research, require a rigorous methodological approach. They require significant planning and forethought to develop your methodology as you plan and revise appropriate search terms, inclusion/exclusion criteria, and coding schemes before implementing your study. Like school-based research, ensuring you have enough personnel resources can also be a challenge. Your systematic review will be more methodologically rigorous and the results more reliable if more than one researcher determines study eligibility and codes at least a portion, if not all, of the articles and variables. Ensuring you and your collaborators reliably determine study inclusion and code variables typically requires an iterative process of practice coding (and resulting revisions to the criteria and codes), all of which can take an extensive amount of time.

There are also methodological challenges associated with systematic reviews. Attempting to review studies that have been conducted in different ways and incorporate different variables or measures can present a quandary as to how best to capture study methods and appropriately synthesize or meta-analyze findings. Further, meta-analytical approaches continue to advance including, but not limited to, the use of multilevel modeling, effect size calculation for single-case design studies, and means for assessing publication bias, all of which require methodological and analytical skill fluency. It may be helpful to consult or collaborate with a methodological or quantitative expert for procedural and statistical support.

More systematic reviews and meta-analyses have been conducted during the global pandemic across the world. However, a recent meta-analysis of medical studies pertaining to COVID-19 interventions has uncovered substantial flaws in research design, thus leading to potentially problematic medical intervention decisions based on the results. This finding also raises the issue of promoting expedited science and the need for scientific rigor regardless of global context. School psychology researchers must understand the nuances of this method and work to produce rigorous studies in order to support effective decisions among stakeholders using study results (e.g., practitioners, policy makers, other researchers).

Introductory Readings and Resources

 The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) provides guidance on the reporting of systematic review methodology and results. It outlines a protocol for conducting systematic reviews, the information that should be included in each section of the resulting manuscript, and provides a diagram template for reporting of study inclusion procedures. Many journals will encourage or require that systematic reviews are conducted and reported in line with the PRISMA guidelines.

For general information on conducting systematic reviews and meta-analyses:

• Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020). Cochrane, 2020. Available from http://www.training.cochrane.org/handbook.
• Siddaway, A. P., Wood, A. M., & Hedges, L. V (2019). How to do a systematic review: A best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annual Review of Psychology, 70, 747-770.
• Uman, L. S. (2011). Systematic reviews and meta-analyses. Journal of the Canadian Academy of Child and Adolescent Psychiatry, 20, 57-59.
• Zelinksky, N. A. M. & Shadish . W. (2016). A demonstration of how to do a meta-analysis that combines single-case designs with between-groups experiments: The effects of choice making on challenging behaviors performed by people with disabilities. Developmental Neurorehabilitation, 21, 266-278. https://doi.org/10.3109/17518423.2015.1100690

For information on methodological considerations for systematic reviews:

• Ahn, S., Ames, A. J., & Myers, M. D. (2012). A review of meta-analyses in education: Methodological strengths and weaknesses. Review of Educational Research, 82, 436-476. https://doi.org/10.3102/0034654312458162
• Hohn, R. E., Slaney, K. L., & Tafreshi, D. (2019). Primary study quality in psychological meta-analyses: An empirical assessment of recent practice. Frontiers in Psychology, 9, 1-15. https://doi.org/10.3389/fpsyg.2018.02667

For information on examining and addressing publication bias in meta-analyses, see:

• McClain,M. B., Callan, G., Harris, B., Floyd, R. G., Golson, M. E., Haverkamp, C. R., Longhurst, D. N., Benallie, K. J. (2021). Methods for addressing publication bias in school psychology journals: A descriptive review of meta-analyses from 1980 to 2019. Journal of School Psychology, 84, 74–94.