Skip to Main Content

Evidence Synthesis and Systematic Review

Steps in an evidence synthesis

The importance of preparing a protocol for evidence synthesis
  • An evidence synthesis protocol outlines the rationale, hypothesis, and planned methodology, providing a structured framework for conducting the review.
  • The protocol serves as both a planning document and a roadmap, enabling efficient and accurate execution of the systematic review while offering team members a clear reference throughout the research process. This structured approach also streamlines the manuscript writing process.
  • Many journals now require registered protocols to accompany submitted systematic reviews.
  • The PRISMA Reporting Standard identifies information about the systematic review protocol as an "essential element".
  • The Cochrane Handbook, the Institute of Medicine Standards, and other reputable organizations emphasize the completion of a protocol as a crucial step for a successful systematic review.

The protocol should include
  • The rationale for the review
  • Key questions broken into PICO components (See "3. Develop Your Research Question")
  • Inclusion/exclusion criteria
  • Literature searches for published / unpublished literature
  • Research data management
  • Assessment of methodological quality and risk of bias
  • Data synthesis
  • Grading the evidence for each key question

Resources for authors preparing a protocol for a systematic or scoping review

Protocol templates

 

Where to register protocols?

Cochrane

As you get ready to propose a Cochrane Review, you can access many of the resources available to Cochrane authors. Use these free resources (no login required) to understand Cochrane methods and to improve your review proposal.

Disciplines: Healthcare

Campbell Collaboration

Published on behalf of the Campbell Collaboration, Campbell Systematic Reviews is an open access journal publishing systematic reviews, evidence and gap maps, and methods research papers.

Disciplines: Multidisciplinary - including Ageing; Business and Management; Children and Young Persons Wellbeing; Climate Solutions; Crime and Justice; Disability; Education; International Development; Knowledge Translation and Implementation; Methods; Sexual Orientation and Gender Identity; and Social Welfare.

PROSPERO

PROSPERO is an international systematic review registry that aims to promote transparency and open science, reduce reporting bias and help prevent unintended duplication and research waste. The PROSPERO database currently includes records of over 328000 prospectively registered systematic reviews with health related outcomes

Disciplines: Health and Social Care, Welfare, Public Health, Education, Crime, Justice, and International Development

 

 

 

 

Develop your research question 

Developing your research question is one of the most important steps in the evidence synthesis process. At this stage in the process, you and your team have identified a knowledge gap in your field and are aiming to answer a specific question:

  • If (intervention/ medication) is prescribed, then (outcome/ effect) will happen to patients?

OR assess an intervention:

  • How does (intervention/ medication) affect (outcome/ effect)?

OR synthesize the existing evidence

  • What is the nature of (intervention/ medication)? ​

​​Whatever your aim, formulating a clear, well-defined research question of appropriate scope is key to a successful evidence synthesis. The research question will be the foundation of your synthesis and from it your research team will identify 2-5 possible search concepts. 

Research question frameworks

A research question framework can help formulate and structure your evidence synthesis question. There are many research question frameworks, such as PEO, PICO, SPIDER, SPICE, and ECLIPS, that can assist in formulating a focused research question. Check out this table featuring various frameworks, along with their definitions and examples, from the Medical University of South Carolina (MUSC) Libraries.

The PICO question format is useful for clinical and quantitative research topics, such as therapy, diagnosis, prognosis, etiology/harm, and prevention questions.

P Patient / Population / Problem
I Intervention / Exposure
C Comparison (optional)
O Outcome

Research question: In infants diagnosed with necrotizing enterocolitis (NEC), what is the effect of early enteral refeeding on NEC recurrence compared with late enteral refeeding?

 

 

Source: Richardson, W. S., Wilson, M. C., Nishikawa, J., & Hayward, R. S. (1995). The well-built clinical question: A key to evidence-based decisions. ACP journal club, 123(3), A12-A12
Inclusion and exclusion criteria 

Inclusion criteria are the elements of an article that must be present in order for it to be eligible for inclusion in a review. 

For example, included studies must:

  • have compared certain treatments
  • be experimental or observational or both
  • have been published in a certain timeframe (must have compelling reason)
  • be certain publication type(s)  (e.g. journal articles)
  • have minimum sample size

Exclusion criteria are the elements of an article that disqualify the study from inclusion in a review.  

For example, excluded studies: 

  • used qualitative methodology
  • used a certain study design (e.g. observational)
  • of a certain publication type (e.g. systematic reviews)
  • were published before a certain year (must have compelling reason)
  • used animal models
  • which was published in a language other than English
Dive deeper into your chosen topic using AI tools

  • Utilize the "research assistant" in databases (e.g. Web of Science and LibSearch) to delve deeper into your chosen topic.

  • Follow "guided tasks" to gain a clearer understanding of your subject and to collect and assess relevant references for your research.

  • Explore "topic maps" to spark inspiration and refine your research question effectively.

Each database functions differently, primarily focusing on published, peer-reviewed literature. A systematic review  requires a thorough and methodical search to include all relevant evidence. Selecting the appropriate databases is crucial, and you should document both the chosen databases and your search strategies in your review to ensure transparency and replicability.

Library subscribed databases

Free databases
  • PubMed Central (PMC)
    PMC is an electronic archive of full-text journal articles, offering free access to its contents. PMC contains more than 5 million articles, most of which have a corresponding entry in PubMed.

 

For reference: 

Gusenbauer M, Haddaway NR. Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Research Synthesis Methods. 2020; 11: 181–217. https://doi.org/10.1002/jrsm.1378

Grey (or gray) literature refers to materials produced by individuals or organizations outside traditional commercial and academic publishing channels. This category includes government reports, conference proceedings, graduate theses, unpublished clinical trials, technical reports, and more.

In evidence synthesis, the goal is to integrate all relevant evidence pertaining to your research question. It’s important to recognize that scientific publishing often favors studies demonstrating significant effects, leading to a publication bias. Consequently, many studies and trials that report no effect remain unpublished. However, understanding that an intervention has no effect is equally crucial for informed decision-making in practice and policy.

While grey literature is not peer-reviewed, it constitutes a valuable source of information that can fill gaps left by published studies. Incorporating grey literature into your review enriches the evidence base, providing a more comprehensive understanding of the topic and facilitating better-informed conclusions.

Theses & dissertations

Clinical trials

Other online resources
  • MedNar
    A free, medical-focused deep web search engine.
  • OpenGrey
    A multidisciplinary European database covering science, technology, biomedical science, economics, social science and humanities. 
  • Global Index Medicus
    Collated and aggregated by WHO Regional Office Libraries, provides worldwide access to biomedical and public health literature produced by and within low-middle income countries.
  • WHO Library Database
    It provides access to knowledge, including governing documents, reports and technical documentation, from WHO as well as from other sources of scientific literature produced around the world.

 

Maintain a balance
The goal of systematic review searches is to identify all relevant studies on a topic. While these searches should be extensive, it's essential to balance comprehensiveness with relevance in your search strategy. Additionally, ensure that searches are well-documented and reproducible.

Considerations for creating a systematic review search
  • Include all major concepts in the search strategy
  • Use appropriate controlled vocabulary terms for each database, e.g. MeSH (Medical Subject Headings) and Emtree (See note below)
  • Employ explosion, subheadings, and floating subheadings effectively.
  • Incorporate natural language (text words) alongside controlled vocabulary.
  • Utilize relevant synonyms and acronyms.
  • Apply truncation and spelling variations as necessary.
  • Use limits and filters appropriately.
  • Apply Boolean operators correctly.
  • Verify indexing of exemplar articles.
  • Adapt the search strategy for at least three different databases.
  • Include grey literature sources

 

Note: What are the differences between Emtree and MeSH? --  Emtree and MeSH are both comprehensive biomedical and life science thesauri, respectively used to index the biomedical literature in Embase and MEDLINE.

Evidence synthesis methods require authors to search multiple databases, and not all databases accept the same search "syntax." Each individual database requires use of specialized search syntax, and therefore evidence synthesis search strategies must be 'translated' between databases. 

For example, a search for vitamin D[tiab] in PubMed will show you all citations with the phrase "vitamin D" in the title, abstract, or keywords, but a search for vitamin D[tiab] in Web of Science will not work at all. 

Tools for translating search strategies
  • Database Syntax Guide
    The document provides specific syntax for different databases such as Cochrane Library, CSA databases, EBSCOhost, Web of Science, LILACS, Ovid, POPLINE, ProQuest, and WHOLIS, detailing unique search commands and their functions.
  • MEDLINE Transpose
    Translate a search query between PubMed syntax and (1) MEDLINE (via Ovid) or (2) MEDLINE (via EBSCOhost)

A citation management tool will save you a lot of time when doing your evidence synthesis. Tools like Endnote Online, Mendeley or Zotero will store and organize the citations collected during your screening, de-duplicate the results and automatically format in-text citations and bibliographies in your project.

Check out this guide for Helps on Managing Citations 

Use EndNote Online to manage citations

Article screening : step-by-step

  

  • Define purpose - The goal is to remove studies that are clearly unrelated to your topic.

  • Establish inclusion/ exclusion Criteria - Create criteria to determine relevance to your research question.

  • Screen titles and abstracts - Review the titles and abstracts of the studies using your criteria to assess relevance.

  • Retrieve full texts - For studies that pass the initial screening, obtain the full texts for further evaluation.

  • Screen full texts - Assess the full texts to determine if the studies meet the eligibility criteria for your synthesis.

  • Involve independent reviewers - It is highly recommended that at least two reviewers conduct the screening.

  • Resolve disagreements - Address any areas of disagreement by reaching a consensus or involving a third-party.

  • Utilize screening tools - Consider using available tools for article screening to facilitate the process.

  

Tools for managing evidence syntheses

  
(A) Free tools
  
  • Rayyan is a free online tool that can be used for independent screening and coding of studies in an evidence synthesis. Rayyan will pre-populate inclusion and exclusion criteria, but you can customize these criteria. It also uses tagging and filtering to code and organize references.  Title and abstract screening can be conducted in one project, while full text screening can be conducted in a second project. 
  • CADIMA is a free web tool facilitating the conduct and assuring for the documentation of systematic reviews, systematic maps and further literature reviews. More information about CADIMA systematic review.
  • Colandr is a free, web-based, open-access tool for conducting evidence synthesis projects, such as systematic and scoping reviews. More information about Colandr systematic review.
     
  • Microsoft Excel is the most basic tool for the management of article screening. Lists of references can be exported from citation managers into Excel format for screening. 
  
(B) Tools that require a subscription
  
 

 

Why assess for bias and quality?

Conducting a risk of bias assessment - often referred to as quality assessment or critical appraisal - is a key aspect of the systematic review process. It enhances the methodological rigor and transparency of reported results and findings.

Tools for assessing bias and quality

Commonly extracted fields for most systematic reviews
  • Article citation and corresponding author
  • Study characteristics (e.g. study type)
  • Participant characteristics
  • Interventions and settings
  • Outcome data and results

Manuals that provide starting templates for data extraction

Writing your systematic review actually starts much earlier!

The majority of what you put into your review protocol can be used to jumpstart your article. This is part of why a high-quality protocol is so important. Below are important considerations:

  • Adhere to a reporting guideline - PRISMA is a common reporting standard for systematic reviews.
    (See "1. Prepare a protocol")
  • Check your protocol - it should include much of your introduction and a significant portion of your methods section
  • Team effort - make sure you include all team members in your systematic review and everyone contributes to the process
  • Document your searches - include a copy of all fully reproducible search strategies as supplementary material in your systematic review.

Resources to help you write