CMSRU Evidence Synthesis Service: Steps in a Systematic Review

Before starting your review, determine if a systematic review is the best approach to answer your question. (What are other types of reviews to consider?)

• Will it fill a meaningful gap in existing literature?
• Is there already an existing review? 
  • If you identify an existing review, assess its quality. (see box below)
  • If a high quality review exists but was completed several years ago, a new review may be justified.
• Do you have the necessary time and resources to complete a systematic review?

One of the features that distinguishes a systematic review from a narrative review is the pre-specification of criteria for including and excluding studies in the review (eligibility criteria). Explicit criteria, based on the review’s scope and question(s), are used to include and exclude studies. 

A large number of references (study titles and abstracts) will have been found at the searching stage of the review. A proportion of these will look as though they are relevant to the review's research questions. So, having explicit criteria against which to assess studies makes the process more efficient in terms of time.

More importantly, it also helps to avoid hidden bias by having clear consistent rules about which studies are being used to answer the review's specific research questions. 

Each study needs to be compared against same criteria. To be included in the review, a study needs to meet all inclusion criteria and not meet any exclusion criteria. Inclusion/eligibility criteria iclude participants, interventions and comparisons and often study design. Outcomes are usually not part of the criteria, though some reviews do legitimately restrict eligibility to specific outcomes.

The review protocol sets out the methods to be used in the review and provides an explicit plan for your work. Decisions about the review question, inclusion criteria, search strategy, study selection, data extraction, quality assessment, data synthesis and plans for dissemination should be addressed. If modifications to the protocol are required, these should be clearly documented and justified. Modifications may arise from a clearer understanding of the review question, and should not be made because of an awareness of the results of individual studies. 

Consider registering your protocol. PROSPERO, from the Centre for Reviews and Dissemination at the University of York, is an international database of registered reviews in health and social care. Key features from registered reviews and recorded and maintained as a permanent record. Registration helps to promote transparency in the review process and also reduces the potential for duplication.

A systematic search to identify studies must be comprehensive and it must strike a balance between recall and precision. In other words, don't expect to retrieve only relevant articles.

Most studies used in the review will be identified using electronic databases (e.g. PubMed), but identifying unpublished studies is important as well. Librarians can help to identify sources for finding grey literature.

Expect a large number of results.  Depending on the scope of your topic, your search may result in as few as a couple hundred or as many as several thousand articles.

Some key steps in searching for studies include:

• Identifying databases to be searched.
• Identifying search terminology.
• Constructing and running database searches.
• Conducting hand searches of specialized journals.
• Searching reference lists of relevant studies.
• Contacting recognized experts working in the field.
• Searching relevant grey literature sources (e.g. clinical trials registers, conference proceedings).

It’s important to note that the IOM recommends working with a librarian to plan and execute your searches. Plan to meet with a librarian to discuss your review.

A large quantity of studies needs to be assessed at this stage of the review.  This process should involve at least two members of your group to help reduce bias.  It is useful to develop an appropriate form to help select and keep track of articles that meet eligibility criteria. To be included, a study must meet all inclusion criteria and not meet any exclusion criteria. 

Employ the following process:

1. Examine titles/abstracts
2. Retrieve the full-text of relevant studies
3. Examine the full-text to determine eligibility
4. Make final decisions on study inclusion

Remember to document your decisions!  It is important to have a record of decisions made in order to ensure reproducibility and minimize errors.

This PRISMA flow diagram shows the number of studies remaining at each stage. It's a simple and useful way of documenting the study selection process and should be included in your final report.

Here, data simply refers to information about or details from a study including its methods and design, participants, setting, interventions, and results. Two researchers should independently extract data from each study in order to minimize errors and reduce the potential for bias. At a minimum, one researcher should extract data with a second person checking for accuracy.  

To standardize the process and improve the validity of the results, time and thought should be given to creating a data extraction form. These can be created using general word processing software or spreadsheets, or you can use the data extraction features in Covidence.

It is important to collect enough information during this process to sufficiently assess each study. Collecting too much or too little information may be a waste of time and result in the omission of crucial data.

Assessment of study quality gives an indication of the strength of the evidence provided by your review. It helps determine whether the studies are vigorous enough to guide treatment, prevention, diagnostic or policy decisions.

Quality assessment of any study is likely to consider:

• Appropriateness of study design to the research objective
• Risk of bias
• Other issues related to study quality
  • Choice of outcome measure
  • Statistical issues
  • Quality of reporting
  • Quality of the intervention
  • Generalizability

From: Centre for Reviews and Dissemination, University of York (2008) Systematic Reviews: CRD's Guidance for Undertaking Reviews in Health Care.

Synthesis involves combining the results of the studies included in the review, summarizing their findings and drawing reliable conclusions based on the quality of the evidence. Synthesis may be done quantitatively using statistical techniques, such as a meta-analysis, or through a narrative approach.  

In general, making recommendations for practice does not fall within the purview of systematic reviews. This is typically the domain of clinical practice guidelines. Systematic review authors can make conclusions about the need for further research or draw conclusions about the usefulness of an intervention.

When preparing your report or article, refer to the PRISMA Checklist.  Many commissioning bodies and journals have adopted PRISMA as the required methods for reporting systematic reviews.