Assessing the quality of included studies

Having decided which articles are relevant to your review it is now time to assess their quality; their quality will indicate how much weight can be given to the results they contain.

Ask the following sorts of questions:

  • Was the population forming the subject of the study appropriately chosen?
  • Were the research methods used, and the methods of data analysis, appropriate?
  • Was the research designed, conducted and the results analysed in a way which minimised bias?
  • Are the results generalisable?
  • Are there any ethical doubts about the research?

Please refer to our Critical Appraisal guide for more detailed information. The key tools section provides links to a number of highly regarded resources including the Joanna Briggs Institute critical appraisal tools and the Critical Appraisal Skills Programme (CASP).  Both of these resources provide checklists for different types of research including randomised controlled trials (RCTs) and qualitative studies.  Each checklist provides a list of questions to consider when analysing the quality of a particular type of study.

The Centre for Reviews and Dissemination's guidance for undertaking reviews (2009) includes a useful section on assessing quality (see section 1.3.4).

Guidance on appraisal and bias from systematic review bodies

Members of staff conducting systematic reviews may have access to particular tools from sponsoring bodies to assist them with the critical appraisal, aggregation and synthesis of literature.

The Cochrane Handbook provides detailed information on assessing the risk of bias in included studies in chapter 8.

As well as the Joanna Briggs Institute critical appraisal tools, reviewers for the Joanna Briggs Institute may also have access to SUMARI (System for the Unified Management, Assessment and Review of Information) the JBI's premier software for the systematic review of literature.

Extracting data

The Centre for Reviews and Dissemination's guidance (section 1.3.3) contains helpful information on extracting data from studies in preparation for writing the review. 

The Cochrane Handbook (Table 7.3.a) provides a Checklist of the sort of data you may wish to record including:

  • Citation details (author, title, source, year etc.)
  • Study design (as appropriate to the review but may include blinding)
  • Number and types of participants (population) and sample size
  • Interventions
  • Results

It is useful to set up a form or table to record the relevant data types for your particular review.

Keeping all key information for all studies included in the review together can be extremely useful and timesaving when you begin writing up.