A systematic review must incorporate a search for unpublished materials as well as published materials. This mitigates against the risk of "publication bias" (studies reporting positive effects of treatment or successful therapies may be more likely to be published than those which report negative effects).
Grey literature is literature which is not formally or commercially published. It includes, but is not limited to:
- reports, working papers, documents, standard operating procedures etc. produced and published by government agencies, academic institutions and other groups that are not distributed or indexed by commercial publishers
- conference papers or posters
- theses and dissertations
- technical notes
- unpublished research reports
By its very nature grey literature can be hard to trace. Try searching these resources:
- General: OpenGrey
- University theses, working papers, research reports: OpenDOAR ; Open Access Theses and Dissertations ; OAIster ; ETHoS (British Library)
- Government reports etc.: Gov.UK ; Scottish Government
- Health (general): NICE Evidence Search ; Trip (Turning Research Into Practice)
- Clinical Trials: ISRCTN Registry ; Be Part of Research ; ClinicalTrials.gov (US);
- Social care: Social Care Online
For additional resources please see our Guide to Grey Literature.
What about Google ot Google Scholar?
While not to be used for literature searching, a Google search can be used to find relevant organisations which may be conducting or sponsoring research as yet unpublished and which may be sources of grey literature. Try other search engines like dogpile and DuckDuckGo.
Using the Advanced search on Google and limiting your results to PDF files may find useful reports.
In all cases check the information comes from a reputable body.
Google Scholar limits your results to scholarly material and may produce details of working papers, theses and some reports not seen elsewhere.
Handsearching involves "scanning the content of journals, conference proceedings and abstracts, page by page" (Centre for Reviews and Dissemination 2009 p. 18).
Databases are not infallible; items like letters, editorial notes, comments, opinion pieces, brief details of ongoing trials or studies may not always be added. Even if they have been added it may not be easy to find them by keyword searching (and they may not have been assigned suitable subject index terms). Searching the journals which have most frequently appeared in your results lists by hand will ensure that nothing has been missed in these journals.
Some relevant journals may not be indexed on databases. Hand searching such journals could uncover research not reported elsewhere.
Contacts and recognised experts
Academic staff conducting fully comprehensive systematic reviews will wish to contact experts in the field of the review to see if they can provide details of any ongoing or unpublished trials or studies.
The Cochrane Handbook states:
"Colleagues can be an important source of information about unpublished studies, and informal channels of communication can sometimes be the only means of identifying unpublished data. Formal letters of request for information can also be used to identify completed but unpublished studies. One way of doing this is to send a comprehensive list of relevant articles along with the inclusion criteria for the review to the first author of reports of included studies, asking if they know of any additional studies (published or unpublished) that might be relevant. It may also be desirable to send the same letter to other experts and pharmaceutical companies or others with an interest in the area" (Cochrane Collaboration 2011 para. 6.2.3.).
Some systematic review teams set up websites with details of articles they have already found and invite comment from interested parties; others may use social media. It may also be worth seeking details of any unpublished and ongoing trials by sending appropriate messages to professional email lists.
Searching reference lists
Once you have located relevant articles look through the list of references at the end of these articles - they may contain details of older relevant articles not found in your keyword search.
Some databases will give "Times Cited" or "Cited by" information for articles in your results lists. Click on one of these links to view details of newer articles which have included that article in their own reference lists; these newer articles may also be relevant to your search.
Web of Science database includes three citation indexes. Entering details of potentially important articles found elsewhere may produce details of additional articles citing those articles on the topic of your review.