Introduction
Technology Assisted Reviews (TAR) is a relatively new computer science field that employs information retrieval, machine learning and natural language processing (NLP) techniques, to reduce the workload on screening for Systematic Literature Reviews (SLRs). SLRs seek to collect evidence from research publications that fit strict pre-specified eligibility criteria to answer a specific research question. They aim to minimize bias by using explicit, systematic methods documented in advance with a protocol. The preparation stage includes the tasks of developingthe research question, searching for relevant SLRs, writing the needed protocol, and defining a search strategy. The search strategy includes Boolean queries adapted for each medical database. These Boolean queries typically have very complicated syntax and are usually built by highly trained information specialists. The queries are submitted to medical databases during the retrieval stage, resulting in a vast set of possibly relevant studies. Subsequently, every study is screened in the appraisal stage, using the title and abstract (abstract-level assessment), and irrelevant studies are removed. Additional assessment is conducted based on the full-text (content-level assessment) of the remaining studies. In the last stage, data are extracted, converted, and synthesized from the relevant studies. The final scientific paper incorporates all this data with the addition of a meta-analysis of the included studies.
Current methods of creating an SLR are labor-intensive and time-consuming, involving large monetary costs. On average, an SLR costs approximately more than $140K and the time a scientist spends to complete it is 1.72 years. The most cost-intensive and time-consuming part when creating a systematic review is the screening process, i.e. the appraisal stage. On average, more than 5,500 documents are returned from the databases and less than 4.7% (3%) of them are relevant at the abstract (document) level.
Our Contribution
The primary contribution of this work is a novel method for TAR with state-of-the-art results. Our method implements a full pipeline from protocol to screening papers with state-of-the-art results, assisting the researcher in three parts of the process of preparing an SLR: the preparation, the retrieval, and the appraisal. The key novel characteristics of our method are:
- it relies solely on the protocol of a systematic review, overcoming the need for constructing a specialized Boolean query (preparation).
- it combines an inter-review learning-to-rank method with an iterative intra-review relevance feedback mechanism (retrieval and appraisal)
- it incorporates domain specialized features deriving from the latest advances of the NLP field (retrieval and appraisal).
As a secondary contribution, we publish an updated version of the Task II of the CLEF 2019 eHealth lab dataset. The updated dataset fixes previous format issues and provides an up-to-date version of the dataset that includes the latest revisions of the included SLRs.
Achievements
Our methods achieved the second best in result in Task II of CLEF eHealth lab in 2017 and 2018.