Combining Inter-Review Learning-to-Rank and Intra-Review Incremental Training for Title and Abstract Screening in Systematic Reviews

A. Anagnostou, A. Lagopoulos, G. Tsoumakas, I. Vlahavas (2017) Combining Inter-Review Learning-to-Rank and Intra-Review Incremental Training for Title and Abstract Screening in Systematic Reviews, eHealth Lab of the 8th Conference and Labs of the Evaluation Forum (CLEF)

Antonios Anagnostou, Athanasios Lagopoulos, Grigorios Tsoumakas, and I. Vlahavas

We describe the approach we employed for Task II of CLEF eHealth 2017, concerning title and abstract screening in diagnostic test accuracy reviews. Our approach combines a learning-to-rank model trained across multiple reviews with a model focused on the given review, incrementally trained based on relevance feedback. Our learning-to-rank model is built using extreme gradient boosting on features computed by considering the similarity of different fields of the documents (title, abstract), with different fields of the topics (title, query). Our incrementally trained model is a support vector machine trained on a TF-IDF representation of title and abstract of the documents. The results of our approach are promising, reaching 0.658 normalized cumulative gain in the top 10 ranked documents in the simple evaluation setting and 0.846 in the cost-effective evaluation setting, the latter assuming feedback can be obtained from an intermediate user/oracle instead of the end-user.