Reinforcement Learning with Classifier Selection for Focused Crawling

I. Partalas, G. Paliouras, I. Vlahavas, “Reinforcement Learning with Classifier Selection for Focused Crawling”, 18th European Conference on Artificial Intelligence, IOS Press, pp. 759-760, 2008.

18th European Conference on Artificial Intelligence, IOS Press, pp. 759-760, 2008.

Focused crawlers are programs that wander in the Web, using its graph structure, and gather pages that belong to a specific topic. The most critical task in Focused Crawling is the scoring of the URLs as it designates the path that the crawler will follow, and thus its effectiveness. In this paper we propose a novel scheme for assigning scores to the URLs, based on the Reinforcement Learning (RL) framework. The proposed approach learns an adaptive behavior of selecting the best classifier for ordering the URLs. This formulation reduces the size of the search space for the RL method and makes the problem tractable. We evaluate the proposed approach on-line on a number of topics, which offers us a realistic view of its performance, comparing it also with a RL method and a simple but effective classifier-based crawler. The results demonstrate the strength of the proposed approach.