Authors:
Ari Pirkola
and
Tuomas Talvensaari
Affiliation:
University of Tampere, Finland
Keyword(s):
Focused web crawling, Genomics, Genetics.
Related
Ontology
Subjects/Areas/Topics:
Artificial Intelligence
;
Biomedical Engineering
;
Business Analytics
;
Cardiovascular Technologies
;
Computing and Telecommunications in Cardiology
;
Data Engineering
;
Data Mining
;
Databases and Information Systems Integration
;
Datamining
;
Enterprise Information Systems
;
Health Engineering and Technology Applications
;
Health Information Systems
;
Medical and Nursing Informatics
;
Sensor Networks
;
Signal Processing
;
Soft Computing
Abstract:
A focused crawler is a program that fetches Web pages that are relevant to a pre-defined domain. In this paper we consider focused crawling in the domains of genomics and genetics. Crawling is often started with seed URLs that point to central North-American and European universities, research institutions, and other organizations in North-America and Europe. We investigate how strongly this central region of the Web is connected to other large geographical regions of the Web: Australia (top level domain .au), China (.cn), and five South-American countries (.ar, .br, .cl, .mx, and .uy). We consider what implications the observed global link structure has for the selection of seed URLs for focused crawling. The results showed that the proportion of out-links from the North-American and European region to the other regions is low whereas pages in the other regions often point to the central region. We also found that two focused crawling processes, one started from the central region a
nd the other from another large region, overlap only to a small extent. Overall, the results suggest that the effectiveness of focused crawling can be improved considerably if crawling is started with a geographically heterogeneous seed URL set.
(More)