Authors:
Shervin Daneshpajouh
1
;
Mojtaba Mohammadi Nasiri
1
and
Mohammad Ghodsi
2
Affiliations:
1
Sharif University of Technology, Iran, Islamic Republic of
;
2
Sharif University of Technology; School of Computer Science, Institute for Studies in Theoretical Physics and Mathematics, Iran, Islamic Republic of
Keyword(s):
Crawling, Communities, Seed Quality Metric, Crawl Quality Metric, HITS, Web Graph, Hyperlink Analysis.
Related
Ontology
Subjects/Areas/Topics:
Artificial Intelligence
;
Knowledge Discovery and Information Retrieval
;
Knowledge-Based Systems
;
Searching and Browsing
;
Social Information Systems
;
Society, e-Business and e-Government
;
Soft Computing
;
Symbolic Systems
;
Web Information Systems and Technologies
;
Web Interfaces and Applications
;
Web Mining
Abstract:
In this paper, we present a new and fast algorithm for generating the seeds set for web crawlers. A typical crawler normally starts from a fixed set like DMOZ links, and then continues crawling from URLs found in these web pages. Crawlers are supposed to download more good pages in less iterations. Crawled pages are good if they have high PageRanks and are from different communities. In this paper, we present a new algorithm with O(n) running time for generating crawler's seeds set based on HITS algorithm. A crawler can download qualified web pages, from different communities, starting from generated seeds set using our algorithm in less iteration.