How Web Crawlers Work

 linklicious free version information