iVia

The Expert-Guided Crawler with Drill-down

The Expert-Guided Crawler with Drill-down is an iVia component that is used to by indexers to explore Web sites and discover new Internet resources. It is used in many iVia applications.

Applications

The INFOMINE Virtual Library Crawler uses the Expert-Guided Crawler to explore each virtual library and return the contents of each library.

The Expert-Guided Crawler with Drill-down and Drill-out tool in the iVia Adders' interface lets iVia users search a Web site for useful resources, and then automatically create records for them and add them to the iVia database..

The Expert-Guided Crawl Service in the Remote iVia Service Interface lets people (and computer programs) use the Expert-guided Crawler with Drill-down to crawl a Web site, discover the useful resources on that Web site, and create a set of iVia records that describe the discovered resources.

Technical details

The Expert-Guided Crawler is a component in the libiViaCore library, which is part of the iVia Software available from our download page.

The crawler is asked to crawl a site, starting from a specified URL, and to discover resources on that site that match specific types (e.g. PDF files, HTML pages, JPG images) and locations (e.g. on-site resources, off-site resources). There are numerous options that can be set by the user, including logging options, email notification options, and limits to the length, depth and duration of the crawl.