Holger's Java API

## com.antelmann.net
Interface CrawlerSetting

All Known Implementing Classes:
  MailScanner, MediaCrawler, SampleCrawlerSetting

    public interface CrawlerSetting

CrawlerSetting defines callback functions that determine the behavior in which a web search algorithm goes through the net and calculates its results. A CrawlerSetting can be used with a com.antelmann.net.Spider.

Author:
  Holger Antelmann
See Also:
  `Spider`, `Spider.crawlWeb(CrawlerSetting, int, com.antelmann.util.logging.Logger)`

Method Summary
`boolean` `followLinks(URL url, URL referer, int depth, List<URL> resultURLList, List<URL> closedURLList, List<Spider.URLWrapper> searchURLWrapperList)` followLinks() determines whether the given URL is to be searched for its links to be examined further in the next level.
`boolean` `matchesCriteria(URL url, URL referer, int depth, List<URL> resultURLList, List<URL> closedURLList)` This method decides whether either the URL itself or its content qualifies for what this CrawlerSetting searches for; as this function is also called on every URL encountered, it is also the place for any custom parsing this CrawlerSetting wants to do.

Method Detail

### matchesCriteria

    boolean matchesCriteria(URL url,
                            URL referer,
                            int depth,
                            List<URL> resultURLList,
                            List<URL> closedURLList)

  This method decides whether either the URL itself or its content qualifies for what this CrawlerSetting searches for; as this function is also called on every URL encountered, it is also the place for any custom parsing this CrawlerSetting wants to do. The two List objects allow the CrawlerSetting to act on potential constrains that may result from e.g. a maximum number of total nodes to be examined (or any other custom checking imaginable). Note that it is the responsibility of the calling object to ensure that this function isn't called multiple times on the same URL if that's not desired. The url may include any URL, including non-HTTP protocols (such as mailto:, ftp:) and image or media URLs

  Parameters:
    `url` - the URL in question to satisfy the criteria
    `referer` - url's referer URL
    `depth` - link distance from the original root URL where the search began
    `resultURLList` - List of URLs that have already been found to match this CrawlerSetting's criteria
    `closedURLList` - List of URLs that have already been found not to match the CrawlerSetting's criteria

### followLinks

    boolean followLinks(URL url,
                        URL referer,
                        int depth,
                        List<URL> resultURLList,
                        List<URL> closedURLList,
                        List<Spider.URLWrapper> searchURLWrapperList)

  followLinks() determines whether the given URL is to be searched for its links to be examined further in the next level. The three List objects allow the CrawlerSetting to act on potential constrains that may result from e.g. a maximum number of total nodes to be examined (or any other custom checking imaginable). The url may include any URL, including non-HTTP protocols (such as mailto:, ftp:) and image or media URLs.

  Parameters:
    `url` - the URL that is to be examined for its links
    `referer` - url's referer URL
    `depth` - distance from the original root URL where the search began
    `resultURLList` - List of URLs that have already been found to match this CrawlerSetting's criteria
    `closedURLList` - List of URLs that have already been found not to match the CrawlerSetting's criteria
    `searchURLWrapperList` - List of Spider.URLWrapper objects already identified to be examined in the next level
  See Also:
    `Spider.URLWrapper`

(c) Holger Antelmann since 2001- all rights reserved (contact: info@antelmann.com)
see www.antelmann.com/developer for further details and available downloads