Would be awesome to apply a selector to limit scope of crawled links

[duggi]

for example:

crawler.crawl({
  url: "http://localhost:8080/locations/",
  selector: ".main-content"

would only follow the links found inside .main-content

this way i don't have to keep crawling the header, footer, sidebars, etc on every page

---

thank you for writing this!

[amoilanen]

Hi,

It can be an interesting feature, the only problem is that at the moment the crawler does not deal with the page content as DOM, it is just a text content. But maybe we can limit the section of the page which should be crawled in some other way. I will investigate this a bit more.