Wombat is a scraper. I don't think it is a crawler. about wombat

[felipecsl]

@bluemont Correct, right now, wombat definitely would be better classified as a scraper, and not as a crawler, since it is not able to navigate through pages. I have plans to add some sort of crawling functionality in the future, but for the time being, I am gonna update the documentation to make it clearer. Thanks for the feedback!

[felipecsl]

Just released version 2.0.0 which addresses this issue mainly via these 2 changes:

• Created new syntatic sugar method Crawler#scrape that aliases to Crawler#crawl;
• Added the ability to follow links via the :follow option in a property. Gonna add some documentation on how it works soon. For now, it will only crawl 1 level deep by default, which means it won't keep clicking links. Next release will add the ability to specify a custom depth.

I think these changes addresses the issue. Closing it.

[StephenOTT]

@felipecsl was depth control ever added?

[felipecsl]

@StephenOTT not yet. It sounds like it could use that feature though. I'll see if I can get it implemented for next version

[StephenOTT]

I am looking to recreate the recursive functionality that FMiner provides:
http://www.fminer.com

The Ability to CSS or Xpath some links on the page and continue to follow/iterate through each of the pages. On each page the user should be able to capture information. Common examples of pages with "Next" Buttons or Hierarchical based data structures that are spread over multiple pages.