Some feature questions about php-spider

[danvuquoc]

@koolma, I'm not a contributor or maintainer of this project but I have used this spider.

1. It doesn't support javascript, this isn't a headless browser. It simply uses guzzle http 6 to grab pages. Typically you then use a css/xpath discovery class to find new uris to find more pages to crawl.
2. It doesn't follow robots.txt files, but you could easily write a filter that implements the PreFetchFilterInterface and uses tomverran/robots to match the uri path about to be grabbed by the spider against the robots.txt. I've done this as well with robots headers and robots meta tags implementing a PostFetchFilterInterface.

I'm not sure about 3 and 4, hope this helps.

[solverat]

3. no, not without any additional work
4. yes, just implement some middleware

[mvdbos]

Good answers by @danvuquoc and @solverat. Closing this ancient issue. :-)