atomicptr / crab Goto Github PK

When running crab crawl:sitemap https://www.domain.com/sitemap.xml --header user-agent=some-whitelisted-user-agent the request for the sitemap doesn't use the user agent header.

A few months ago the snap part of the pipeline broke and I haven't been able to get it working again since then so I decided to remove snap support for now (might still push updates manually we'll see...).

Since I don't use Ubuntu nor Snap my interest in this is also kinda low anyway. If someone wants to help fix this I'm open to it.

Preferably as some sort of integration with the vegeta load testing tool.

$ crab crawl:sitemap https://domain.com/sitemap.xml --output output.log
# NO prints here...

It might probably make sense to have a different option for persisting the logs as a valid json file...

$ crab crawl:sitemap https://domain.com/sitemap.xml --output-json output.json
$ cat output.json
[
    {"status": 200, "url": "https://domain.com/page-1", ...},
    ...
]

It should be possible to filter crawl logs, depending on what I'm doing I might just want to look for non 200 pages

An idea on how this could look like:

$ crawl crawl:sitemap https://domain.com/sitemap.xml --filter-status=!200
# Only lists results that are not 200...