Returns disallowed paths from robots.txt found on your target domain and snapshotted by the Wayback Machine.
Inspired by mhmdiaa/waybackrobots.py
go install github.com/vodafon/waybackrobots@latest
Or
go get -u github.com/vodafon/waybackrobots
waybackrobots -d target.com
And you can use the -raw
flag to print the robots files as-is.
waybackrobots -d target.com -raw