avilum / smart-url-fuzzer

Explore URLs of domains fast and efficiently using fuzzing techniques

What's in this website?

An asynchronous, robust website URLs explorer
Highly configurable, yet fast and simple.
I developed this for pentesting purposes.
nmap's http module does quite the same, but with a lot of dependencies. Companies provide this service for money, so I decided to do it on my own.
Please use it responsibly.

Examples

git clone [email protected]:avilum/smart-url-fuzzer.git && cd smart-url-fuzzer
pip install -r requirements.txt

# This will find all the active endpoints in https://www.example.com
$ ./fuzz

Custom words lists

$ # Or, Using Python
$ python fuzz.py --help
Usage: fuzz.py -u https://example.com/

An Asynchronous, robust websites endpoint discovery tool with smart error
handling. Locates resources in websites based on a list of paths. Check out
the "words_list"" directory for lists examples.

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -u BASE_URL, --url=BASE_URL
                        The target website to scan.
  -l LIST_FILE, --list=LIST_FILE
                        A file containing the paths to check (separated with
                        lines).

You can use a custom paths lists, based on the website type, or based on your needs.
The directory 'words_lists' contains a some lists of the most common endpoints.

$ python fuzz.py -u https://www.facebook.com -l words_lists/list-php.txt
####-##-## ##:##:##,### - fuzzing - INFO - Getting the endpoints of the website https://www.facebook.com with list file "words_lists/list-php.txt" and 100 async workers.
# ...
https://www.facebook.com/comment_edit.php : 200
https://www.facebook.com/webdb_view_test.php : 200
https://www.facebook.com/sp_feedgenerator.php : 200
https://www.facebook.com/xp_publish.php : 200
https://www.facebook.com/categories_0222.php : 200
https://www.facebook.com/3d_exhibits1.php : 200
https://www.facebook.com/adr_cell.php : 200
####-##-## ##:##:##,### - fuzzing - INFO - The endpoints were exported to "endpoints.txt"

Workers

If the fuzzing failed for any http reason, it continues with less workers automatically.
Some sites have DDOS protection mechanisms.
The fuzzer will reach the optimal number of workers automatically, without getting blocked.

Logs

All the activity is logged under /logs folder by default.

smart-url-fuzzer's Issues

suggestions

scan sitemap.xml
make an option to start the URL fuzzing with a set amount of async workers, -r 20 to start with 20 workers and if that doesn't work because of a DDoS engine or 429, it will automatically go down to 10 async workers. If you could set 25 instead of limits in 10's like 10, 20, 30, if it were set to 25 make it jump down to the next lowest one which would be 20 on the default program from 30 async workers except now cause of custom limit it's 25.
make an option to decline the async workers instead of in 10's from 100 to 90 etc add an option to make them go down in a custom set interval of like 2 workers or 5 so on...

error executing?

# python -V Python 2.7.17

/smart-url-fuzzer# python fuzz.py -u https://www.facebook.com -l words_lists/list-php.txt Traceback (most recent call last): File "fuzz.py", line 8, in <module> import requests ImportError: No module named requests

Use proxy/vpn

Its possible to use a proxy underneath the fuzzer, so every request will be sent from a different ip and we won't get blocked.
Something like luminaty but free, that can be installed on every computer.

No Subdomains fuzzing

Its possible to get the subomains of the website with reverse ip engines, and fuzz the urls as well.

Add more common urls lists.

For APIs (/user,/admin,/photos,/upload and such).

Automatic seever recognition

Add automatic server recognition and choosing the words list accordingly.

Scan robots.txt

Make a set of all the endpoints in robots.txt, and add them to the active endpoints.