EndPointer is a command-line tool designed to crawl a given URL and list all discovered endpoints. Utilizing the Colly web scraping library, EndPointer efficiently navigates through websites to provide a comprehensive list of endpoints, allowing for deeper insight into the structure and available resources of a site. This was made with a bounty in mind ๐ฐ
Prerequisite | Version |
---|---|
Go | <=1.22 |
apt install golang-go || brew install go
git clone https://github.com/CharlesTheGreat77/EndPointer
cd EndPointer
go mod init EndPointer
go mod tidy
go build -o endpointer main.go
To use EndPointer, run the compiled binary with the desired flags:
./endpointer -url https://example.com [options]
./endpointer -h
Usage of ./endpointer:
-custom-headers string
specify a file containing headers to send on request [separated by line]
-depth int
specify the max depth for crawling [default: 2] (default 2)
-h show usage
-proxies string
specify a file containing http/https/socks5 proxies [separated by line]
-threads int
specify the number of threads [default: 2] (default 2)
-timeout int
specify a timeout (seconds) [default: 3] (default 3)
-url string
specify a url [https://example.com]
-user-agent string
specify a custom user agent
โข URL Crawling: Specify a target URL to begin crawling.
โข Custom User Agent: Option to set a custom user-agent for requests.
โข Custom Headers: Load custom headers from a file to include in requests. (proxy rotation)
โข Proxy Support: Use a list of proxies (HTTP/HTTPS/SOCKS5) to send requests.
โข Thread Control: Set the number of concurrent threads for crawling.
โข Depth Control: Define the maximum depth for crawling.
โข Timeout Setting: Specify a timeout for requests to handle slow responses.
Example
./endpointer -url https://example.com -user-agent "MyCustomAgent/1.0" -custom-headers headers.txt -proxies proxies.txt -threads 5 -depth 3 -timeout 5
Contributions are welcome! Please feel free to submit a Pull Request.
โข Colly - Elegant Scraper and Crawler Framework for Golang.