hueristiq / xurlfind3r Goto Github PK

Description
Hi, thanks for the amazing tool. When I playing around using 'commoncrawl' as the source I found it sometimes can print out urls but sometimes can't. So I modified the code to print out the errors alongside then I found commoncrawl returned tons of 503 and timeout

Steps To Reproduce
./sigurlfind3r -d tesla.com --include-subs -uS commoncrawl

Screenshots

Additional context
I tested over my home network as well as a VPS located on Los Angeles.
I changed a bit of the code to avoid parallel processing and increased the timeout to 60s from 10s. Then it worked as expected.
I haven't found any official documented rate limit of commoncrawl though, but reliability is the top concern in my opinion, especially in terms of automation. Just for your reference and thanks again for your work :)

.

Really quick tool, it would be great if it parses only robots.txt file

Hey there, I hope all is good.

While we already discussed this, I will create an issue for it to keep things on track.

The tool does indeed issue a request to intelx /phonebook/search path, however, it doesn't return any results in return.

./hqurlfind3r --use-sources otx -d google.com

returns nothing for otx

how use cat sub.txt | sigurlfind3r

Hello , Really great tool but I noticed that there is no option for saving output in a file that causes a lot of problem for me. It would be really really best if you kindly add this feature to this tool for saving output in a file.

I see this alot but figured i would ask you, why would you use import "github.com/signedsecurity/sigurlfind3r/pkg/sigurlfind3r/session"

when you can just

import crawler_utils "path_to_File_name_of_package"

Hi,
As it stands, the URLScan check seems to me to be limited for two reasons :

• It's not possible to specify an API key, so you can quickly get stuck if you scan several URLs.

  • See https://urlscan.io/docs/api/ > "Quotas & Rate Limiting"

• Results extraction seems to be limited to 100 (i.e. the first page). Is necessary to check if the response contains more than 100 results and, if so, iterate to retrieve the other results.

  • See https://urlscan.io/docs/api/ > "Search API"
  • An example : https://github.com/lc/gau/blob/master/pkg/providers/urlscan/urlscan.go

  Regards

Hi,
Thanks for this tool !

I think it would be interesting to add a flag to filter out certain extensions, something like :

-f     --filter                            comma(,) separated extensions to excludes (ex: jpg,png,svg)

Regards

Describe the bug
panic: runtime error: index out of range [0] with length 0

Steps To Reproduce
./sigurlfind3r -d example.com -iS
Output
[ INF ] fetching urls for

panic: runtime error: index out of range [0] with length 0

goroutine 18 [running]:
github.com/signedsecurity/sigurlfind3r/pkg/sigurlfind3r/scraping/sources/commoncrawl.(*Source).Run.func1()
/Users/emilaltynbaev/Documents/bugbounty/tools/sigurlfind3r/pkg/sigurlfind3r/scraping/sources/commoncrawl/commoncrawl.go:82 +0x7bc
created by github.com/signedsecurity/sigurlfind3r/pkg/sigurlfind3r/scraping/sources/commoncrawl.(*Source).Run
/Users/emilaltynbaev/Documents/bugbounty/tools/sigurlfind3r/pkg/sigurlfind3r/scraping/sources/commoncrawl/commoncrawl.go:63 +0xf2

Additional context
MacOS 12.1

Hey there,

config.yaml file is no longer under $HOME/.hueristiq/xurlfind3r/config.yaml , it's currently under /$HOME/.config/xurlfind3r/config.yaml , please update it in the readme file

Hi,
I think it would be great to add a flag to specify the config file location, something like :

-c     --config                            config file location

Regards