ihebski / linksf1nd3r

web information gathering / Grab links

linksF1nd3r

Status: Development

About LinksF1nder

Extracts links from an HTML page and display them with friendly way ,this tool could be used for web information gathering ,to get more details about the web application.

Features

• Extract all the links not only the a href="#" tags
• Identifying the extention of the linked file
• Generate report
• Count the links based on the extension

Usage

$ python linksF1nd3r.py URL

Example:

• Testing on the local host

python linksF1nd3r.py http://127.0.0.1/links.php

• Testing on real website [stackoverflow.com]

python linksF1nd3r.py https://stackoverflow.com/questions/11487049/python-list-of-lists

How to install

Clone

• Clone the repository with:

$ git clone https://github.com/ihebski/LinksF1nd3r.git
$ cd LinksF1nd3r
$ python linksF1nd3r.py

Dependencies

• Install the required dependencies with:

$ sudo pip install -r requirements.txt

License

The MIT License (MIT)

linksf1nd3r's Issues

Automatically set Apostrophe around url

@ihebski
you must automatically set Apostrophe around input url.
like this:

python linksF1nd3r.py 'URL'

Automatically putting Apostrophe around the url, will allow LinksF1nd3r to deal with complex urls.
example:

https://somesite.com/search?somestuff&q=created=&l=

UnicodeEncodeError: 'ascii' codec can't encode characters

@ihebski
My guess is that you next to set utf-8 as default instead of ascii

example:

python linksF1nd3r.py https://github.com/zbyuan/pruning_yolov3

error:

[21:18:48] - [ ! ] - 
[21:18:48] - [ ! ] - /join?source_repo=zbyuan%2Fpruning_yolov3
Traceback (most recent call last):
  File "linksF1nd3r.py", line 123, in <module>
    start(sys.argv[1:])
  File "linksF1nd3r.py", line 66, in start
    report(url)	
  File "linksF1nd3r.py", line 114, in report
    print Fore.GREEN+"[{time}] - [ ! ] - {mylink}".format(mylink=link,time=time.strftime("%H:%M:%S"))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 1-2: ordinal not in range(128)

list of pages to grab the links

@ihebski how to use a list of pages and grab the links of all of them?

SyntaxError: invalid syntax

@ihebski
When running python linksF1nd3r.py I get error:

$ python linksF1nd3r.py www.imdb.com
  File "linksF1nd3r.py", line 28
    print "[!] "+str(err)
               ^
SyntaxError: invalid syntax

Please Help file error line 121, in <module> start(sys.argv[1:])

I have updated all the requirements.txt to use with python3 now I'm having trouble with

line 121, in
start(sys.argv[1:])

Can you please help me amend the following errors so I can actually use the program????

---

Traceback (most recent call last):
File "/data/data/com.termux/files/home/LinksF1nd3r/linksF1nd3r.py", line 121, in
start(sys.argv[1:])
File "/data/data/com.termux/files/home/LinksF1nd3r/linksF1nd3r.py", line 67, in start
report(url)
^^^^^^^^^^^
File "/data/data/com.termux/files/home/LinksF1nd3r/linksF1nd3r.py", line 97, in report
links = serchLinks(url)
^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/home/LinksF1nd3r/linksF1nd3r.py", line 75, in serchLinks
website = urllib.request.urlopen(url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/urllib/request.py", line 216, in urlopen
return opener.open(url, data, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/urllib/request.py", line 525, in open
response = meth(req, response)
^^^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/urllib/request.py", line 634, in http_response
response = self.parent.error(
^^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/urllib/request.py", line 563, in error
return self._call_chain(*args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/urllib/request.py", line 496, in _call_chain
result = func(*args)
^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/urllib/request.py", line 643, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

---

Help

Used this once b4 and it was great
This time around keep getting this error

python linksF1nd3r.py

/data/data/com.termux/files/home/LinksF1nd3r/linksF1nd3r.py:7: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp [!] module 'html5lib.treebuilders' has no attribute '_base'

Help