shadawck / awesome-check-link

Check if the urls contained in a markdown file are down or not.

awesome-check-link

Check if links in md file and more particularly in awesome-list are down or not.

Install

You can install awesome-check-link either via pip (PyPI) or from source.

python3 -m pip install awesome-check-link

Or manually:

git clone https://github.com/remiflavien1/awesome-check-link
cd awesome-check-link
python3 setup.py install

To upgrade aclinks:

python3 -m pip  install --upgrade awesome-check-link

Usage

CLI

$ aclinks --help  
Awesome Check Links 

Usage:
    aclinks [--verbose --exit --down] -f FILE
    aclinks (-h | --help | --version)

Options:
    -f --file       Markdown file to scan.
    -e --exit       Stop and raise error if a site is down.
    -d --down       Show only down links and their line number.
    -v --verbose    Verbose mode. Print more to stdout.
    -h --help       Show this help.
    --version       Show version.

Example of output for the Awesome-anti-forensic

$ git clone https://github.com/remiflavien1/awesome-anti-forensic
$ cd awesome-anti-forensic
$ aclinks -vd -f README.md # only display down links

At line 24 : https://www.paterva.com/web7/buy/maltego-clients/casefile.php : Moved Permanently ( 301 )
At line 61 : https://www.bishopfox.com/resources/tools/other-free-tools/mafia/ : Moved Permanently ( 301 )
At line 64 : https://www.paterva.com/web7/buy/maltego-clients/maltego-ce.php : Moved Permanently ( 301 )

With exit mode (--exit flag), exit the program if one link is down. Can be integrated in a CI/CD workflow.

License

MIT

awesome-check-link's Issues

Make difference between 404 and other non-200 header

Try...catch Request Timout -> error 408

Add multiprocessing and Pool

Add allow_redirect to get_down_link

Maybe add it as an cli options

Perform HTTPS check when a website have moved from HTTP to HTTPS

Print remediation

Processing link with "www"

Try removing www part and perform a new request with àllow_redirection=True.
In some case you will get the redirection :
Example :
https://www.sno.phy.queensu.ca/~phil/exiftool/ don't work
but https://sno.phy.queensu.ca/~phil/exiftool/ redirect to exiftool.org

Make more test for non-200 header

• : Not Acceptable ( 406 )
• : Not Allowed ( 405 )
• : Service Unavailable ( 503 )
• : Forbidden ( 403 )
• : Request Timeout ( 408 )
• : Temporary redirect ( 307 )