byrnereese / linkchecker-mkdocs

Python asyncio + aiohttp Markdown *.md URL link checker: 10,000 files/second

Link Checker for Mkdocs-based static generated sites

This project was designed to help validate links associated with markdown-based, staticly generated website -- especially those published via Mkdocs. It is a fork of linkcheckmd, and offers many enhancements over its predecessor. This project has the following features:

• Scan and validate links for over 10,000 markdown files per second
• Check local (relative) and remote links
• Recurse through an entire documentation tree
• Check remote links using a synchronous or asynchronous process
• Exclude links from being checked
• Output useful summary reports to help you track down and fix broken links

While development focused on testing mkdocs-generated sites, this project should in theory work with any markdown-based website generator.

Install

For latest release:

% python -m pip install mkdocs-linkcheck

Or, for latest development version.

% git clone https://github.com/byrnereese/linkchecker-mkdocs
% pip install -e linkchecker-mkdocs

Usage

The static site generator does NOT have to be running for these tests. This program looks at the Markdown .md files directly.

If any local or remote links are determined to be missing, the following happens:

• the file containing the bad link and the link is printed to "stdout"
• the program will exit with code 22 instead of 0 after all files are checked

The bad links are printed to stdout since the normal operation of this program is to check for errors. Due to the fast, concurrent checking and numerous pages checked, there may be diagnostics printed to stderr. That way library error messages can be kept separate from the missing page locations printed on stdout.

The examples assume webpage Markdown files have top-level directory ~/docs.

Python code

import mkdocs-linkcheck as lc
lc.check_links("~/docs")

Command-line

This program may be invoked by either:

mkdocs-linkcheck

or

python -m mkdocs-linkcheck

Command link arguments

Usage

mkdocs-linkcheck [-h] [-ext EXT] [-m {get,head}] [-v] [--sync] [--exclude EXCLUDE] [-local] [-r] path [domain]

Positional arguments:

• path - path to Markdown files
• domain - check only links to this domain (say github.com without https etc.)

Optional arguments:

• -h, --help - show a help message and exit
• -ext <str> - file extension to scan
• -m {get,head}, --method {get,head} - The HTTP method to use when testing remote links. The "head" method is faster but gives false positives. The "get" method is reliable but slower
• --sync - enable synchronous checking of remote links, or do not use asyncio
• --exclude str - a pattern for a file or path to exclude from being checked; use this argument multiple times to exclude multiple files. Regular expressions are ok.
• -local - check local files only
• -r, --recurse - recurse through all directories under path
• -v or --verbose -prints the URLs as they are checked

Git precommit

See ./examples/pre-commit script for a Git hook pre-commit Python script.

Tox and CI

This program can also be used as a check for bad links during continuous integration testing or when using tox.

linkchecker-mkdocs's Issues

Add option to check check only remote files

As the parameter --local to just check local files. Add a parameter --remote to check only remote files.
This two paremters must be mutually exclusive.

Confusion about versions/releases

Hey @byrnereese,

I got interested in using your Python package in a project of mine and wanted to integrate it with pre-commit-hooks.nix for resulting support in devenv.
However when packaging it for nixpkgs, some confusion arose about versions/releases.

On GitHub the latest release is tagged with 1.4.0, while on PyPi the latest version is 1.0.6.

Could you please create a new release on both platforms 🙏

P.S. In case you could need some help in maintaining this package, I'd be interested 🙂

Pypi homepage is dead

The pypi site links to https://github.com/byrnereese/mkdocs-linkchecker which seems dead.

requests dependency not declared

I tried using the linkchecker in a project which had the requests library not installed. It seems the project is not declaring that dependency.

There is something odd going on with images

I have the following project:

site_name: Dummy - Site
site_url: https://example.com/dummy
site_dir: public
nav:
  - Index: index.md
  - Foo: foo.md

Both markdown files contain the following:

# Lemming

![Lemming](img/lemming.jpg)

If I run mkdocs-linkchecker it finds one problem in file foo.md:

☠ mkdocs-linkcheck -l docs/
Total files checked: 2
Total links checked: 2
        Local links: 2
       Remote links: 0
        Empty links: 0
       Broken links: 1
      Skipped links: 0

/tmp/dummy/docs/foo.md:
[✖] img/lemming.jpg
0.00154 seconds to check links
Bad links were found. Exiting with an error.

Please note it is happy about index.md even though it's the same file.
Also note that mkdocs serve is perfectly happy, and the site displays the image on both pages.

Now if I use ../img/lemming.jpg in foo.md instead:

# Lemming

![Lemming](../img/lemming.jpg)

Then, mkdocs-linkcheck is happy:

☠ mkdocs-linkcheck -l docs/
Total files checked: 2
Total links checked: 2
        Local links: 2
       Remote links: 0
        Empty links: 0
       Broken links: 0
      Skipped links: 0
0.00129 seconds to check links

However, mkdocs serve isn't:

☠ mkdocs serve
INFO     -  Building documentation...
INFO     -  Cleaning site directory
WARNING  -  Documentation file 'foo.md' contains a link to '../img/lemming.jpg' which
            is not found in the documentation files.
INFO     -  Documentation built in 0.06 seconds
INFO     -  [12:19:41] Watching paths for changes: 'docs', 'mkdocs.yml'
INFO     -  [12:19:41] Serving on http://127.0.0.1:8000/dummy/

Please note that both configurations work fine, both pages display the picture fine.

Files with broken links are not mentioned.

I would be nice that the files containing the broken links are shown in the output. At the moment you just get a list of broken links, without any clue where to search to fix them.

mkdocs extra links are not considered

Brief

URLs formed from the mkdocs extra variables are not checked.

Details

Following the mkdocs-theme documentation it's possible to have link variables defined/included in the mkdocs.yml by adding the bock

mkdocs.yml

extra:
  X:
    Y: "https://www.github.com/"

Then using it as following

docs\mydoc.md

[This is a URL]({{X.Y}}/byrnereese/linkchecker-mkdocs)

However, usage of variables while traversing mkdocs docs is ignored.

Technical Details

Note: that would require parsing mkdocs.yml for the extra node

Missing requests dependency

Missing requests dependency:

linkchecker-mkdocs/setup.cfg

Lines 31 to 32 in a75d765

	install_requires =
	aiohttp >= 3.3

When installing linkchecker in an isolated environment then it doesn't work

$ pipx install mkdocs-linkcheck
$ mkdocs-linkcheck
Traceback (most recent call last):
  File "/Users/mliiv/.local/bin/mkdocs-linkcheck", line 5, in <module>
    from mkdocs_linkcheck.__main__ import main
  File "/Users/mliiv/.local/pipx/venvs/mkdocs-linkcheck/lib/python3.9/site-packages/mkdocs_linkcheck/__init__.py", line 4, in <module>
    from .base import check_links, check_local, check_remotes
  File "/Users/mliiv/.local/pipx/venvs/mkdocs-linkcheck/lib/python3.9/site-packages/mkdocs_linkcheck/base.py", line 11, in <module>
    from .sync import check_urls as check_urls_sync
  File "/Users/mliiv/.local/pipx/venvs/mkdocs-linkcheck/lib/python3.9/site-packages/mkdocs_linkcheck/sync.py", line 2, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'

All relative links are marked as invalid.

When refering to a local link or image, the link is always marked as broken, even if the objects linked are there.

e.g.

![5x1](./5x1.png)
- **SPARC Platform** functionality (For more information, please refer to [**SPARC**](../060_SPARC/061_SPARC_Introduction.md) documentation)

results in the following errors:

INFO:root:Broken image: ./5x1.png
INFO:root:Broken link: ../060_SPARC/061_SPARC_Introduction.md