- Version
0.2pre
pylinkchecker is a standalone and pure python link checker and crawler that traverses a web site and reports errors (e.g., 500 and 404 errors) encountered. The crawler can also download resources such as images, scripts and stylesheets.
pylinkchecker's performance can be improved by installing additional libraries that require a C compiler, but these libraries are optional.
We created pylinkchecker so that it could be executed in environments without access to a compiler (e.g., Microsoft Windows, some *nix production environments) or with an old version of python (e.g., Centos).
pylinkchecker is highly modular and has many configuration options, but the only required parameter is the starting url: pylinkcheck.py http://www.example.com/
pylinkchecker can also be used programmatically by calling one of the functions in pylinkchecker.api
Install pylinkchecker with pip or easy_install:
pip install pylinkcheckerCrawl all pages from a site and show progress:
pylinkcheck.py -P http://www.example.com/pylinkchecker does not require external libraries if executed with python 2.x. It requires beautifulsoup4 if executed with python 3.x. It has been tested on python 2.6, python 2.7, and python 3.3.
For production use, it is strongly recommended to use lxml or html5lib because the default HTML parser provided by python is not very lenient.
These libraries can be installed to enable certain modes in pylinkchecker:
- lxml
beautifulsoup can use lxml to speed up the parsing of HTML pages. Because lxml requires C libraries, this is only an optional requirement.
- html5lib
beautifulsoup can use html5lib to process incorrect or strange markup. It is slower than lxml, but believed to be more lenient.
- gevent
this non-blocking io library enables pylinkchecker to use green threads instead of processes or threads. gevent could potentially speed up the crawling speed on web sites with many small pages.
- cchardet
this library speeds up the detection of document encoding.
This is a list of all available options. See the end of the README file for usage examples.
Usage: pylinkcheck.py [options] URL ...
Options:
--version show program's version number and exit
-h, --help show this help message and exit
-V VERBOSE, --verbose=VERBOSE
Crawler Options:
These options modify the way the crawler traverses the site.
-O, --test-outside fetch resources from other domains without crawling
them
-H ACCEPTED_HOSTS, --accepted-hosts=ACCEPTED_HOSTS
comma-separated list of additional hosts to crawl
(e.g., example.com,subdomain.another.com)
-i IGNORED_PREFIXES, --ignore=IGNORED_PREFIXES
comma-separated list of host/path prefixes to ignore
(e.g., www.example.com/ignore_this_and_after/)
-u USERNAME, --username=USERNAME
username to use with basic HTTP authentication
-p PASSWORD, --password=PASSWORD
password to use with basic HTTP authentication
-t TYPES, --types=TYPES
Comma-separated values of tags to look for when
crawlinga site. Default (and supported types):
a,img,link,script
-T TIMEOUT, --timeout=TIMEOUT
Seconds to wait before considering that a page timed
out
-C, --strict Does not strip href and src attributes from
whitespaces
-P, --progress Prints crawler progress in the console
-N, --run-once Only crawl the first page.
-S, --show-source Show source of links (html) in the report.
Performance Options:
These options can impact the performance of the crawler.
-w WORKERS, --workers=WORKERS
Number of workers to spawn
-m MODE, --mode=MODE
Types of workers: thread (default), process, or green
-R PARSER, --parser=PARSER
Types of HTML parse: html.parser (default) or lxml
Output Options:
These options change the output of the crawler.
-f FORMAT, --format=FORMAT
Format of the report: plain
-o OUTPUT, --output=OUTPUT
Path of the file where the report will be printed.
-W WHEN, --when=WHEN
When to print the report. error (only if a
crawling error occurs) or always (default)
-E REPORT_TYPE, --report-type=REPORT_TYPE
Type of report to print: errors (default, summary and
erroneous links), summary, all (summary and all links)
-c, --console Prints report to the console in addition to other
output options such as file or email.
Email Options:
These options allows the crawler to send a report by email.
-a ADDRESS, --address=ADDRESS
Comma-separated list of email addresses used to send a
report
--from=FROM_ADDRESS
Email address to use in the from field of the email
(optional)
-s SMTP, --smtp=SMTP
Host of the smtp server
--port=PORT Port of the smtp server (optional)
--tls Use TLS with the email server.
--subject=SUBJECT Subject of the email (optional)
--smtp-username=SMTP_USERNAME
Username to use with the smtp server (optional)
--smtp-password=SMTP_PASSWORD
Password to use with the smtp server (optional)- Crawl a site and show progress
pylinkcheck.py --progress http://example.com/- Crawl a site starting from 2 URLs
pylinkcheck.py http://example.com/ http://example2.com/- Crawl a site (example.com) and all pages belonging to another host
pylinkcheck.py -H additionalhost.com http://example.com/- Report status of all links (even successful ones)
pylinkcheck.py --report-type=all http://example.com/- Report status of all links and HTML show source of these links
pylinkcheck.py --report-type=all --show-source http://example.com/- Only crawl starting URLs and access all linked resources
pylinkcheck.py --run-once http://example.com/- Only access links (a href) and ignore images, stylesheets and scripts
pylinkcheck.py --types=a http://example.com/- Crawl a site with 4 threads (default is one thread)
pylinkcheck.py --workers=4 http://example.com/- Crawl a site with 4 processes (default is one thread)
pylinkcheck.py --mode=process --workers=4 http://example.com/- Crawl a site and use LXML to parse HTML (faster, must be installed)
pylinkcheck.py --parser=LXML http://example.com/- Print debugging info
pylinkcheck.py --verbose=2 http://example.com/
To crawl a site from a single URL:
from pylinkchecker.api import crawl
crawled_site = crawl("http://www.example.com/")
number_of_crawled_pages = len(crawled_site.pages)
number_of_errors = len(crawled_sites.error_pages)To crawl a site and pass some configuration options (the same supported by the command line interface):
from pylinkchecker.api import crawl_with_options
crawled_site = crawl_with_options(["http://www.example.com/"], {"run-once":
True, "workers": 10})
number_of_crawled_pages = len(crawled_site.pages)
number_of_errors = len(crawled_sites.error_pages)- I cannot find pylinkcheck.py on Windows with virtualenv
This is a known problem with virtualenv on windows. The interpreter is different than the one used by the virtualenv. Prefix pylinkcheck.py with the full path:
python c:\myvirtualenv\Scripts\pylinkcheck.py- I see Exception KeyError ... module 'threading' when using --mode=green
This output is generally harmless and is generated by gevent patching the python thread module. If someone knows how to make it go away, patches are more than welcome :-)
This software is licensed under the New BSD License. See the LICENSE file in the for the full license text. It includes the beautifulsoup library which is licensed under the MIT license.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent