spatie / http-status-check

CLI tool to crawl a website and check HTTP status codes

Check the HTTP status code of all links on a website

This repository provides a tool to check the HTTP status code of every link on a given website.

Support us

We invest a lot of resources into creating best in class open source packages. You can support us by buying one of our paid products.

We highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using. You'll find our address on our contact page. We publish all received postcards on our virtual postcard wall.

Installation

This package can be installed via Composer:

composer global require spatie/http-status-check

Usage

This tool will scan all links on a given website:

http-status-check scan https://example.com

It outputs a line per link found. Here's an example on Laracast website scan:

When the crawling process is finished a summary will be shown.

By default the crawler uses 10 concurrent connections to speed up the crawling process. You can change that number by passing a different value to the --concurrency option:

http-status-check scan https://example.com --concurrency=20

You can also write all urls that gave a non-2xx or non-3xx response to a file:

http-status-check scan https://example.com --output=log.txt

When the crawler finds a link to an external website it will by default crawl that link as well. If you don't want the crawler to crawl such external urls use the --dont-crawl-external-links option:

http-status-check scan https://example.com --dont-crawl-external-links

By default, requests timeout after 10 seconds. You can change this by passing the number of seconds to the --timeout option:

http-status-check scan https://example.com --timeout=30

By default, the crawler will respect robots data. You can ignore them though with the --ignore-robots option:

http-status-check scan https://example.com --ignore-robots

If your site requires a basic authentification, you can use the --auth option:

http-status-check scan https://example.com --auth=username:password

Testing

To run the tests, first make sure you have Node.js installed. Then start the included node based server in a separate terminal window:

cd tests/server
npm install
node server.js

With the server running, you can start testing:

vendor/bin/phpunit

Changelog

Please see CHANGELOG for more information on what has changed recently.

Contributing

Please see CONTRIBUTING for details.

Security

If you've found a bug regarding security please mail [email protected] instead of using the issue tracker.

Credits

• Freek Van der Herten
• Sebastian De Deyne
• All Contributors

License

The MIT License (MIT). Please see License File for more information.

http-status-check's Issues

php 5.6

Hello

does it really requirs php 5.6?

Invalid status code in Response.php

I was running scan against a site and got the following error:

In Response.php line 151:

  Status code must be an integer value between 1xx and 5xx.

Presumably, something has used a non-standard status code, but I don't know the URL as it doesn't come out in the error, also do you know where Response.php is as its not part of this package from what I can see.

Not working on L5.6

When trying to install via composer I get:

Problem 1
- Installation request for spatie/http-status-check ^3.0 -> satisfiable by spatie/http-status-check[3.0.0].
- Conclusion: remove symfony/console v3.3.12
- Conclusion: don't install symfony/console v3.3.12
- spatie/http-status-check 3.0.0 requires symfony/console ^4.0 -> satisfiable by symfony/console[v4.0.0, v4.0.1, v4.0.2, v4.0.3, v4.0.4, v4.0.5, v4.0.6, v4.0.7, v4.0.8].
- Can only install one of: symfony/console[v4.0.0, v3.3.12].
- Can only install one of: symfony/console[v4.0.1, v3.3.12].
- Can only install one of: symfony/console[v4.0.2, v3.3.12].
- Can only install one of: symfony/console[v4.0.3, v3.3.12].
- Can only install one of: symfony/console[v4.0.4, v3.3.12].
- Can only install one of: symfony/console[v4.0.5, v3.3.12].
- Can only install one of: symfony/console[v4.0.6, v3.3.12].
- Can only install one of: symfony/console[v4.0.7, v3.3.12].
- Can only install one of: symfony/console[v4.0.8, v3.3.12].
- Installation request for symfony/console (locked at v3.3.12) -> satisfiable by symfony/console[v3.3.12].

Add a --no-external flag

Add a --no-external flag that doesn't crawl external urls.

Error after updating from 3.0.0 to 3.1.0 (getReasonPhrase on null)

$ http-status-check scan https://project.dev/ -x

Start scanning https://project.dev/

PHP Fatal error:  Uncaught Error: Call to a member function getReasonPhrase() on null in /Users/ben/.composer/vendor/spatie/http-status-check/src/CrawlLogger.php:146

PHP8 version doesnt work

When installing http-status-check the following error occured:

  Problem 1
    - spatie/crawler[4.7.0, ..., 4.7.5] require php ^7.1 -> your php version (8.0.2) does not satisfy that requirement.

I was working on a PR to update the composer to the crawler 6.* version but in version 5 there were breaking changes that are not explained in an update guide.

So when i changed the version to 6, all the tests break because Spatie\Crawler\CrawlAllUrls has been removed. I think there are several deeper issues now, and i dont think i can solve them the way you want. It is also more a thing like: "in what way you want it to work"

Not every option is included

The "--output" option does not exist.

Add CSV output option and treat specified URL as erroneous

Enhancement request: Is it possible to add a CSV output type along with an option to consider links from a specific URL as errors?

Perhaps delimiting the fields as "Status Code, URL, Referrer".

For example, say I wanted to check the status of a website and wanted to identify all links going to google.com as bad links so that they show up in a CSV output type to make for easier viewing in Excel.

how it work in ubuntu

hey i install it with bash

$ sudo composer require spatie/http-status-check

also in bash

$ http-status-check scan https://example.com

http-status-check: command not found

Add support for Js executing

Would you be interested by adding an option --execute-js ?

cf https://github.com/spatie/crawler#executing-javascript

Add custom options to guzzle client

The crawler is very nice, but I am missing the option to add custom (or more) options to the guzzle client.
Like the option to use a custom user agent/custom headers or just the verify flag to false.
Is this a possible future option?

Check HTTP status of images

Hello,

first I have to say that it's a very useful tool to check http status.

My question would be, is it possible to check the http status of images?

Code: <img src="...">

If not, have you considered to implement this feature?

Feature request for delay/wait timer between requests

hello,
is it possible to add a wait timer option?
greetings,
chris

Version 3.0 returns 301 on website to check

This command works well with 2.5, but with 3.0 not anymore:

> vendor/bin/http-status-check scan https://tomasvotruba.cz --output invalid-links.txt

Start scanning https://tomasvotruba.cz

[2018-02-19 10:34:26] 301 Moved Permanently - https://tomasvotruba.cz/

Might be related to Cloudflare I'm using.

Only scans one URL

I've been using the tool for a while, but on some URLs it only scans the homepage, and i can't understand why.

These are Wordpress sites, default .htaccess

./http-status-check scan https://anti-radar.org --concurrency=5 --dont-crawl-external-links

Start scanning https://anti-radar.org

[2017-04-19 20:58:23] 200 OK - https://anti-radar.org/

Crawling summary
----------------
Crawled 1 url(s) with statuscode 200

Not picking up some 404's

I cant figure this one out! I noticed the results were missing some known 404's so crawled one directly. I thought it was because its behind a 301 it is not picked up:

$ ~/vendor/bin/http-status-check scan https://hudevad.com/hudevad-1890

Start scanning https://hudevad.com/hudevad-1890

[2020-02-17 11:59:55] 301 Moved Permanently - https://hudevad.com/hudevad-1890

Crawling summary
----------------
Crawled 1 url(s) with statuscode 301

However when I tried to recreate this on another domain it seems to work as expected:

$ ~/vendor/bin/http-status-check scan https://jhodges.co.uk/test301-to-404.html

Start scanning https://jhodges.co.uk/test301-to-404.html

[2020-02-17 11:59:44] 301 Moved Permanently - https://jhodges.co.uk/test301-to-404.html
[2020-02-17 11:59:44] 404: Not Found - https://jhodges.co.uk/test404.html (found on https://jhodges.co.uk/test301-to-404.html)

Crawling summary
----------------
Crawled 1 url(s) with statuscode 301
Crawled 1 url(s) with statuscode 404

I checked the server is returning all headers correctly, curl follows it just fine.

Error on --output option

Hi there,

using this command:

http-status-check scan http://www.mywebsite.com --output=mywebsite.log.txt

I am getting:

[Symfony\Component\Console\Exception\RuntimeException]
The "--output" option does not exist.

Any ideas? Am I missing anything?

Thank you

BasicHTTP authentication support

Hi,

would it be possible to add options to enable crawling a website which is only available through http basic auth?

Or as a workaround: how do I have to structure my cmdline call to add authentiction headers?

Output to file

Currently it outputs a summary in the console, but because the actual links are printed while the scan runs it's quite messy having to scroll back up and find the broken links.

A nice feature would be to pass an option to allow printing to a file with for example the links not returning a 2xx response.

Tests are failing

One of the tests is failing on GitHub Actions. This should be fixed.

Build PHAR

Hi,

First, thanks for this good & useful package.
I used it in different applications and it helped to find broken pages.

I am now moving to docker and would like to include it in my image.
Is it possible to distribute this package as a PHAR file so it is easier to deploy ?

Thanks,

Confusing --output overwrite (append) behaviour

When using --output <filename> if the file exists we are asked if we want to overwrite the file. If we respond with "yes" then the file is actually appended to rather than overwritten.

Feature Request: Full output log in parsable format

I'm wanting to use the full crawl output to use in other apps.

Currently I can redirect stdout to a file and then open that in the other apps, but parsing the format requires a bit of regexp, and due to spaces in response reasons and not quoting it may be troublesome to perfect. I'm thinking CSV format output may be better.

I'm half way through implementing:

• Add a new flag -f --csv <filename> that outputs the entire crawl as a csv file.

But then thought that maybe:

• Modify the -o --output format to be CSV, and add another flag -f --full to specify for all responses not just errors to be outputted.

Would you be interested in this? Shall I make a PR so you can see what I've done, or just link to my repo?

Deprecated notices for PHP 8.1.6

I love this package and it is working perfectly fine.

I just saw a couple of notices that I don't believe affect the package but I'm sharing them here just in case:

`PHP Deprecated: Return type of Tightenco\Collect\Support\Collection::offsetExists($key) should either be compatible with ArrayAccess::offsetExists(mixed $offset): bool, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/tightenco/collect/src/Collect/Support/Collection.php on line 1789

Deprecated: Return type of Tightenco\Collect\Support\Collection::offsetExists($key) should either be compatible with ArrayAccess::offsetExists(mixed $offset): bool, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/tightenco/collect/src/Collect/Support/Collection.php on line 1789
PHP Deprecated: Return type of Tightenco\Collect\Support\Collection::offsetGet($key) should either be compatible with ArrayAccess::offsetGet(mixed $offset): mixed, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/tightenco/collect/src/Collect/Support/Collection.php on line 1800

Deprecated: Return type of Tightenco\Collect\Support\Collection::offsetGet($key) should either be compatible with ArrayAccess::offsetGet(mixed $offset): mixed, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/tightenco/collect/src/Collect/Support/Collection.php on line 1800
PHP Deprecated: Return type of Tightenco\Collect\Support\Collection::offsetSet($key, $value) should either be compatible with ArrayAccess::offsetSet(mixed $offset, mixed $value): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/tightenco/collect/src/Collect/Support/Collection.php on line 1812

Deprecated: Return type of Tightenco\Collect\Support\Collection::offsetSet($key, $value) should either be compatible with ArrayAccess::offsetSet(mixed $offset, mixed $value): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/tightenco/collect/src/Collect/Support/Collection.php on line 1812
PHP Deprecated: Return type of Tightenco\Collect\Support\Collection::offsetUnset($key) should either be compatible with ArrayAccess::offsetUnset(mixed $offset): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/tightenco/collect/src/Collect/Support/Collection.php on line 1827

Deprecated: Return type of Tightenco\Collect\Support\Collection::offsetUnset($key) should either be compatible with ArrayAccess::offsetUnset(mixed $offset): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/tightenco/collect/src/Collect/Support/Collection.php on line 1827
PHP Deprecated: Return type of Tightenco\Collect\Support\Collection::count() should either be compatible with Countable::count(): int, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/tightenco/collect/src/Collect/Support/Collection.php on line 1768

Deprecated: Return type of Tightenco\Collect\Support\Collection::count() should either be compatible with Countable::count(): int, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/tightenco/collect/src/Collect/Support/Collection.php on line 1768
PHP Deprecated: Return type of Tightenco\Collect\Support\Collection::getIterator() should either be compatible with IteratorAggregate::getIterator(): Traversable, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/tightenco/collect/src/Collect/Support/Collection.php on line 1747

Deprecated: Return type of Tightenco\Collect\Support\Collection::getIterator() should either be compatible with IteratorAggregate::getIterator(): Traversable, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/tightenco/collect/src/Collect/Support/Collection.php on line 1747
PHP Deprecated: Return type of Tightenco\Collect\Support\Collection::jsonSerialize() should either be compatible with JsonSerializable::jsonSerialize(): mixed, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/tightenco/collect/src/Collect/Support/Collection.php on line 1716

Deprecated: Return type of Tightenco\Collect\Support\Collection::jsonSerialize() should either be compatible with JsonSerializable::jsonSerialize(): mixed, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/tightenco/collect/src/Collect/Support/Collection.php on line 1716
Start scanning http://roboamp.com

PHP Deprecated: Return type of Spatie\Crawler\CrawlObserverCollection::offsetExists($offset) should either be compatible with ArrayAccess::offsetExists(mixed $offset): bool, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/spatie/crawler/src/CrawlObserverCollection.php on line 71

Deprecated: Return type of Spatie\Crawler\CrawlObserverCollection::offsetExists($offset) should either be compatible with ArrayAccess::offsetExists(mixed $offset): bool, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/spatie/crawler/src/CrawlObserverCollection.php on line 71
PHP Deprecated: Return type of Spatie\Crawler\CrawlObserverCollection::offsetGet($offset) should either be compatible with ArrayAccess::offsetGet(mixed $offset): mixed, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/spatie/crawler/src/CrawlObserverCollection.php on line 57

Deprecated: Return type of Spatie\Crawler\CrawlObserverCollection::offsetGet($offset) should either be compatible with ArrayAccess::offsetGet(mixed $offset): mixed, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/spatie/crawler/src/CrawlObserverCollection.php on line 57
PHP Deprecated: Return type of Spatie\Crawler\CrawlObserverCollection::offsetSet($offset, $value) should either be compatible with ArrayAccess::offsetSet(mixed $offset, mixed $value): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/spatie/crawler/src/CrawlObserverCollection.php on line 62

Deprecated: Return type of Spatie\Crawler\CrawlObserverCollection::offsetSet($offset, $value) should either be compatible with ArrayAccess::offsetSet(mixed $offset, mixed $value): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/spatie/crawler/src/CrawlObserverCollection.php on line 62
PHP Deprecated: Return type of Spatie\Crawler\CrawlObserverCollection::offsetUnset($offset) should either be compatible with ArrayAccess::offsetUnset(mixed $offset): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/spatie/crawler/src/CrawlObserverCollection.php on line 76

Deprecated: Return type of Spatie\Crawler\CrawlObserverCollection::offsetUnset($offset) should either be compatible with ArrayAccess::offsetUnset(mixed $offset): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/spatie/crawler/src/CrawlObserverCollection.php on line 76
PHP Deprecated: Return type of Spatie\Crawler\CrawlObserverCollection::current() should either be compatible with Iterator::current(): mixed, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/spatie/crawler/src/CrawlObserverCollection.php on line 52

Deprecated: Return type of Spatie\Crawler\CrawlObserverCollection::current() should either be compatible with Iterator::current(): mixed, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/spatie/crawler/src/CrawlObserverCollection.php on line 52
PHP Deprecated: Return type of Spatie\Crawler\CrawlObserverCollection::next() should either be compatible with Iterator::next(): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/spatie/crawler/src/CrawlObserverCollection.php on line 81

Deprecated: Return type of Spatie\Crawler\CrawlObserverCollection::next() should either be compatible with Iterator::next(): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/spatie/crawler/src/CrawlObserverCollection.php on line 81
PHP Deprecated: Return type of Spatie\Crawler\CrawlObserverCollection::key() should either be compatible with Iterator::key(): mixed, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/spatie/crawler/src/CrawlObserverCollection.php on line 86

Deprecated: Return type of Spatie\Crawler\CrawlObserverCollection::key() should either be compatible with Iterator::key(): mixed, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/spatie/crawler/src/CrawlObserverCollection.php on line 86
PHP Deprecated: Return type of Spatie\Crawler\CrawlObserverCollection::valid() should either be compatible with Iterator::valid(): bool, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/spatie/crawler/src/CrawlObserverCollection.php on line 91

Deprecated: Return type of Spatie\Crawler\CrawlObserverCollection::valid() should either be compatible with Iterator::valid(): bool, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/spatie/crawler/src/CrawlObserverCollection.php on line 91
PHP Deprecated: Return type of Spatie\Crawler\CrawlObserverCollection::rewind() should either be compatible with Iterator::rewind(): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/spatie/crawler/src/CrawlObserverCollection.php on line 96

Deprecated: Return type of Spatie\Crawler\CrawlObserverCollection::rewind() should either be compatible with Iterator::rewind(): void, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /Users/taytus/.composer/vendor/spatie/crawler/src/CrawlObserverCollection.php on line 96`

Difficulties installing via composer on Mac OSX with PHP 7.0.11

Dear Spatie team,

First of all thanks for all your hard work!

I am having difficulties installing the http-status-check package on OSX with PHP 7.0.11 (installed via LIIP).

The following is the command I fire and the errors returned.

$ composer global require spatie/http-status-check

hanged current directory to /Users/t.d.wit/.composer
You are running composer with xdebug enabled. This has a major impact on runtime performance. See https://getcomposer.org/xdebug
Using version ^1.0 for spatie/http-status-check
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Conclusion: don't install spatie/http-status-check 1.0.1
    - Conclusion: remove symfony/console v3.1.3
    - Installation request for spatie/http-status-check ^1.0 -> satisfiable by spatie/http-status-check[1.0.0, 1.0.1].
    - Conclusion: don't install symfony/console v3.1.3
    - spatie/http-status-check 1.0.0 requires symfony/console ^2.7 -> satisfiable by symfony/console[v2.7.0, v2.7.1, v2.7.10, v2.7.11, v2.7.12, v2.7.13, v2.7.14, v2.7.15, v2.7.16, v2.7.17, v2.7.18, v2.7.19, v2.7.2, v2.7.3, v2.7.4, v2.7.5, v2.7.6, v2.7.7, v2.7.8, v2.7.9, v2.8.0, v2.8.1, v2.8.10, v2.8.11, v2.8.12, v2.8.2, v2.8.3, v2.8.4, v2.8.5, v2.8.6, v2.8.7, v2.8.8, v2.8.9].
    - Can only install one of: symfony/console[v2.7.0, v3.1.3].
    - Can only install one of: symfony/console[v2.7.1, v3.1.3].
    - Can only install one of: symfony/console[v2.7.10, v3.1.3].
    - Can only install one of: symfony/console[v2.7.11, v3.1.3].
    - Can only install one of: symfony/console[v2.7.12, v3.1.3].
    - Can only install one of: symfony/console[v2.7.13, v3.1.3].
    - Can only install one of: symfony/console[v2.7.14, v3.1.3].
    - Can only install one of: symfony/console[v2.7.15, v3.1.3].
    - Can only install one of: symfony/console[v2.7.16, v3.1.3].
    - Can only install one of: symfony/console[v2.7.17, v3.1.3].
    - Can only install one of: symfony/console[v2.7.18, v3.1.3].
    - Can only install one of: symfony/console[v2.7.19, v3.1.3].
    - Can only install one of: symfony/console[v2.7.2, v3.1.3].
    - Can only install one of: symfony/console[v2.7.3, v3.1.3].
    - Can only install one of: symfony/console[v2.7.4, v3.1.3].
    - Can only install one of: symfony/console[v2.7.5, v3.1.3].
    - Can only install one of: symfony/console[v2.7.6, v3.1.3].
    - Can only install one of: symfony/console[v2.7.7, v3.1.3].
    - Can only install one of: symfony/console[v2.7.8, v3.1.3].
    - Can only install one of: symfony/console[v2.7.9, v3.1.3].
    - Can only install one of: symfony/console[v2.8.0, v3.1.3].
    - Can only install one of: symfony/console[v2.8.1, v3.1.3].
    - Can only install one of: symfony/console[v2.8.10, v3.1.3].
    - Can only install one of: symfony/console[v2.8.11, v3.1.3].
    - Can only install one of: symfony/console[v2.8.12, v3.1.3].
    - Can only install one of: symfony/console[v2.8.2, v3.1.3].
    - Can only install one of: symfony/console[v2.8.3, v3.1.3].
    - Can only install one of: symfony/console[v2.8.4, v3.1.3].
    - Can only install one of: symfony/console[v2.8.5, v3.1.3].
    - Can only install one of: symfony/console[v2.8.6, v3.1.3].
    - Can only install one of: symfony/console[v2.8.7, v3.1.3].
    - Can only install one of: symfony/console[v2.8.8, v3.1.3].
    - Can only install one of: symfony/console[v2.8.9, v3.1.3].
    - Installation request for symfony/console (locked at v3.1.3) -> satisfiable by symfony/console[v3.1.3].


Installation failed, reverting ./composer.json to its original content.

$ which php

/usr/local/php5/bin/php

$ php -v

PHP 7.0.11 (cli) (built: Sep 23 2016 20:33:19) ( NTS )
Copyright (c) 1997-2016 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.0.11, Copyright (c) 1999-2016, by Zend Technologies
    with Xdebug v2.4.1, Copyright (c) 2002-2016, by Derick Rethans

Thank you for looking into this.

Tom

Make use of Guzzle promises to increase performance

Checking would be a whole lot faster if the package would make use of guzzle promises. Maybe something for underlying crawler.

Add --ignore-robots option

With the latest version of the crawler, it will take robots rules into account. There should be an option to ignore them.

Tel: links are crawled

When you have a link to a phone number: <a href="tel:[phone]">Call me</a> the crawler tries to pick up on it and returns a 404.

It doesn't do this for mailto so I guess an exception should be added.

Show on which page a broken link is displayed

Right now the package only show the broken link itself.

Difficulties installing on a Win10 environment

Attempts at either '''composer global require spatie/http-status-check''' or cloning the repo and then running ''composer install' from within the directory both generate the same errors.

PHP and Composer versions are managed via latest from localwp.com

Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - spatie/crawler 4.1.0 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.1.1 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.1.2 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.1.3 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.1.4 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.1.5 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.1.6 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.1.7 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.2.0 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.3.0 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.3.1 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.3.2 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.4.0 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.4.1 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.4.2 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.4.3 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.5.0 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.6.0 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.6.1 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.6.2 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.6.3 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.6.4 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.6.5 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.6.6 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.6.7 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.6.8 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.6.9 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.7.0 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.7.1 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.7.2 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.7.3 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/crawler 4.7.4 requires spatie/browsershot ^3.14 -> satisfiable by spatie/browsershot[3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.16.1, 3.17.0, 3.18.0, 3.19.0, 3.20.0, 3.20.1, 3.22.0, 3.22.1, 3.23.0, 3.23.1, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.26.1, 3.26.2, 3.26.3, 3.27.0, 3.29.0, 3.30.0, 3.31.0, 3.31.1, 3.32.0, 3.32.1,
3.32.2, 3.33.0, 3.33.1, 3.34.0, 3.35.0, 3.36.0, 3.37.0, 3.37.1, 3.37.2].
    - spatie/browsershot 3.26.1 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.26.2 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.26.3 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.27.0 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.29.0 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.30.0 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.31.0 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.31.1 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.32.0 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.32.1 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.32.2 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.33.0 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.33.1 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.34.0 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.35.0 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.36.0 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.37.0 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.37.1 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.37.2 requires spatie/image ^1.5.3 -> satisfiable by spatie/image[1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.14.0 requires spatie/image ~1.3.5 -> satisfiable by spatie/image[1.3.5].
    - spatie/browsershot 3.14.1 requires spatie/image ^1.4 -> satisfiable by spatie/image[1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.15.0 requires spatie/image ^1.4 -> satisfiable by spatie/image[1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.16.0 requires spatie/image ^1.4 -> satisfiable by spatie/image[1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.16.1 requires spatie/image ^1.4 -> satisfiable by spatie/image[1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.17.0 requires spatie/image ^1.4 -> satisfiable by spatie/image[1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.18.0 requires spatie/image ^1.4 -> satisfiable by spatie/image[1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.19.0 requires spatie/image ^1.4 -> satisfiable by spatie/image[1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.20.0 requires spatie/image ^1.4 -> satisfiable by spatie/image[1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.20.1 requires spatie/image ^1.4 -> satisfiable by spatie/image[1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.22.0 requires spatie/image ^1.4 -> satisfiable by spatie/image[1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.22.1 requires spatie/image ^1.4 -> satisfiable by spatie/image[1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.23.0 requires spatie/image ^1.4 -> satisfiable by spatie/image[1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.23.1 requires spatie/image ^1.4 -> satisfiable by spatie/image[1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.24.0 requires spatie/image ^1.4 -> satisfiable by spatie/image[1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.25.0 requires spatie/image ^1.4 -> satisfiable by spatie/image[1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.25.1 requires spatie/image ^1.4 -> satisfiable by spatie/image[1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/browsershot 3.26.0 requires spatie/image ^1.4 -> satisfiable by spatie/image[1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6].
    - spatie/image 1.5.3 requires league/glide ^1.4 -> satisfiable by league/glide[1.4.0, 1.5.0, 1.6.0].
    - spatie/image 1.6.0 requires league/glide ^1.4 -> satisfiable by league/glide[1.4.0, 1.5.0, 1.6.0].
    - spatie/image 1.7.0 requires league/glide ^1.4 -> satisfiable by league/glide[1.4.0, 1.5.0, 1.6.0].
    - spatie/image 1.7.1 requires league/glide ^1.4 -> satisfiable by league/glide[1.4.0, 1.5.0, 1.6.0].
    - spatie/image 1.7.2 requires league/glide ^1.4 -> satisfiable by league/glide[1.4.0, 1.5.0, 1.6.0].
    - spatie/image 1.7.3 requires league/glide ^1.4 -> satisfiable by league/glide[1.4.0, 1.5.0, 1.6.0].
    - spatie/image 1.7.4 requires league/glide ^1.4 -> satisfiable by league/glide[1.4.0, 1.5.0, 1.6.0].
    - spatie/image 1.7.5 requires league/glide ^1.4 -> satisfiable by league/glide[1.4.0, 1.5.0, 1.6.0].
    - spatie/image 1.7.6 requires league/glide ^1.4 -> satisfiable by league/glide[1.4.0, 1.5.0, 1.6.0].
    - spatie/image 1.3.5 requires league/glide ^1.2 -> satisfiable by league/glide[1.2.0, 1.2.1, 1.2.2, 1.3.0, 1.4.0, 1.5.0, 1.6.0].
    - spatie/image 1.4.0 requires league/glide ^1.2 -> satisfiable by league/glide[1.2.0, 1.2.1, 1.2.2, 1.3.0, 1.4.0, 1.5.0, 1.6.0].
    - spatie/image 1.4.1 requires league/glide ^1.2 -> satisfiable by league/glide[1.2.0, 1.2.1, 1.2.2, 1.3.0, 1.4.0, 1.5.0, 1.6.0].
    - spatie/image 1.4.2 requires league/glide ^1.2 -> satisfiable by league/glide[1.2.0, 1.2.1, 1.2.2, 1.3.0, 1.4.0, 1.5.0, 1.6.0].
    - spatie/image 1.5.0 requires league/glide ^1.2 -> satisfiable by league/glide[1.2.0, 1.2.1, 1.2.2, 1.3.0, 1.4.0, 1.5.0, 1.6.0].
    - spatie/image 1.5.1 requires league/glide ^1.2 -> satisfiable by league/glide[1.2.0, 1.2.1, 1.2.2, 1.3.0, 1.4.0, 1.5.0, 1.6.0].
    - spatie/image 1.5.2 requires league/glide ^1.2 -> satisfiable by league/glide[1.2.0, 1.2.1, 1.2.2, 1.3.0, 1.4.0, 1.5.0, 1.6.0].
    - league/glide 1.4.0 requires intervention/image ^2.4 -> satisfiable by intervention/image[2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.5.0, 2.5.1].
    - league/glide 1.5.0 requires intervention/image ^2.4 -> satisfiable by intervention/image[2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.5.0, 2.5.1].
    - league/glide 1.6.0 requires intervention/image ^2.4 -> satisfiable by intervention/image[2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.5.0, 2.5.1].
    - league/glide 1.2.0 requires intervention/image ^2.1 -> satisfiable by intervention/image[2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.3.10, 2.3.11, 2.3.12, 2.3.13, 2.3.14, 2.3.2, 2.3.3, 2.3.4, 2.3.5, 2.3.6, 2.3.7, 2.3.8, 2.3.9, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.5.0, 2.5.1].
    - league/glide 1.2.1 requires intervention/image ^2.1 -> satisfiable by intervention/image[2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.3.10, 2.3.11, 2.3.12, 2.3.13, 2.3.14, 2.3.2, 2.3.3, 2.3.4, 2.3.5, 2.3.6, 2.3.7, 2.3.8, 2.3.9, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.5.0, 2.5.1].
    - league/glide 1.2.2 requires intervention/image ^2.1 -> satisfiable by intervention/image[2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.3.10, 2.3.11, 2.3.12, 2.3.13, 2.3.14, 2.3.2, 2.3.3, 2.3.4, 2.3.5, 2.3.6, 2.3.7, 2.3.8, 2.3.9, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.5.0, 2.5.1].
    - league/glide 1.3.0 requires intervention/image ^2.1 -> satisfiable by intervention/image[2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.3.10, 2.3.11, 2.3.12, 2.3.13, 2.3.14, 2.3.2, 2.3.3, 2.3.4, 2.3.5, 2.3.6, 2.3.7, 2.3.8, 2.3.9, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.5.0, 2.5.1].
    - intervention/image 2.5.1 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.5.0 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.4.3 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.4.2 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.4.1 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.4.0 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.3.9 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.3.8 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.3.7 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.3.6 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.3.5 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.3.4 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.3.3 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.3.2 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.3.14 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.3.13 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.3.12 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.3.11 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.3.10 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.3.1 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.3.0 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.2.2 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.2.1 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.2.0 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.1.4 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.1.3 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.1.2 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.1.1 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - intervention/image 2.1.0 requires ext-fileinfo * -> the requested PHP extension fileinfo is missing from your system.
    - Installation request for spatie/crawler ^4.1.0 -> satisfiable by spatie/crawler[4.1.0, 4.1.1, 4.1.2, 4.1.3, 4.1.4, 4.1.5, 4.1.6, 4.1.7, 4.2.0, 4.3.0, 4.3.1, 4.3.2, 4.4.0, 4.4.1, 4.4.2, 4.4.3, 4.5.0, 4.6.0, 4.6.1, 4.6.2, 4.6.3, 4.6.4, 4.6.5, 4.6.6, 4.6.7, 4.6.8, 4.6.9, 4.7.0, 4.7.1, 4.7.2, 4.7.3, 4.7.4].

How to combine CrawlProfiles?

ive written a new CrawlProfile for Spatie\Crawler and like to add it as a Command.

$crawlProfile = $input->getOption('dont-crawl-external-links')
? new CrawlInternalUrls($baseUrl) : $input->getOption('ignore-querystrings')
? new IgnoreQueryStrings($baseUrl) : new CrawlAllUrls();

This does not give errors, but does not combine the Profiles. Any hints?