staabm / phpstan-baseline-analysis

Analyzes PHPStan baseline files and creates aggregated error trend-reports

Analyzes phpstan baseline files

Analyzes PHPStan baseline files and creates aggregated error trend-reports.

Read more in the Blog post.

You need at least one of the supported PHPStan RuleSets/Rules configured in your project, to get meaningful results.

Installation

composer require staabm/phpstan-baseline-analysis --dev

Supported Rules

PHPStan RuleSets

• https://github.com/phpstan/phpstan-deprecation-rules

PHPStan Rules

• PHPStan\Rules\PhpDoc\InvalidPhpDocTagValueRule

Symplify PHPStan Rules

• Symplify\PHPStanRules\Rules\Explicit\NoMixedMethodCallerRule
• Symplify\PHPStanRules\Rules\Explicit\NoMixedPropertyFetcherRule

tomasvotruba/cognitive-complexity Rules

• TomasVotruba\CognitiveComplexity\Rules\ClassLikeCognitiveComplexityRule

tomasvotruba/type-coverage Rules

• TomasVotruba\TypeCoverage\Rules\ParamTypeCoverageRule
• TomasVotruba\TypeCoverage\Rules\PropertyTypeCoverageRule
• TomasVotruba\TypeCoverage\Rules\ReturnTypeCoverageRule

tomasvotruba/unused-public Rules

• TomasVotruba\UnusedPublic\Rules\UnusedPublicClassConstRule
• TomasVotruba\UnusedPublic\Rules\UnusedPublicClassMethodRule
• TomasVotruba\UnusedPublic\Rules\UnusedPublicPropertyRule

example report

Starting from the current directory, the command will recursively search for files matching the glob pattern and report a summary for each baseline found.

$ phpstan-baseline-analyze *phpstan-baseline.neon
Analyzing app/portal/phpstan-baseline.neon
  Overall-Errors: 41
  Classes-Cognitive-Complexity: 70
  Deprecations: 2
  Invalid-Phpdocs: 5
  Unknown-Types: 1
  Anonymous-Variables: 4
  Native-Property-Type-Coverage: 1
  Native-Param-Type-Coverage: 27
  Native-Return-Type-Coverage: 4
  Unused-Symbols: 3

example error filtering

Filter a existing baseline and output only errors NOT matching the given filter key:

Tip

This can be helpful to remove a class of errors out of an existing baseline, so PHPStan will start reporting them again.

$ phpstan-baseline-filter *phpstan-baseline.neon --exclude=Unknown-Types

Filter a existing baseline and output only errors matching the given filter key:

$ phpstan-baseline-filter *phpstan-baseline.neon --include=Invalid-Phpdocs

Currently supported filter keys can be found in the source.

example graph analysis

$ git clone ...

$ phpstan-baseline-analyze *phpstan-baseline.neon --json > now.json

$ git checkout `git rev-list -n 1 --before="1 week ago" HEAD`
$ phpstan-baseline-analyze '*phpstan-baseline.neon' --json > 1-week-ago.json

$ git checkout `git rev-list -n 1 --before="2 week ago" HEAD`
$ phpstan-baseline-analyze '*phpstan-baseline.neon' --json > 2-weeks-ago.json

$ php phpstan-baseline-graph '*.json' > result.html

example trend analysis

the following example shows the evolution of errors in your phpstan baselines. see the trend between 2 different points in time like:

$ git clone ...

$ phpstan-baseline-analyze '*phpstan-baseline.neon' --json > now.json

$ git checkout `git rev-list -n 1 --before="1 week ago" HEAD`

$ phpstan-baseline-analyze '*phpstan-baseline.neon' --json > reference.json

$ phpstan-baseline-trend reference.json now.json
Analyzing Trend for app/portal/phpstan-baseline.neon
  Overall-Errors: 30 -> 17 => improved
  Classes-Cognitive-Complexity: 309 -> 177 => improved
  Deprecations: 1 -> 2 => worse
  Invalid-Phpdocs: 3 -> 1 => good
  Unknown-Types: 5 -> 15 => worse
  Anonymous-Variables: 4 -> 3 => good
  Unused-Symbols: 1 -> 1 => good
  Native-Return-Type-Coverage: 20 -> 2 => worse
  Native-Property-Type-Coverage: 3 -> 3 => good
  Native-Param-Type-Coverage: 4 -> 40 => improved

Usage example in a scheduled GitHub Action with Mattermost notification

Copy the following workflow into your repository. Make sure to adjust as needed:

• adjust the cron schedule pattern
• actions/checkout might require a token - e.g. for private repos
• adjust the comparison period, as you see fit
• adjust the notification to your needs - e.g. use Slack, Discord, E-Mail,..

name: Trends Analyse

on:
  workflow_dispatch:
  schedule:
    - cron: '0 8 * * 4'

jobs:

  behat:
    name: Trends
    runs-on: ubuntu-latest
    timeout-minutes: 10

    steps:
      - run: "composer global require staabm/phpstan-baseline-analysis"
      - run: echo "$(composer global config bin-dir --absolute --quiet)" >> $GITHUB_PATH

      - uses: actions/checkout@v2
        with:
          fetch-depth: 50 # fetch the last X commits.

      - run: "phpstan-baseline-analyze '*phpstan-baseline.neon' --json > ../now.json"

      - run: git checkout `git rev-list -n 1 --before="1 week ago" HEAD`

      - run: "phpstan-baseline-analyze '*phpstan-baseline.neon' --json > ../reference.json"

      - name: analyze trend
        shell: php {0}
        run: |
          <?php
          exec('phpstan-baseline-trend ../reference.json ../now.json > ../trend.txt', $output, $exitCode);
          $project = '${{ github.repository }}';

          if ($exitCode == 0) {
            # improvements
            file_put_contents(
              'mattermost.json',
              json_encode(["username" => "github-action-trend-bot", "text" => $project ." :tada:\n". file_get_contents("../trend.txt")])
            );
          }
          elseif ($exitCode == 1) {
            # steady
            file_put_contents(
              'mattermost.json',
              json_encode(["username" => "github-action-trend-bot", "text" => $project ." :green_heart:\n". file_get_contents("../trend.txt")])
            );
          }
          elseif ($exitCode == 2) {
            # got worse
            file_put_contents(
              'mattermost.json',
              json_encode(["username" => "github-action-trend-bot", "text" => $project ." :broken_heart:\n". file_get_contents("../trend.txt")])
            );
          }

      - run: 'curl -X POST -H "Content-Type: application/json" -d @mattermost.json ${{ secrets.MATTERMOST_WEBHOOK_URL }}'
        if: always()

💌 Give back some love

Consider supporting the project, so we can make this tool even better even faster for everyone.

phpstan-baseline-analysis's Issues

bash read/write hack

to read and write the same file within a single pipe we need a hack

maybe we should instead add a new option like --out-file or similar so we can write the file from within php instead of using stdout redirection on the bash

up to discussion

Use error identifiers from baseline (1.11)

In the upcoming PHPStan 1.11 we'll see // identifier: comments. Could this be useful to further analyze the baseline?

phpstan/phpstan-src@4b32cac

Count unknown classes/types

Count invalid phpdocs

Count deprecations

update sea level rules

use https://github.com/TomasVotruba/type-coverage

support "Argument of an invalid type" errors

support errors like Argument of an invalid type Angkdet supplied for foreach, only iterables are supported and treat it like a Unknown-Type

support sea-level rules / type coverage

https://tomasvotruba.com/blog/how-to-measure-your-type-coverage/

phpstan-baseline-filter supports reading a baseline in PHP format, but writes it back to NEON

It's great that phpstan-baseline-filter supports reading a baseline from PHP format.

But it would be great if it would then also write it back in PHP format, instead of the default NEON format.

invalid phpdocs

these errors should be counted as invalid hppdoc

------ ---------------------------------------------------------------------- 
  Line   app/shared/controllers/KMCmsfrontendController.php                    
 ------ ---------------------------------------------------------------------- 
  8      PHPDoc tag @var has invalid value (DataLayerError|null;): Unexpected  
         token ";", expected TOKEN_HORIZONTAL_WS at offset 35                  
 ------ ---------------------------------------------------------------------- 

 ------ --------------------------------------------------------------------- 
  Line   lib/KunzmannRemoteOAuthRequest.php                                   
 ------ --------------------------------------------------------------------- 
  46     PHPDoc tag @return has invalid value (array('id' => string, 'email'  
         => string, 'firstname' => string, 'lastname' => string,              
         'authProvider' => string}): Unexpected token "(", expected           
         TOKEN_HORIZONTAL_WS at offset [24](https://github.com/complex-gmbh/kunzmann/actions/runs/4818214651/jobs/8579907342?pr=4056#step:16:25)                                     
 ------ ---------------------------------------------------------------------

Support baseline in PHP format

phpstan now supports baselines written in PHP format (to avoid the need for parsing neon, which helps with huge baselines). It would be great to be able to analyze them as well.

Support filtering of baselines

atm the tool is able to analyze existing baselines and provide a summary as a result, grouped into categories:

Overall-Errors: 41
  Classes-Cognitive-Complexity: 70
  Deprecations: 2
  Invalid-Phpdocs: 5
  Unknown-Types: 1
  Anonymous-Variables: 4
  Native-Property-Type-Coverage: 1
  Native-Param-Type-Coverage: 27
  Native-Return-Type-Coverage: 4
  Unused-Symbols: 3

it would be great if I could filter existing baselines, based on category, and get back a new baseline.
this would allow me to work with focus on a certain problem area, without manually searching the existing full-project-baseline

---

example 1 (which outputs all filtered content on stdout)

phpstan-baseline-analyze '*phpstan-baseline.neon' --filter=Deprecations

example 2(which outputs the content into new-baseline.neon)

phpstan-baseline-analyze '*phpstan-baseline.neon' --filter=Invalid-Phpdocs > new-baseline.neon

the result should be rendered in neon-format, like the default PHPStan baseline, so we can further process it with whatever tool which supports the default phpstan baseline format