anuraag016 / jest-coverage-diff

Jest Coverage report diff as a comment for Pull requests

Jest coverage diff

Use the action to get jest coverage diff for pull requests as a comment on the pull request Helps the code reviewer to get the high level view of code coverage changes without leaving the pull request window

example:

Code coverage comparison master vs testBranch:

File	% Stmts	% Branch	% Funcs	% Lines
total	99.55 97.73	100 96.97	97.96 95.92	99.54 97.7
src/Error/TestError.ts	100 77.78	100 100	100 66.67	100 77.78
src/Utility/Utility.ts	96.67 90	100 75	88.89 88.89	96.67 90

How It Works

uses the following jest command to get code coverage summary as json for the pull request.

npx jest --coverage --coverageReporters="json-summary"

Then switches branch to the base branch on which the pull request has been raised and runs the same command again. Calculates the diff between the two reports to figure out additions, removals, increase or decrease in code coverage. And then posts that diff as a comment on the PR

NOTE : The action will work perfectly only for pull requests. Have not been tested with other events or on schedule workflows

Configuration

The action assumes jest configuration and jest module already present in the workflow and uses the installed module and the already present config to run the tests.

NEW:

• The action now supports runnning a command just after switching to base branch, this is extremely helpful in cases where there might be some packages removed in the pull request raised, in such cases there now is a possibility to re run the npm ci/npm install commands before running the jset coverage on base branch. Use afterSwitchCommand variable to pass a custom command to be run after switching to base branch.
• The action now supports custom run command, for custom use cases, using the variable runCommand, you can now pass your own command to run. Following is an example where we want to collect coverage from only few files out of all the code and want to use custom options such as forceExit & detectOpenHandles.

   runCommand: "npx jest --collectCoverageFrom='[\"src/**/*.{js,jsx,ts,tsx}\"]' --coverage --collectCoverage=true --coverageDirectory='./' --coverageReporters='json-summary' --forceExit --detectOpenHandles test/.*test.*"

NOTE: If using custom command, --coverage --collectCoverage=true --coverageDirectory='./' --coverageReporters='json-summary', these options are necessary for the action to work properly. These options tells jest to collect the coverage in json summary format and put the final output in the root folder. Since these are necessary, will make the action add them automatically in the next version.

• Do you want to fail the workflow if the commited code decreases the percentage below a tolerable level? Do you to start a healthy culture of writing test cases? The action now also supports failing the run if percentage diff is more than a specified delta value for any file, you can specify the delta value using the variable delta

  delta: 1 // the action will fail if any of the percentage dip is more than 1% for any changed file

Sample workflow for running this action

name: Node.js CI

on: pull_request

jobs:
  build:
    strategy:
      matrix:
        node-version: [14.x]
        platform: [ubuntu-latest]
    runs-on: ${{ matrix.platform }}
    steps:
    - uses: actions/checkout@v2
    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v1
      with:
        node-version: ${{ matrix.node-version }}
    - run: npm ci
    - name: TestCoverage
      id: testCoverage
      uses: anuraag016/Jest-Coverage-Diff@master
      with:
        fullCoverageDiff: false // defaults to false, if made true whole coverage report is commented with the diff
        runCommand: "npx jest --collectCoverageFrom='[\"src/**/*.{js,jsx,ts,tsx}\"]' --coverage --collectCoverage=true --coverageDirectory='./' --coverageReporters='json-summary' --forceExit --detectOpenHandles test/.*test.*"
        delta: 0.5

jest-coverage-diff's Issues

Failed test in base branch fails the build

Hi, I want to report an issue I've found where our base branch has a failed test, and I'm attempting to PR against that in order to fix the test, but when the github action runs for this package, the build fails because the test failed on the base branch.

I'm not sure what the ideal solution is, but allowing to pass an arg that would prevent breaking the build on a failed test in the base branch, if proposed branch test suite was successful? i.e. If not fixed by the PR, they would be caught during the initial run and the build would fail then.

afterSwitchCommand doesn't appear in current version

Hello,
Currently trying to use the repo but am getting an issue here:

Hoping to use this case for when we remove node modules in the pr that are being used in some tests

As a workaround I am installing before and after but that is quite slow :(

Thanks for the awesome project :)

Ability to compare the "total" difference against a different delta value

Hello,

Right now, thedelta value will compare against all entries in the coverage report. This will include the total repo coverage. We would like to be able to have more fine grained control that will allow us to have a larger per file delta, and if the coverage of the whole repo falls below another delta (say total repo delta) to also fail

Thanks so much!

I have created a PR here, hopefully it helps :)

https://github.com/anuraag016/Jest-Coverage-Diff/pull/36/files

proposal: Use `jest.config` to Establish Coverage Threshold for New Files

As a user of jest-coverage-diff I would like to be able to fail a build if new files don't have "expected coverage".

I noticed that this plugin has a great feature that prepends ✨ 🆕. to all new files. Since we seem to already have these files available to us - could we leverage Jest and the jest.config.{ts|js} to fail the build if a given file does not meet the codebases expected thresholds?

Create a workflow that automatically adds new code to build files

Recently made a mistake of not updating built files before publishing a new release, this caused the new version to not have the latest code, causing new features that were supposed to be supported, to not work.

V1.2 assets dont have latest changes

Hi there,

Recently i opened and closed #19 a bit too quickly. After testing on our repo, I have realized that the assets found here: https://github.com/anuraag016/Jest-Coverage-Diff/releases/tag/V1.2
use the same code in the dist folder as in master (https://github.com/anuraag016/Jest-Coverage-Diff/blob/master/dist/index.js#L2027) which does not include the latest changes to support afterSwitchCommand

I suspect the dist folder needs to be updated and then another version published?

My reproduction steps are:

1. Create a PR that deletes a package from package.json
2. Push
3. observe that afterSwitchCommand is not being called

Thanks again for your support!

feat: Use Same Comment for Updates

Hey there! Thanks for building out this coverage diff action!

I was using jest-coverage-reporting-action previously and it was taking an exceptionally long time to run. I gave your action a shot and it was far faster, so 👏 . One thing that was nice though - was their ability to comment back to the same issue. I think that would be a valuable add here so that PRs aren't receiving a new comment with every commit.

Let me know what you think - I'd be happy to try and take a stab at it too if you don't have the bandwidth.

Run failing: 'coverage-summary.json' does not exist

I just started using anuraag016/Jest-Coverage-Diff@master on one of my projects and I'm getting an error with the following output:

Run anuraag016/Jest-Coverage-Diff@master
  with:
    fullCoverageDiff: true
    accessToken: ***
    runCommand: npx jest --coverage --coverageReporters="json-summary" --coverageDirectory="./"
    delta: 100

PASS src/****
PASS src/****
PASS src/****

Test Suites: 3 passed, 3 total
Error: Error: ENOENT: no such file or directory, open 'coverage-summary.json'
Tests:       6 passed, 6 total
Snapshots:   0 total
Time:        5.09 s
Ran all test suites.

It looks like it's trying to open coverage-summary.json before npx jest --coverage finished?

Running npx jest --coverage --coverageReporters="json-summary" --coverageDirectory="./" runs without issue and generates a coverage-summary.json file on the repo's root.

Error: Command failed

I'm trying to use this action in a microservice and I keep getting this error:
Error: Command failed: npx jest --coverage --coverageReporters="json-summary"

Here are my build steps:

steps:
            - name: Checkout repository
              uses: actions/checkout@v2

            - name: Set up Node.js
              uses: actions/setup-node@v1
              with:
                  node-version: ${{ matrix.node-version }}
                  registry-url: 'https://registry.npmjs.org'

            - run: npm ci
              env:
                  NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

            - name: TestCoverage
              id: testCovergae
              uses: anuraag016/Jest-Coverage-Diff@master
              with:
                  fullCoverageDiff: false
                  delta: 0.5

I have node-version as 12.x and platform as ubuntu-latest.
There's not a lot of information in the error message so I'm not sure what else to try.

bug: Deltas Fail PRs when File is Deleted

We've observed on a few occasions that deleted files are causing deltas to fail CI.

❌ | App/My/File.tsx | 89.47 | 69.23 | 100 | 89.47
-- | -- | -- | -- | -- | --

It isn't entirely clear if this is the reason - but it seems likely that the diff between main and branch is marking deleted coverage as a reduced delta.

This may potentially come down to philosophy/semantics. I think technically the term delta would apply correctly here in that it is a net change in reduced coverage. However, I think for the purposes of this plugin the delta metric should be utilized to determine if there is "lost" coverage on "existing files".

I'd be curious to hear thoughts on this one too. I may not being viewing it through the correct lens.

Thanks in advance!

Can't add custom working directory

I'm working on a monolithic project but I'm not able to use this action. Any chance to add it anytime soon?

Fetching base branch fails

Running the action fails when trying to fetch the base branch.

Setup:

Github Actions runs on: ubuntu-latest

- uses: anuraag016/Jest-Coverage-Diff@master
  name: Jest Tests & Coverage Report
  with:
    runCommand: "npx jest --coverage --collectCoverage=true --coverageDirectory='./' --coverageReporters='json-summary'"
    accessToken: ${{ secrets.GITHUB_TOKEN }}
    delta: 10

( the secret secrets.GITHUB_TOKEN is used by multiple workflows and is correct)

Expected Behavior:

It should generate a coverage report

Actual Behavior:

Any idea what that problem could be?
Thanks a lot already!

Errors when Pull Request removes packages

If master has dependency x and the pull request removes x, switching branch to master does not install packages and the dependency x is not available for the master branch.

Adding the possibility of running a command after switching branch would fix this issue, as I would do another npm install / npm ci after switching branch.

"No changes to code coverage between base and head branch" when fullCoverageDiff: false

As a test I have written some additional unit tests in the PR implementing this coverage diff tool but however many unit tests I add I still get the same message.

I run this on an Angular application which I have jest configured for with jest-preset-angular. The coverage is definitely being generated, because when I set fullCoverageDiff to true I get a wall of text with the whole coverage (which unfortunately is just unreadable given the length).

      - name: Run unit tests and check coverage
        uses: anuraag016/Jest-Coverage-Diff@master
        with:
          fullCoverageDiff: false
          runCommand: npm run test:ci
          delta: 0.5

"test:ci": "ng test --ci --silent --bail --coverage --collect-coverage --coverage-directory='./' --coverage-reporters=json-summary",