boa-dev / criterion-compare-action Goto Github PK

View Code? Open in Web Editor NEW

60.0 60.0 25.0 694 KB

⚡️📊 Compare the performance of Rust project branches

License: ISC License

JavaScript 93.19% Dockerfile 6.81%

benchmark criterion github-actions hacktoberfest rust

criterion-compare-action's People

Contributors

Stargazers

Watchers

criterion-compare-action's Issues

Fix calculations

Arises from:
boa-dev/boa#419

Location here:
https://github.com/boa-dev/criterion-compare-action/blob/master/main.js#L88-L116

[BUG] Adding a new benchmark causes a `split` undefined error

Description

Impacted PR: dotenv-linter/dotenv-linter#395

Given an existing benchmark exists, adding a new benchmark function causes an undefined JS error. Output taken from this run:

roup                    changes                                master
-----                    -------                                ------
dotenv_linter check      1.05     65.1±3.38µs        ? ?/sec    1.00     62.2±3.41µs        ? ?/sec
dotenv_linter compare    1.00     27.2±1.46µs        ? ?/sec  
TypeError: Cannot read property 'split' of undefined
    at /home/runner/work/_actions/boa-dev/criterion-compare-action/v2.0.0/dist/index.js:1:46988
    at Array.map (<anonymous>)
    at convertToMarkdown (/home/runner/work/_actions/boa-dev/criterion-compare-action/v2.0.0/dist/index.js:1:46732)
    at main (/home/runner/work/_actions/boa-dev/criterion-compare-action/v2.0.0/dist/index.js:1:46106)
    at async /home/runner/work/_actions/boa-dev/criterion-compare-action/v2.0.0/dist/index.js:1:47904
Error: Unhanded error:
TypeError: Cannot read property 'split' of undefined

Investigating the code and results, I believe the issue is that the output has trailing spaces breaking the undefined checks. A simple .trimRight at this line could fix the issue:

// ORIGINAL
let resultLines = results.split("\n");
// NEW
let resultLines = results.trimRight().split("\n");

Will file a PR for this fix.

Cannot read property of undefined

/home/runner/.cargo/bin/critcmp base changes
group                  base                                   changes
-----                  ----                                   -------
deku_read_bits         1.00    739.4±4.61ns        ? ?/sec    1.20    889.8±0.65ns        ? ?/sec
deku_read_byte         1.00     24.0±0.22ns        ? ?/sec    1.04     24.8±0.35ns        ? ?/sec
deku_read_enum         3.46     39.7±0.29ns        ? ?/sec    1.00     11.5±0.04ns        ? ?/sec
deku_read_vec          28.43 1950.2±19.90ns        ? ?/sec    1.00     68.6±0.14ns        ? ?/sec
deku_read_vec_perf     1.00  1866.4±13.35ns        ? ?/sec  
deku_write_bits        1.05    133.4±0.12ns        ? ?/sec    1.00    127.3±0.05ns        ? ?/sec
deku_write_byte        1.00     72.7±0.81ns        ? ?/sec    2.42    176.0±0.54ns        ? ?/sec
deku_write_enum        1.05    122.1±0.58ns        ? ?/sec    1.00    116.5±0.78ns        ? ?/sec
deku_write_vec         1.27      5.5±0.01µs        ? ?/sec    1.00      4.3±0.00µs        ? ?/sec
deku_write_vec_perf    1.00      5.8±0.01µs        ? ?/sec  
TypeError: Cannot read properties of undefined (reading 'slice')
    at /home/runner/work/_actions/boa-dev/criterion-compare-action/v3/dist/index.js:7:333473
    at Array.map (<anonymous>)
    at convertToMarkdown (/home/runner/work/_actions/boa-dev/criterion-compare-action/v3/dist/index.js:7:333291)
    at main (/home/runner/work/_actions/boa-dev/criterion-compare-action/v3/dist/index.js:7:332471)
    at async /home/runner/work/_actions/boa-dev/criterion-compare-action/v3/dist/index.js:7:334496
Error: Unhanded error:
TypeError: Cannot read properties of undefined (reading 'slice')

`cargo install critcmp` installs if install is cached by CI runner

Perhaps the action could first check if cargo critcmp exists and only if that fails run the installation? Thanks for your help!

Here's the error log:

2023-08-21T03:31:39.5650135Z ##[group]Run boa-dev/criterion-compare-action@v3
2023-08-21T03:31:39.5650424Z with:
2023-08-21T03:31:39.5651056Z   branchName: master
2023-08-21T03:31:39.5651854Z   token: ***
2023-08-21T03:31:39.5651960Z   defaultFeatures: true
2023-08-21T03:31:39.5652052Z ##[endgroup]
2023-08-21T03:31:39.6729665Z [command]/home/runner/.cargo/bin/cargo install critcmp
2023-08-21T03:31:39.7747230Z     Updating crates.io index
2023-08-21T03:31:39.9570372Z error: binary `critcmp` already exists in destination
2023-08-21T03:31:39.9570853Z Add --force to overwrite
2023-08-21T03:31:39.9589680Z Error: The process '/home/runner/.cargo/bin/cargo' failed with exit code 101
2023-08-21T03:31:39.9590262Z     at ExecState._setResult (/home/runner/work/_actions/boa-dev/criterion-compare-action/v3/dist/index.js:1:24691)
2023-08-21T03:31:39.9590917Z     at ExecState.CheckComplete (/home/runner/work/_actions/boa-dev/criterion-compare-action/v3/dist/index.js:1:24251)
2023-08-21T03:31:39.9592134Z     at ChildProcess.<anonymous> (/home/runner/work/_actions/boa-dev/criterion-compare-action/v3/dist/index.js:1:23092)
2023-08-21T03:31:39.9592652Z     at ChildProcess.emit (node:events:513:28)
2023-08-21T03:31:39.9592929Z     at maybeClose (node:internal/child_process:1100:16)
2023-08-21T03:31:39.9593253Z     at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)
2023-08-21T03:31:39.9616420Z ##[error]Unhanded error:
Error: The process '/home/runner/.cargo/bin/cargo' failed with exit code 101

Show benchmarks on direct commits

Would it be possible to show benchmarks on direct commits? Whether it be through a comment on the commit, or Github Actions output stuff?

We don't allow benchmarking only one package

In this commit, we removed the -p Boa from the command. This makes it portable for other things, but it removes the posibility to benchmark a certain package in a workspace.

This is causing all boa builds to fail. We should include an optional "package" parameter in the action configuration, and add it to the command line arguments.

It doesn't create comments from fork projects

Hi there and thank you for this useful project!

I'm facing the problem that this project doesn't create comments in PR's from fork projects.
dotenv-linter/dotenv-linter#425

The output:

/usr/share/rust/.cargo/bin/critcmp master changes
group                            changes                                master
-----                            -------                                ------
dotenv_linter check              1.00     71.1±4.59µs        ? ?/sec    1.00     71.1±3.90µs        ? ?/sec
dotenv_linter compare            1.00     58.5±2.38µs        ? ?/sec    1.02     59.4±2.70µs        ? ?/sec
dotenv_linter fix                1.24  242.3±233.02µs        ? ?/sec    1.00   195.3±15.41µs        ? ?/sec
dotenv_linter fix with backup    1.07  260.3±130.14µs        ? ?/sec    1.00   243.3±20.08µs        ? ?/sec
┌─────────┬─────────────────────────────────┬───────────────────┬─────────────────────┬────────────┐
│ (index) │              name               │  changesDuration  │   masterDuration    │ difference │
├─────────┼─────────────────────────────────┼───────────────────┼─────────────────────┼────────────┤
│    0    │      'dotenv_linter check'      │   '71.1±4.59µs'   │    '71.1±3.90µs'    │   '0.0'    │
│    1    │     'dotenv_linter compare'     │ '**58.5±2.38µs**' │    '59.4±2.70µs'    │   '-2.0'   │
│    2    │       'dotenv_linter fix'       │ '242.3±233.02µs'  │ '**195.3±15.41µs**' │   '+24'    │
│    3    │ 'dotenv_linter fix with backup' │ '260.3±130.14µs'  │ '**243.3±20.08µs**' │   '+7.0'   │
│    4    │               ''                │     undefined     │      undefined      │   '+NaN'   │
└─────────┴─────────────────────────────────┴───────────────────┴─────────────────────┴────────────┘

If the PR is not from a fork project, it works perfectly
dotenv-linter/dotenv-linter#424

The output:

/usr/share/rust/.cargo/bin/critcmp master changes
group                            changes                                master
-----                            -------                                ------
dotenv_linter check              1.01     75.2±5.01µs        ? ?/sec    1.00     74.3±4.73µs        ? ?/sec
dotenv_linter compare            1.05     64.6±4.04µs        ? ?/sec    1.00     61.8±5.43µs        ? ?/sec
dotenv_linter fix                1.00   194.5±16.18µs        ? ?/sec    1.10   213.8±20.74µs        ? ?/sec
dotenv_linter fix with backup    1.00   260.8±24.35µs        ? ?/sec    1.05   273.3±73.96µs        ? ?/sec

Linked commit has the wrong target

The linked commit in the action's comment is incorrect. It links to a commit that says "This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository."

Instead, it should be linking to the commit that triggered the workflow run, which is part of the PR branch.

See this comment: FuelLabs/sway#5546 (comment)

Example of the action's comment: FuelLabs/sway#5546 (comment)

Which links to: FuelLabs/sway@c1daf98

Using nightly rust

Hi I need nightly rust for my project due to some features that are not yet available in stable rust. How would I do this currently?

bug: Wrong column when adding new benchmarks

I just added a new benchmark in a PR, but the PR results appear in the base branch column

But it looks good in the logs

[Feature Request] Update the benchmark comment, don't create a new one

It can sometimes be really frustrating to navigate in a PR with a lot of comments, and adding a new benchmark comment every time you push something does not make it easier.

So I propose the following feature we get the first comment with the benchmarks and every time we push code after that it should update/edit the previous comment so we only have one comment with the benchmark and it will be on the top of the PR (the first benchmark comment).

Question: what if I want to see previous snapshots benchmarks of the code I pushed?
Github allows us to see previous revisions to the comments.

Question: how can we implement this?
I have no idea.

BTW: codecov github action does this (it's possible at least 😃 )

What do you think? @Razican @jasonwilliams

Run custom command between benchmark

I use my benchmarks to test the binaries my code creates, but to do this and test two different versions like this action allows I need to run a command before benchmarking the new branch.

See https://github.com/wcampbell0x2a/backhand/blob/fd4d8fa829de78f023082b8dc27a3ba235872f9a/.github/workflows/benchmark.yml#L15

Implement github action for automaticaly compiling the code

Currently we have to push the feature code as well as the compiled dist, this can be bug prone (and maybe a security issue, if the mangled compiled code is not trusted).

It would be nice if we had continuous deployment that should compile the code on main branch then push, it do another branch with other needed files.

If moving the the dist folder is a problem the we can create a dev branch where we keep the code then push to main brnahc the compiled code.

@Razican what do you think about this?

git checkout error

just wanna let you know, just ran into an error during a git checkout

/usr/bin/git checkout master
error: pathspec 'master' did not match any file(s) known to git
Error: The process '/usr/bin/git' failed with exit code 1
    at ExecState._setResult (/home/runner/work/_actions/boa-dev/criterion-compare-action/v3.2.0/dist/index.js:1:19002)
    at ExecState.CheckComplete (/home/runner/work/_actions/boa-dev/criterion-compare-action/v3.2.0/dist/index.js:1:18562)
    at ChildProcess.<anonymous> (/home/runner/work/_actions/boa-dev/criterion-compare-action/v3.2.0/dist/index.js:1:17403)
    at ChildProcess.emit (node:events:390:28)
    at maybeClose (node:internal/child_process:1064:16)
    at Socket.<anonymous> (node:internal/child_process:450:11)
    at Socket.emit (node:events:390:28)
    at Pipe.<anonymous> (node:net:687:12)
Error: Unhanded error:
Error: The process '/usr/bin/git' failed with exit code 1

https://github.com/felipenoris/hyper-reverse-proxy/runs/6054295355?check_suite_focus=true

Can't seem to control what branch is compared against

My repository uses the nonstandard name "trunk" as the default branch.

When running this action, I see the following output in my Actions logs:

/usr/bin/git checkout master
error: pathspec 'master' did not match any file(s) known to git
Error: The process '/usr/bin/git' failed with exit code 1

The marketplace site and the contents of #28 both suggest that I can use the key branchName to control this behavior. However, if I try to set this, I get the following warning in the Actions logs:

Warning: Unexpected input(s) 'branchName', valid inputs are ['token', 'cwd']

and then the Action proceeds to fail in the same way. Is there a way to override this action so that master is not the branch that's compared against?

I don't know if it makes a difference, but this is a self-hosted runner with runner software v2.285.0 on Rocky Linux 8.5.

The docs on marketplace seemed to disagree whether the correct name was boa-dev/[email protected] or jasonwilliams/criterion-compare-action@move_to_actions. Does this make a difference?

Contents of `pull_request.yml`

name: PR Performance check

on: [pull_request]

jobs:
  runBenchmark:
    name: Criterion Compare PRs
    runs-on: [self-hosted, benchmark]
    steps:
      - name: "Enable Cargo"
        run: echo "${HOME}/.cargo/bin" >> $GITHUB_PATH
      - name: "Check PATH"
        run: echo "$PATH"
      - uses: actions/checkout@v2
      - name: Check Against trunk
        uses: boa-dev/[email protected]
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          branchName: trunk

Full GitHub Actions Log of Failed Run

2021-12-07T05:35:35.1499437Z Found online and idle self-hosted runner in the current repository that matches the required labels: 'self-hosted , benchmark'
2021-12-07T05:35:35.1717478Z Waiting for the runner to pick up this job...
2021-12-07T05:35:35.5255468Z Job is about to start running on the runner: ghastly (repository)
2021-12-07T05:35:39.9256382Z Current runner version: '2.285.0'
2021-12-07T05:35:39.9261205Z Runner name: 'ghastly'
2021-12-07T05:35:39.9261847Z Runner group name: 'Default'
2021-12-07T05:35:39.9263098Z Machine name: 'ghastly'
2021-12-07T05:35:39.9266947Z ##[group]GITHUB_TOKEN Permissions
2021-12-07T05:35:39.9268290Z Actions: write
2021-12-07T05:35:39.9268788Z Checks: write
2021-12-07T05:35:39.9269261Z Contents: write
2021-12-07T05:35:39.9269777Z Deployments: write
2021-12-07T05:35:39.9270308Z Discussions: write
2021-12-07T05:35:39.9270798Z Issues: write
2021-12-07T05:35:39.9271271Z Metadata: read
2021-12-07T05:35:39.9271758Z Packages: write
2021-12-07T05:35:39.9272243Z Pages: write
2021-12-07T05:35:39.9272748Z PullRequests: write
2021-12-07T05:35:39.9273360Z RepositoryProjects: write
2021-12-07T05:35:39.9273987Z SecurityEvents: write
2021-12-07T05:35:39.9274582Z Statuses: write
2021-12-07T05:35:39.9275191Z ##[endgroup]
2021-12-07T05:35:39.9278812Z Secret source: Actions
2021-12-07T05:35:39.9279912Z Prepare workflow directory
2021-12-07T05:35:40.0071110Z Prepare all required actions
2021-12-07T05:35:40.0083158Z Getting action download info
2021-12-07T05:35:40.5182088Z Download action repository 'actions/checkout@v2' (SHA:ec3a7ce113134d7a93b817d10a8272cb61118579)
2021-12-07T05:35:41.3638285Z Download action repository 'boa-dev/[email protected]' (SHA:51d2def841e5e950b6d83e7337baeb13c0bd2b85)
2021-12-07T05:35:44.0420545Z ##[group]Run echo "${HOME}/.cargo/bin" >> $GITHUB_PATH
2021-12-07T05:35:44.0421601Z �[36;1mecho "${HOME}/.cargo/bin" >> $GITHUB_PATH�[0m
2021-12-07T05:35:44.0439513Z shell: /usr/bin/bash -e {0}
2021-12-07T05:35:44.0440052Z ##[endgroup]
2021-12-07T05:35:44.0572490Z ##[group]Run echo "$PATH"
2021-12-07T05:35:44.0573021Z �[36;1mecho "$PATH"�[0m
2021-12-07T05:35:44.0591215Z shell: /usr/bin/bash -e {0}
2021-12-07T05:35:44.0591656Z ##[endgroup]
2021-12-07T05:35:44.0653660Z /home/ghactions/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
2021-12-07T05:35:44.0809561Z ##[group]Run actions/checkout@v2
2021-12-07T05:35:44.0810089Z with:
2021-12-07T05:35:44.0810579Z   repository: chipbuster/fast-hampath
2021-12-07T05:35:44.0811650Z   token: ***
2021-12-07T05:35:44.0812051Z   ssh-strict: true
2021-12-07T05:35:44.0812562Z   persist-credentials: true
2021-12-07T05:35:44.0813054Z   clean: true
2021-12-07T05:35:44.0813446Z   fetch-depth: 1
2021-12-07T05:35:44.0813839Z   lfs: false
2021-12-07T05:35:44.0814225Z   submodules: false
2021-12-07T05:35:44.0814727Z ##[endgroup]
2021-12-07T05:35:44.2259176Z Syncing repository: chipbuster/fast-hampath
2021-12-07T05:35:44.2261145Z ##[group]Getting Git version info
2021-12-07T05:35:44.2262898Z Working directory is '/home/ghactions/actions-runner/_work/fast-hampath/fast-hampath'
2021-12-07T05:35:44.2279974Z [command]/usr/bin/git version
2021-12-07T05:35:44.2337402Z git version 2.27.0
2021-12-07T05:35:44.2363934Z ##[endgroup]
2021-12-07T05:35:44.2371849Z [command]/usr/bin/git config --local --get remote.origin.url
2021-12-07T05:35:44.2410562Z https://github.com/chipbuster/fast-hampath
2021-12-07T05:35:44.2422791Z ##[group]Removing previously created refs, to avoid conflicts
2021-12-07T05:35:44.2426868Z [command]/usr/bin/git rev-parse --symbolic-full-name --verify --quiet HEAD
2021-12-07T05:35:44.2463462Z HEAD
2021-12-07T05:35:44.2473376Z [command]/usr/bin/git rev-parse --symbolic-full-name --branches
2021-12-07T05:35:44.2514025Z ##[endgroup]
2021-12-07T05:35:44.2515637Z ##[group]Cleaning the repository
2021-12-07T05:35:44.2518406Z [command]/usr/bin/git clean -ffdx
2021-12-07T05:35:44.2906877Z Removing Cargo.lock
2021-12-07T05:35:44.2907937Z Removing target/
2021-12-07T05:35:44.2915470Z [command]/usr/bin/git reset --hard HEAD
2021-12-07T05:35:44.3001840Z HEAD is now at 573785c Merge 8da67003bb9e8d202f0f76c85e8e3b81c2e0e45a into 61e0e0827e9cfb3fb8005b38a722b6e832936d92
2021-12-07T05:35:44.3004914Z ##[endgroup]
2021-12-07T05:35:44.3008795Z ##[group]Disabling automatic garbage collection
2021-12-07T05:35:44.3011426Z [command]/usr/bin/git config --local gc.auto 0
2021-12-07T05:35:44.3053686Z ##[endgroup]
2021-12-07T05:35:44.3060235Z ##[group]Setting up auth
2021-12-07T05:35:44.3066893Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2021-12-07T05:35:44.3114527Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2021-12-07T05:35:44.3522170Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2021-12-07T05:35:44.3561431Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2021-12-07T05:35:44.3988507Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2021-12-07T05:35:44.4051435Z ##[endgroup]
2021-12-07T05:35:44.4053017Z ##[group]Fetching the repository
2021-12-07T05:35:44.4055602Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +ee5b8027fb7afe36708e1b55454a6ecebb4e8950:refs/remotes/pull/1/merge
2021-12-07T05:35:44.7445451Z remote: Enumerating objects: 12, done.        
2021-12-07T05:35:44.7456531Z remote: Counting objects:   8% (1/12)        
2021-12-07T05:35:44.7457550Z remote: Counting objects:  16% (2/12)        
2021-12-07T05:35:44.7458102Z remote: Counting objects:  25% (3/12)        
2021-12-07T05:35:44.7458651Z remote: Counting objects:  33% (4/12)        
2021-12-07T05:35:44.7459205Z remote: Counting objects:  41% (5/12)        
2021-12-07T05:35:44.7459752Z remote: Counting objects:  50% (6/12)        
2021-12-07T05:35:44.7460318Z remote: Counting objects:  58% (7/12)        
2021-12-07T05:35:44.7460863Z remote: Counting objects:  66% (8/12)        
2021-12-07T05:35:44.7461418Z remote: Counting objects:  75% (9/12)        
2021-12-07T05:35:44.7462600Z remote: Counting objects:  83% (10/12)        
2021-12-07T05:35:44.7463232Z remote: Counting objects:  91% (11/12)        
2021-12-07T05:35:44.7463772Z remote: Counting objects: 100% (12/12)        
2021-12-07T05:35:44.7466863Z remote: Counting objects: 100% (12/12), done.        
2021-12-07T05:35:44.7475075Z remote: Compressing objects:  33% (1/3)        
2021-12-07T05:35:44.7476068Z remote: Compressing objects:  66% (2/3)        
2021-12-07T05:35:44.7476828Z remote: Compressing objects: 100% (3/3)        
2021-12-07T05:35:44.7477610Z remote: Compressing objects: 100% (3/3), done.        
2021-12-07T05:35:44.7479154Z remote: Total 5 (delta 2), reused 3 (delta 1), pack-reused 0        
2021-12-07T05:35:44.7545096Z From https://github.com/chipbuster/fast-hampath
2021-12-07T05:35:44.7550619Z  + 573785c...ee5b802 ee5b8027fb7afe36708e1b55454a6ecebb4e8950 -> pull/1/merge  (forced update)
2021-12-07T05:35:44.7569294Z ##[endgroup]
2021-12-07T05:35:44.7571103Z ##[group]Determining the checkout info
2021-12-07T05:35:44.7572557Z ##[endgroup]
2021-12-07T05:35:44.7573824Z ##[group]Checking out the ref
2021-12-07T05:35:44.7575552Z [command]/usr/bin/git checkout --progress --force refs/remotes/pull/1/merge
2021-12-07T05:35:44.7627490Z Warning: you are leaving 1 commit behind, not connected to
2021-12-07T05:35:44.7628531Z any of your branches:
2021-12-07T05:35:44.7629013Z 
2021-12-07T05:35:44.7630045Z   573785c Merge 8da67003bb9e8d202f0f76c85e8e3b81c2e0e45a into 61e0e0827e9cfb3fb8005b38a722b6e832936d92
2021-12-07T05:35:44.7630990Z 
2021-12-07T05:35:44.7631710Z If you want to keep it by creating a new branch, this may be a good time
2021-12-07T05:35:44.7632456Z to do so with:
2021-12-07T05:35:44.7633471Z 
2021-12-07T05:35:44.7634936Z  git branch <new-branch-name> 573785c
2021-12-07T05:35:44.7635586Z 
2021-12-07T05:35:44.7636672Z HEAD is now at ee5b802 Merge 596a4a66f025580edbdcf6427bb3ac60dc3ff53f into 61e0e0827e9cfb3fb8005b38a722b6e832936d92
2021-12-07T05:35:44.7638624Z ##[endgroup]
2021-12-07T05:35:44.7676086Z [command]/usr/bin/git log -1 --format='%H'
2021-12-07T05:35:44.7726477Z 'ee5b8027fb7afe36708e1b55454a6ecebb4e8950'
2021-12-07T05:35:44.7852180Z ##[warning]Unexpected input(s) 'branchName', valid inputs are ['token', 'cwd']
2021-12-07T05:35:44.7873086Z ##[group]Run boa-dev/[email protected]
2021-12-07T05:35:44.7873710Z with:
2021-12-07T05:35:44.7875176Z   token: ***
2021-12-07T05:35:44.7875606Z   branchName: trunk
2021-12-07T05:35:44.7876019Z ##[endgroup]
2021-12-07T05:35:44.8594589Z [command]/home/ghactions/.cargo/bin/cargo install critcmp
2021-12-07T05:35:44.9572613Z     Updating crates.io index
2021-12-07T05:35:45.8759845Z      Ignored package `critcmp v0.1.7` is already installed, use --force to override
2021-12-07T05:35:45.8796693Z [command]/home/ghactions/.cargo/bin/cargo bench -- --save-baseline changes
2021-12-07T05:35:46.3711515Z     Updating crates.io index
2021-12-07T05:35:46.8852156Z    Compiling autocfg v1.0.1
2021-12-07T05:35:46.8867109Z    Compiling cfg-if v1.0.0
2021-12-07T05:35:47.0298392Z    Compiling libc v0.2.109
2021-12-07T05:35:47.5136740Z    Compiling lazy_static v1.4.0
2021-12-07T05:35:47.5989996Z    Compiling crossbeam-utils v0.8.5
2021-12-07T05:35:47.6993317Z    Compiling serde v1.0.130
2021-12-07T05:35:48.0026151Z    Compiling semver v1.0.4
2021-12-07T05:35:48.2083540Z    Compiling ryu v1.0.6
2021-12-07T05:35:48.4872017Z    Compiling memchr v2.4.1
2021-12-07T05:35:48.7367977Z    Compiling proc-macro2 v1.0.33
2021-12-07T05:35:48.9239367Z    Compiling crossbeam-epoch v0.9.5
2021-12-07T05:35:49.2991501Z    Compiling scopeguard v1.1.0
2021-12-07T05:35:49.3363023Z    Compiling unicode-xid v0.2.2
2021-12-07T05:35:49.4416539Z    Compiling rayon-core v1.9.1
2021-12-07T05:35:49.5326110Z    Compiling serde_json v1.0.72
2021-12-07T05:35:49.7932395Z    Compiling itoa v0.4.8
2021-12-07T05:35:50.0086478Z    Compiling either v1.6.1
2021-12-07T05:35:50.0148460Z    Compiling ppv-lite86 v0.2.15
2021-12-07T05:35:50.2926871Z    Compiling syn v1.0.82
2021-12-07T05:35:50.5648005Z    Compiling plotters-backend v0.3.2
2021-12-07T05:35:50.8798501Z    Compiling unicode-width v0.1.9
2021-12-07T05:35:51.4183447Z    Compiling regex-automata v0.1.10
2021-12-07T05:35:51.4255823Z    Compiling serde_derive v1.0.130
2021-12-07T05:35:51.5160744Z    Compiling half v1.8.2
2021-12-07T05:35:51.7463574Z    Compiling bitflags v1.3.2
2021-12-07T05:35:51.8747667Z    Compiling typed-arena v2.0.1
2021-12-07T05:35:52.0736914Z    Compiling regex-syntax v0.6.25
2021-12-07T05:35:52.5262231Z    Compiling same-file v1.0.6
2021-12-07T05:35:52.8686061Z    Compiling oorandom v11.1.3
2021-12-07T05:35:53.1023712Z    Compiling memoffset v0.6.5
2021-12-07T05:35:53.5976399Z    Compiling num-traits v0.2.14
2021-12-07T05:35:53.9536337Z    Compiling rayon v1.5.1
2021-12-07T05:35:55.0429176Z    Compiling itertools v0.10.3
2021-12-07T05:35:57.5958477Z    Compiling plotters-svg v0.3.1
2021-12-07T05:35:58.3715437Z    Compiling textwrap v0.11.0
2021-12-07T05:35:59.6600999Z    Compiling walkdir v2.3.2
2021-12-07T05:36:01.5723410Z    Compiling regex v1.5.4
2021-12-07T05:36:13.8882957Z    Compiling clap v2.34.0
2021-12-07T05:36:15.6016244Z    Compiling getrandom v0.2.3
2021-12-07T05:36:16.0892550Z    Compiling num_cpus v1.13.0
2021-12-07T05:36:17.7205574Z    Compiling atty v0.2.14
2021-12-07T05:36:17.8696033Z    Compiling crossbeam-channel v0.5.1
2021-12-07T05:36:19.2936278Z    Compiling rustc_version v0.4.0
2021-12-07T05:36:25.3777745Z    Compiling serde_cbor v0.11.2
2021-12-07T05:36:28.0075751Z    Compiling bstr v0.2.17
2021-12-07T05:36:30.3450204Z    Compiling csv-core v0.1.10
2021-12-07T05:36:31.3752402Z    Compiling quote v1.0.10
2021-12-07T05:36:33.2225839Z    Compiling plotters v0.3.1
2021-12-07T05:36:36.0368115Z    Compiling rand_core v0.6.3
2021-12-07T05:36:36.5523271Z    Compiling cast v0.2.7
2021-12-07T05:36:36.9345691Z    Compiling tinytemplate v1.2.1
2021-12-07T05:36:40.3462221Z    Compiling csv v1.1.6
2021-12-07T05:36:43.1256341Z    Compiling crossbeam-deque v0.8.1
2021-12-07T05:36:43.4001542Z    Compiling rand_chacha v0.3.1
2021-12-07T05:36:47.1505649Z    Compiling rand v0.8.4
2021-12-07T05:36:52.2435990Z    Compiling fast-hampath v0.1.0 (/home/ghactions/actions-runner/_work/fast-hampath/fast-hampath)
2021-12-07T05:36:53.9627807Z    Compiling criterion-plot v0.4.4
2021-12-07T05:36:56.1735846Z    Compiling criterion v0.3.5
2021-12-07T05:37:30.0682838Z     Finished bench [optimized] target(s) in 1m 44s
2021-12-07T05:37:30.0799536Z      Running unittests (target/release/deps/time_benchmarks-347e97521a69467d)
2021-12-07T05:37:30.0830315Z WARNING: HTML report generation will become a non-default optional feature in Criterion.rs 0.4.0.
2021-12-07T05:37:30.0833243Z This feature is being moved to cargo-criterion (https://github.com/bheisler/cargo-criterion) and will be optional in a future version of Criterion.rs. To silence this warning, either switch to cargo-criterion or enable the 'html_reports' feature in your Cargo.toml.
2021-12-07T05:37:30.0835082Z 
2021-12-07T05:37:30.0835899Z Gnuplot not found, using plotters backend
2021-12-07T05:37:30.1361648Z Benchmarking solve n=100
2021-12-07T05:37:30.1362307Z Benchmarking solve n=100: Warming up for 3.0000 s
2021-12-07T05:37:35.3789702Z Benchmarking solve n=100: Collecting 100 samples in estimated 5.1299 s (500 iterations)
2021-12-07T05:37:40.5085700Z Benchmarking solve n=100: Analyzing
2021-12-07T05:37:40.9106389Z solve n=100             time:   [10.258 ms 10.259 ms 10.260 ms]
2021-12-07T05:37:40.9107119Z Found 9 outliers among 100 measurements (9.00%)
2021-12-07T05:37:40.9110185Z   1 (1.00%) low severe
2021-12-07T05:37:40.9123111Z   7 (7.00%) low mild
2021-12-07T05:37:40.9135889Z   1 (1.00%) high severe
2021-12-07T05:37:44.9674501Z 
2021-12-07T05:37:44.9734477Z [command]/usr/bin/git checkout master
2021-12-07T05:37:44.9771296Z error: pathspec 'master' did not match any file(s) known to git
2021-12-07T05:37:44.9785989Z Error: The process '/usr/bin/git' failed with exit code 1
2021-12-07T05:37:44.9787926Z     at ExecState._setResult (/home/ghactions/actions-runner/_work/_actions/boa-dev/criterion-compare-action/v2.0.0/dist/index.js:1:12035)
2021-12-07T05:37:44.9790193Z     at ExecState.CheckComplete (/home/ghactions/actions-runner/_work/_actions/boa-dev/criterion-compare-action/v2.0.0/dist/index.js:1:11597)
2021-12-07T05:37:44.9802910Z     at ChildProcess.<anonymous> (/home/ghactions/actions-runner/_work/_actions/boa-dev/criterion-compare-action/v2.0.0/dist/index.js:1:10445)
2021-12-07T05:37:44.9804501Z     at ChildProcess.emit (events.js:314:20)
2021-12-07T05:37:44.9805400Z     at maybeClose (internal/child_process.js:1022:16)
2021-12-07T05:37:44.9806298Z     at Socket.<anonymous> (internal/child_process.js:444:11)
2021-12-07T05:37:44.9807479Z     at Socket.emit (events.js:314:20)
2021-12-07T05:37:44.9815225Z     at Pipe.<anonymous> (net.js:675:12)
2021-12-07T05:37:44.9821034Z ##[error]Unhanded error:
Error: The process '/usr/bin/git' failed with exit code 1
2021-12-07T05:37:44.9925629Z Post job cleanup.
2021-12-07T05:37:45.1236188Z [command]/usr/bin/git version
2021-12-07T05:37:45.1297427Z git version 2.27.0
2021-12-07T05:37:45.1338764Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2021-12-07T05:37:45.1386215Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2021-12-07T05:37:45.1792230Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2021-12-07T05:37:45.1845701Z http.https://github.com/.extraheader
2021-12-07T05:37:45.1866718Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2021-12-07T05:37:45.1884290Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2021-12-07T05:37:45.2340273Z Cleaning up orphan processes

benchmarks fail

The benchmarks are failing with message:

ReferenceError: difference is not defined

Not working on GitHub enterprise

(Use node --trace-deprecation ... to show where the warning was created)
Warning: Failed to comment: HttpError: request to https://mycompany/api/v3/repos/myorg/myrepo/issues/9/comments failed, reason: tunneling socket could not be established, statusCode=503
Commenting is not possible from forks.
┌─────────┬────────────────────────────┬─

- uses: boa-dev/criterion-compare-action@6fdd4e866a4d4539e126afc158249a7631cc842e
  with:
    benchName: "my_benchmark"
    # Needed. The name of the branch to compare with. This default uses the branch which is being pulled against
    branchName: ${{ github.base_ref }}
    # Optional. Default is `${{ github.token }}`.
    token: ${{ secrets.GITHUB_TOKEN }}

Unrecognized option: 'save-baseline'

When I try to use this extension, I get the following error:

error: Unrecognized option: 'save-baseline'
error: bench failed
Error: The process 'cargo' failed with exit code 101
    at ExecState._setResult (/home/runner/work/_actions/jasonwilliams/criterion-compare-action/move_to_actions/dist/index.js:1:7880)
    at ExecState.CheckComplete (/home/runner/work/_actions/jasonwilliams/criterion-compare-action/move_to_actions/dist/index.js:1:7442)
    at ChildProcess.<anonymous> (/home/runner/work/_actions/jasonwilliams/criterion-compare-action/move_to_actions/dist/index.js:1:6406)
    at ChildProcess.emit (events.js:210:5)
    at maybeClose (internal/child_process.js:1021:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)
Error: Unhanded error:
Error: The process 'cargo' failed with exit code 101

CI run
It seems like it is trying to use an option that doesn't exist.
I tried with criterion 0.3.5 and 0.3.3

Run multiple benches

benchName only supports one bench name, but it is possible to pass --bench multiple times in cargo bench.

I also consider splitting different benches to multiple steps, but it seems each step will generate one comment, which is undesirable.

Cache Baseline

This action should cache the benchmark results and retrieve them for next pull requests instead of rerunning if commits are unchanged.
This should reduce the amount of benchmark runs, especially for the baseline.

Examples:

Checkout to `master` is failing

I have a workflow as follows:

name: Rust benchmarks

on: [pull_request]
jobs:
  runBenchmark:
    name: run benchmark
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@master
      - uses: jasonwilliams/criterion-compare-action@move_to_actions
        with:
          token: ${{ secrets.GITHUB_TOKEN }}

It's failing with:

error: pathspec 'master' did not match any file(s) known to git

Am I missing anything here or could it be a bug?

Error: Unrecognized option: 'save-baseline'

The github action fails with the error "Unrecognized option: 'save-baseline'":

error: Unrecognized option: 'save-baseline'
error: bench failed
Error: The process 'cargo' failed with exit code 101
    at ExecState._setResult (/home/runner/work/_actions/jasonwilliams/criterion-compare-action/move_to_actions/dist/index.js:1:7880)
    at ExecState.CheckComplete (/home/runner/work/_actions/jasonwilliams/criterion-compare-action/move_to_actions/dist/index.js:1:7442)
    at ChildProcess.<anonymous> (/home/runner/work/_actions/jasonwilliams/criterion-compare-action/move_to_actions/dist/index.js:1:6406)
    at ChildProcess.emit (events.js:210:5)
    at maybeClose (internal/child_process.js:1021:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)
Error: Unhanded error:
Error: The process 'cargo' failed with exit code 101

The PR where I tried to add this action: tweag/nickel#333

It seems related to this issue bheisler/criterion.rs#193, but I couldn't extract enough information to see what would be the fix.

Add support for custom cargo profile

Add support for running cargo bench --profile=dist, where dist is a custom profile.

CI fails with error

I am getting this error. Any idea why this may happen?

error: Unrecognized option: 'save-baseline'
error: bench failed
Error: The process '/home/runner/.cargo/bin/cargo' failed with exit code 101
    at ExecState._setResult (/home/runner/work/_actions/boa-dev/criterion-compare-action/v3.0.0/dist/index.js:1:19002)
    at ExecState.CheckComplete (/home/runner/work/_actions/boa-dev/criterion-compare-action/v3.0.0/dist/index.js:1:18562)
    at ChildProcess.<anonymous> (/home/runner/work/_actions/boa-dev/criterion-compare-action/v3.0.0/dist/index.js:1:17403)
    at ChildProcess.emit (node:events:390:28)
    at maybeClose (node:internal/child_process:1064:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)

Error: estimates.json: missing field `Mean`

I've been getting this error, not really sure why.

Workflow:
https://github.com/sharksforarms/rust-packet/blob/master/.github/workflows/pull_request.yml

CI:
https://github.com/sharksforarms/rust-packet/pull/25/checks?check_run_id=865820168

/usr/share/rust/.cargo/bin/critcmp master changes
/home/runner/work/rust-packet/rust-packet/target/criterion/bench_raw_from_bytes/new: estimates.json: missing field `Mean` at line 1 column 984
Error: The process '/usr/share/rust/.cargo/bin/critcmp' failed with exit code 1
    at ExecState._setResult (/home/runner/work/_actions/boa-dev/criterion-compare-action/master/dist/index.js:1:12035)
    at ExecState.CheckComplete (/home/runner/work/_actions/boa-dev/criterion-compare-action/master/dist/index.js:1:11597)
    at ChildProcess.<anonymous> (/home/runner/work/_actions/boa-dev/criterion-compare-action/master/dist/index.js:1:10445)
    at ChildProcess.emit (events.js:210:5)
    at maybeClose (internal/child_process.js:1021:16)
    at Socket.<anonymous> (internal/child_process.js:430:11)
    at Socket.emit (events.js:210:5)
    at Pipe.<anonymous> (net.js:659:12)
##[error]Unhanded error:
Error: The process '/usr/share/rust/.cargo/bin/critcmp' failed with exit code 1

Benchmarks with throughput aren't displayed correctly

It looks like the format for benchmarks with throughput isn't parsed correctly, since they are not (always) separated with two spaces anymore. This can be seen in Ogeon/palette#228, where I try to upgrade to this version of the action.

TypeError: Converting circular structure to JSON

TypeError: Converting circular structure to JSON
    --> starting at object with constructor 'TLSSocket'
    |     property '_httpMessage' -> object with constructor 'ClientRequest'
    --- property 'socket' closes the circle
    at JSON.stringify (<anonymous>)
    at Object.toCommandValue (/home/runner/work/_actions/boa-dev/criterion-compare-action/master/dist/index.js:1:7497)
    at escapeData (/home/runner/work/_actions/boa-dev/criterion-compare-action/master/dist/index.js:1:3737)
    at Command.toString (/home/runner/work/_actions/boa-dev/criterion-compare-action/master/dist/index.js:1:3668)
    at Object.issueCommand (/home/runner/work/_actions/boa-dev/criterion-compare-action/master/dist/index.js:1:3161)
    at Object.debug (/home/runner/work/_actions/boa-dev/criterion-compare-action/master/dist/index.js:1:5958)
    at main (/home/runner/work/_actions/boa-dev/criterion-compare-action/master/dist/index.js:1:1119)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
Error: Unhanded error:
TypeError: Converting circular structure to JSON
    --> starting at object with constructor 'TLSSocket'
    |     property '_httpMessage' -> object with constructor 'ClientRequest'
    --- property 'socket' closes the circle

I'm getting this error. Is this a bug?
ref - https://github.com/smrpn/casbin-rs/pull/2/checks?check_run_id=2268954721

Workspaces support?

It would be great if this supported the new --workspace option.

boa-dev / criterion-compare-action Goto Github PK

criterion-compare-action's People

Contributors

Stargazers

Watchers

Forkers

criterion-compare-action's Issues

Description

Recommend Projects

Recommend Topics

Recommend Org