salesforce / best

:trophy: Delightful Benchmarking & Performance Testing

🏆 Best Performance Benchmarks 🏆

Best allows you to write benchmarks in the same way you write unit tests. This allows you to integrate Best into your CI workflow to create a consistent picture of your code's performance over time.

Reproducible Results: Best is designed to run on dedicated hardware, this means that you are running your benchmarks in the same environment everytime.

Expressive Metrics: Best comes packed with ability to measure all the types of metrics you might want to know.

GitHub Integration: If your team uses GitHub then you can easily create a GitHub App so that you can integrate Best into your Pull Request workflow.

The Best Frontend: Best comes built-in with a frontend dashboard that allows you to monitor your benchmarks over the time with each commit.

Getting Started

Ready to jump in? We recommend starting with The "Best" Manifesto first, and then once you have a solid understanding of how Best works, checking out the Developer Guide to read about how to get started!

Demo

Contributing

To get setup to help contribute to Best, we have a guide for that too.

License

This project is licensed under the MIT license.

best's Issues

Locker Integration breaks with version 4.0.0-alpha10 upwards (Runs fine with 4.0.0-alpha8 & 9)

Locker Integration breaks with version 4.0.0-alpha10 , upwards , with the following error :

Running benchmarks...

   DONE     (benchmark-locker) __benchmarks_results__/benchmark-locker/secureEval.benchmark_local_997a85cae6/artifacts/secureEval.benchmark.html
  ERROR     (benchmark-locker) __benchmarks_results__/benchmark-locker/innerHTML.benchmark_local_117541f8bf/artifacts/innerHTML.benchmark.html
  QUEUED    (benchmark-locker) __benchmarks_results__/benchmark-locker/querySelector.benchmark_local_bfe8983b6b/artifacts/querySelector.benchmark.html

    at ExecutionContext._evaluateInternal (/Users/skundu/lockerservice-core/node_modules/@best/runner-headless/node_modules/puppeteer/lib/ExecutionContext.js:122:13)
    at processTicksAndRejections (internal/process/task_queues.js:89:5)
    at async ExecutionContext.evaluate (/Users/skundu/lockerservice-core/node_modules/@best/runner-headless/node_modules/puppeteer/lib/ExecutionContext.js:48:12)
    at async HeadlessBrowser.evaluate (/Users/skundu/lockerservice-core/node_modules/@best/runner-headless/build/headless.js:80:22)
    at async Runner.runServerIterations (/Users/skundu/lockerservice-core/node_modules/@best/runner-headless/build/index.js:70:38)
    at async Runner.run (/Users/skundu/lockerservice-core/node_modules/@best/runner-headless/build/index.js:26:33)
    at async Object.runBenchmarks (/Users/skundu/lockerservice-core/node_modules/@best/runner/build/index.js:61:38)
    at async Object.runBest (/Users/skundu/lockerservice-core/node_modules/@best/cli/build/run_best.js:87:36)
    at async runCLI (/Users/skundu/lockerservice-core/node_modules/@best/cli/build/cli/index.js:104:25)
    at async Object.run (/Users/skundu/lockerservice-core/node_modules/@best/cli/build/cli/index.js:53:9)
  -- ASYNC --
    at ExecutionContext.<anonymous> (/Users/skundu/lockerservice-core/node_modules/@best/runner-headless/node_modules/puppeteer/lib/helper.js:111:15)
    at DOMWorld.evaluate (/Users/skundu/lockerservice-core/node_modules/@best/runner-headless/node_modules/puppeteer/lib/DOMWorld.js:112:20)
    at processTicksAndRejections (internal/process/task_queues.js:89:5)
  -- ASYNC --
    at Frame.<anonymous> (/Users/skundu/lockerservice-core/node_modules/@best/runner-headless/node_modules/puppeteer/lib/helper.js:111:15)
    at Page.evaluate (/Users/skundu/lockerservice-core/node_modules/@best/runner-headless/node_modules/puppeteer/lib/Page.js:863:43)
    at Page.<anonymous> (/Users/skundu/lockerservice-core/node_modules/@best/runner-headless/node_modules/puppeteer/lib/helper.js:112:23)
    at HeadlessBrowser.evaluate (/Users/skundu/lockerservice-core/node_modules/@best/runner-headless/build/headless.js:80:38)
    at processTicksAndRejections (internal/process/task_queues.js:89:5)
    at async Runner.runServerIterations (/Users/skundu/lockerservice-core/node_modules/@best/runner-headless/build/index.js:70:38)
    at async Runner.run (/Users/skundu/lockerservice-core/node_modules/@best/runner-headless/build/index.js:26:33)
    at async Object.runBenchmarks (/Users/skundu/lockerservice-core/node_modules/@best/runner/build/index.js:61:38)
    at async Object.runBest (/Users/skundu/lockerservice-core/node_modules/@best/cli/build/run_best.js:87:36)
    at async runCLI (/Users/skundu/lockerservice-core/node_modules/@best/cli/build/cli/index.js:104:25)
  -- ASYNC --
    at Page.<anonymous> (/Users/skundu/lockerservice-core/node_modules/@best/runner-headless/node_modules/puppeteer/lib/helper.js:111:15)
    at HeadlessBrowser.evaluate (/Users/skundu/lockerservice-core/node_modules/@best/runner-headless/build/headless.js:80:38)
    at processTicksAndRejections (internal/process/task_queues.js:89:5)
    at async Runner.runServerIterations (/Users/skundu/lockerservice-core/node_modules/@best/runner-headless/build/index.js:70:38)
    at async Runner.run (/Users/skundu/lockerservice-core/node_modules/@best/runner-headless/build/index.js:26:33)
    at async Object.runBenchmarks (/Users/skundu/lockerservice-core/node_modules/@best/runner/build/index.js:61:38)
    at async Object.runBest (/Users/skundu/lockerservice-core/node_modules/@best/cli/build/run_best.js:87:36)
    at async runCLI (/Users/skundu/lockerservice-core/node_modules/@best/cli/build/cli/index.js:104:25)
    at async Object.run (/Users/skundu/lockerservice-core/node_modules/@best/cli/build/cli/index.js:53:9)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
skundu@ ~/lockerservice-core (testing/best-alpha11) $

create an issue there with whatever name

Im testing migrating all the issues

create an issue there with whatever name

Im testing migrating all the issues

[conventional-changelog] use non-lerna conventional changelog

Because lerna auto-increments release version we need to use standard conventional-changelog library.
See more details in lwc repo

Remove yarn.lock from .gitignore

Currently yarn.lock is gitigored, which is defeating the purpose of locking the dependencies.

noop

Foo
Bar

Add option to determine number of iterations based on statistical significance

One nice feature of Benchmark.js is that it chooses the number of iterations to run based on statistical significance. (It seems based on Mann-Whitney 95% confidence, see source here.)

This would be a nice feature to have in Best, rather than having to choose the number of iterations manually.

Create unit tests

We have zero tests right now... :)

Memory usage tests

It would be convenient if Best could measure not only CPU time, but also memory usage. I can envision two types of tests here:

1. Memory leaks – e.g. take a snapshot, repeat some action n times, take another snapshot, diff the two
2. Absolute memory usage – e.g. take a snapshot at a point in time (maybe after a GC+delay) and report it

best CLI requires git to run

Observations

Following the getting started, when running the yarn best command the CLI throws the following error:

ERROR     Error: Unable to read git information
    at generateProjectConfigs (/Users/p.dartus/code/github/pmdartus/benchmark-ci-test/node_modules/@best/config/build/index.js:10:19)
    at readConfig (/Users/p.dartus/code/github/pmdartus/benchmark-ci-test/node_modules/@best/config/build/index.js:68:45)
    at process._tickCallback (internal/process/next_tick.js:68:7)
error Command failed with exit code 1.

Does best should absolutely require to be run in a git enabled directory? If it is the case then the documentation needs to be updated.

Update:
I took me multiple attempts to make the best CLI happy:

• after running git init, best requires at least 1 commit on the master branch.
• after running git commit, best requires at least a remote to be present.
• after adding a remote, best is still not happy here. I created a specific issue for this error #185

Versions

• node: 10.16.0
• best: 4.0.0-alpha4

CI workflow

Description

The CI is configured in 4 stages:

• build: Install dependencies and build the project
• unit test: Run unit tests
• perf_test: Run performance tests against the example project
• compare_perf: Compare the perf tests

The overall pipeline takes 4:30 minutes to run.

In the current architecture:

• The CI spent ~50% of the time unarchiving the installed dependencies generated in the first step.
• The CI keeps running even if the unit test fails.

Proposal

We should simplify the CI pipeline, with only 2 steps:

• build: Which is the merge of the steps: build and unit_tests
• performance: Trigger after the build step is done. This step is merge of perf_test and compare_perf

Extracting best-runtime from the generated artifact

While adding to the generated benchmark the best-runtime source makes the generated artifact stable and pretty stable to run, it makes hard to keep the backward compatibility as we evolve the framework. Once the artifact is generated the framework has no control about it any more.

For example when the runtime get's update we will need to regenerate the artifact to be able to compare apples to apples.

An alternative approach would that the generated artifact doesn't include runtime and then the runtime get inject before the actual benchmark code into the page before running the page. As long as we don't break between releases the testing primitives like benchmark or run, we will be able to update the runtime after the artifact generation.

Warmup runs

One issue we have in some projects (LDS, Locker) is that we see a big perf difference between the first few iterations and the next n. Most likely this is because the JavaScript engine is doing some JITing, so the first few iterations are measuring the unJITed performance.

Ideally, Best would have a setting to do some number of warmup runs, so we could consistently measure peak performance (i.e. JITed performance).

The alternative (to try to measure only un-JITed performance) is technically possible but unfeasible in practice, since it would mean closing the tab and clearing the browser cache between each iteration (in the worst case – it depends on how aggressive browsers are about caching).

Unexpected benchmark run order

Observations

I realized that my mental model with how the tests are executed by best doesn't match with the way the framework actually run.

benchmark('A', () => {
  console.log('benchmark A');

  before(() => console.log('benchmark A - before'));
  run(() => console.log('benchmark A - run'));
});

benchmark('B', () => {
  console.log('benchmark B');

  before(() => console.log('benchmark B - before'));
  run(() => console.log('benchmark B - run'));
});

My mental model was that the console would print in the following order:

benchmark A
benchmark B

N times:
  - benchmark A - before
  - benchmark A - run

N times:
  - benchmark B - before
  - benchmark B - run

While in reality the messages prints in the following order:

benchmark A
benchmark B

N times:
 - benchmark A - before
 - benchmark A - run
 - benchmark B - before
 - benchmark B - run

One of the side effects with the current ordering is that by alternating between the different benchmark, one of the benchmark performance may be impacted by another benchmark. For example, if benchmark A allocates a lot of short-living objects it may cause the benchmark B to stop longer due to unexpected GC.

In the case where each benchmark independently, there would be a lower chance to have side effects between benchmarks.

More details: here

Versions

• node: 10.16.0
• best: 4.0.0-alpha4

feat: add support for table driven testing with benchmark.each()

Add a benchmark.each() similar to jest's it.each() so that scenarios with the same test logic but with different test data can be written without having to repeat code.

In addition to existing primitives such as benchmark.only, benchmark.skip add a benchmark.each(table)(name, fn) to support data/table driven testing.

CLI has vulnerable yargs version

We got flagged on a vulnerability with yargs 11.0.0 in our repo, and I traced it back to the CLI.

Edit:
To clarify, yargs 11.0.0 requires os-locale 2.0.0 which requires mem 1.1.0, which has a known vuln: WS-2018-0236

Clarify behavior for beforeAll/before/beforeEach/afterAll/after/afterEach

I wrote the following benchmark:

Click to see JS file

benchmark(`test`, () => {
  beforeAll(() => {
    console.log('beforeAll');
  });

  before(() => {
    console.log('before');
  });

  beforeEach(() => {
    console.log('beforeEach');
  });

  run(() => {
    console.log('run');
  });

  afterAll(() => {
    console.log('afterAll');
  });

  after(() => {
    console.log('after');
  });

  afterEach(() => {
    console.log('afterEach');
  });
});

 

It produced the following output when running 10 iterations:

Click to see output

beforeAll
before
run
after
afterAll
beforeAll
before
run
after
afterAll
beforeAll
before
run
after
afterAll
beforeAll
before
run
after
afterAll
beforeAll
before
run
after
afterAll
beforeAll
before
run
after
afterAll
beforeAll
before
run
after
afterAll
beforeAll
before
run
after
afterAll
beforeAll
before
run
after
afterAll
beforeAll
before
run
after
afterAll

 

Observations:

1. beforeEach / afterEach appears to do nothing
2. We don't have a way to specify a routine that should run before or after each iteration.

To fix this, should we make beforeEach / afterEach run before and after each iteration? Or should we add beforeEachIteration / afterEachIteration to be clearer?

Testing issues

Foo
Bar

Feature request: ability to measure benchmark deltas across commits/branches

Use case: Locker Service would like to measure the performance of the browser eval() vs Locker secureEval(). We can write Best benchmarks to independently measure each, but there's no way AFAICT to measure the delta between the two, across commits.

So for instance, if eval() itself regresses between Chrome 71 and Chrome 72, we don't care. But if the delta between our secureEval() and eval() regresses, then we actually care. We can track the two independently and infer this, but it would be better if Best had something built-in to handle this use case.

Other use cases that may want this scenario: measuring LWC vs Aura, polyfill vs native, etc.

Update git integration comment for multi-projects

Create table per project rather than a big table with duplicated.

best-cli module needs to be moved into the @best scope

best-cli module should be moved into the @best scope along with the rest of the best packages.

An npm package that consumes best will have a .npmrc file with the following in it:
@best:registry=https://npm.lwcjs.org

which allows the private registry to be used to get the best packages. However this doesn't work for best-cli because it is not in the @best scope.

As a current workaround, a consumer has to directly put the URI of the tarball in their package.json dependencies map.

Add option to avoid rAF-clamping

It looks like Best, by default, clamps all durations to ~16ms using requestAnimationFrame. This may be useful when measuring component rendering times (although arguably it should be a double rAF or a rAF+setTimeout or rAF+postMessage), but for tests that take a small amount of time, it can distort the results:

It seems to me that this has an impact even on Best's own benchmark. Looking at Best results such as these ones, I see that I'm opening a PR which should have no impact on those tests, and yet it's reporting a "trend" of +0.9ms (3.3%) in one test. But the problem is that the "before" duration is 27.00ms and the "after" duration is 27.90ms, and ~16ms of that (i.e. over 50%) is the rAF clamping. So most of this measurement is just the rAF offset.

Should we add an option to turn off the rAF-clamping? Or alternatively, should we get rid of it entirely and allow folks to choose their own method of clamping?

create an issue there with whatever name

Im testing migrating all the issues

Test needs to time out while waiting for a result

Sometimes tests are buggy and result in an async call that never resolves. Currently best just hangs indefinitely if it tries to run such a test.

Instead, best should have a timeout mechanism that will fail the indefinite test and continue on with the other tests.

Branch name convention

In the way, the CI is configured it will only running against master or branches following this pattern /\w+\/.*/. It took me a while to understand why my PR wasn't picked up by the CI.

What about removing the branch naming convention?

Feature request: ability for agent to register multiple specs against hub

Hello,

With the introduction of the best-agent-hub, we lost the ability to run different browser benchmarks against an agent. When running the best-agent standalone, the client is able to specify any browser specs for the agent to run the benchmark; however, with the introduction of agent-hub, we lost that functionality because the agent is only allowed to register one browser specs.

The feature request is to come up with a proper api for the agent to register multiple specs with the hub. The agent should be able to receive any benchmark to execute that match one of its specs. However, only one benchmark can be run against an agent at a time.

Currently the best-agent configuration is the following:

{
    agentConfig: {
        spec: {
            browser: 'chrome',
            version: browserVersion
        },
        host: agentURL,
        options: { path: '/best' },
        remoteRunner: '@best/runner-headless',
        remoteRunnerConfig: {}
    }
}

We may expand it to something like this:

{
    agentConfig: {
        host: agentUrl,
        options: { path: '/best'},
        envType: "AWS_XLARGE_WIN_2016_SERVER"
        specs:[
            {
                browser: 'chrome',
                version: 76,
                defaultRunner: '@best/runner-headless',
                runnerConfig: {
                    '@best/runner-headless': {},
                    '@best/runner-selenium': {}
                }
            }
        ],
        globalRunnerConfig: {
            '@best/runner-headless': {},
            '@best/runner-selenium': {}
        }
    }
}

Because the browser specs dictate the runner type, they should really be grouped together. For example, in the case of chrome, we can run either in headless mode with puppeteer or against Selenium. In the case above, we may be required to specify a defaultRunner if the same browser specs allow multiple runners.

The hub is also supposed to support heterogeneous mixture of agents from all sort of environments, so we may also want to specify an envType so that the client, for example, can choose to run chrome headless on AWS or Heroku. But this is beyond the scope of this feature request.

Best,
-Trinh

Feature request: Best GitHub bot highlights statistically significant results

Right now, the GitHub bot can post a lot of data:

(That screenshot would be several pages long if I posted the whole thing.)

However, it's not terribly clear how to act on this data. As your eye scans the page, a ":-1:" looks just as salient if it regressed 0.2ms as if it regressed 200ms.

What if instead, we put most of that data behind a <details>/<summary> element like so:

Click here for details

• lots of stuff in here
• whew
• so much stuff

 

Then, in the summary, we can surface only the statistically significant information (i.e. p0.05 or below, or whatever we want). That would make it a lot easier for humans to quickly look at the Best bot comment and know whether or not the PR really likely caused a regression or not.

@dbajric has had some success using the Mann-Whitney U test. Whatever stats test we use, it would be nice to use a details/summary to hide the wall of data.

Testing issues

Foo
Bar

Test

Test conventional Commit

Test plugin integration

Webhooks

Local best cli runs on master but no other branches for lwc-example

Summary

I first identified this problem in the Locker repo, but I was able to reproduce it in the Best repo itself.

After checking out the Best repo and building, I did cd packages/lwc-example and ran yarn start and the tests ran fine. The current branch was master.

But when I created a branch foo (from master) and switched, running yarn start yields:

Looking for Best configurations...fatal: No remote configured to list refs from.

[WARN] - Unable to get git information

  ERROR     Error: Unable to read git information
    at generateProjectConfigs (/Users/manuel.jasso/dev/github/best/packages/@best/config/build/index.js:16:19)
    at readConfig (/Users/manuel.jasso/dev/github/best/packages/@best/config/build/index.js:76:45)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

Simply switching back to master and running yarn start again, the tests run fine.

This seems to be related to #184

Versions

• node: 12.16.1
• best: 4.0.0-beta5

chore: Integrate with conventional commits

For when @apapko is bored :)

git remote URL regexp is too restrictive

Observation

This bug is a follow-up from #184. After following the getting started steps and set up the git, the best CLI was still complaining about the following error:

Error: Unable to parse git repo
    at getRepository (/Users/p.dartus/code/github/pmdartus/benchmark-ci-test/node_modules/@best/config/build/utils/git.js:25:15)
    at process._tickCallback (internal/process/next_tick.js:68:7)

The issue is originated from the git remote URI parsing which is not permissive enough.
https://github.com/salesforce/best/blob/master/packages/@best/config/src/utils/git.ts#L29

The regexp only accepts user name and repository names with alphabetic characters, while my repository name contains a dash.

Versions

• node: 10.16.0
• best: 4.0.0-alpha4

Enable CI on every commit

Currently, the CI only runs on branches associated with PRs. Having the CI run on every commit even on branches that are not associated with PR is nice to have. Because build concurrency will not be an issue soon, I think we should enable this feature.

Best fails when best.config.js has only a single project in the projects property

Best fails to execute properly when best.config.js has only a single project in the projects property.

The root best.config.js file in the repro has a list of 4 projects. If you delete it down to one, the config does not get properly processed. The project config doesn't have the proper project name or rootDir.

Error:
`Building benchmarks...

BUILT examples/simple_benchmark/src/benchmarks/object_keys.benchmark.js (perf-best-examples)
BUILT examples/simple_lwc_benchmark/src/simple-item/benchmarks/simple-item.benchmark.js (perf-best-examples)

'simple-item' is imported by examples/simple_lwc_benchmark/src/simple-item/benchmarks/simple-item.benchmark.js, but could not be resolved – treating it as an external dependency
'engine' is imported by examples/simple_lwc_benchmark/src/simple-item/benchmarks/simple-item.benchmark.js, but could not be resolved – treating it as an external dependency
No name was provided for external module 'simple-item' in options.globals – guessing 'Ctor'
No name was provided for external module 'engine' in options.globals – guessing 'engine'
No name was provided for external module 'simple-item' in options.globals – guessing 'Ctor'
No name was provided for external module 'engine' in options.globals – guessing 'engine'

Running benchmarks...

QUEUED perf-best-examples/object_keys.benchmark/object_keys.benchmark.html (perf-best-examples)
QUEUED perf-best-examples/simple-item.benchmark/simple-item.benchmark.html (perf-best-examples)

ERROR Error: Cannot find module 'default'
at Function.Module._resolveFilename (module.js:555:15)
at Function.Module._load (module.js:482:25)
at Module.require (module.js:604:17)
at require (internal/module.js:11:18)
at runBenchmark (/Users/ahoffman/dev/git/ahoffman-best/packages/best-runner/build/index.js:17:18)
at runBenchmarks (/Users/ahoffman/dev/git/ahoffman-best/packages/best-runner/build/index.js:5:36)
at runBundleBenchmarks (/Users/ahoffman/dev/git/ahoffman-best/packages/best-cli/build/run_best.js:61:36)
at runBest (/Users/ahoffman/dev/git/ahoffman-best/packages/best-cli/build/run_best.js:72:40)
at
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.`

Node version incompatibility

The latest release of Best requires running the latest version of Node, but some modules have dependency that is incompatible with the latest node version.

yarn add @best/runner-hub
yarn add v1.19.1
[1/4] 🔍 Resolving packages...
[2/4] 🚚 Fetching packages...
error [email protected]: The engine “node” is incompatible with this module. Expected version “>=10.0.0 <11.0.0". Got “12.13.0”
error Found incompatible module.

Current workaround is to use "--ignore-engines" flag with yarn add.
e.g. "yarn --ignore-engines add @best/runner-hub"

Making @best/cli leaner

Observations

When installing for the first time @best/cli I noticed that it takes around 20 sec. I also realized @best/cli pull-down 151 MB worth of NPM dependencies.

When looking at the generated yarn.lock you can observe that @best/cli it possible to see that some projects like webpack or jest are also pulled down. When running npm ls --json that out of the 3159 dependencies listed, 2508 are pulled down by the @best/frontend.

Is it possible to make the @best/cli dependency on @best/frontend optional?

Reproduction steps

mkdir test
cd test
yarn init -y
yarn add -D @best/cli

Versions

• node: 10.16.0
• best: 4.0.0-alpha4

Disambiguate results coming from API

Add the projectName as part of the cache key on the client to disambiguate the plot results.

Feature Request: Ability to add selectable attributes for test cases to run

Use Case:
I want to be able to select a set of test cases to run based on attributes that I define for each of my test cases.

For example, I'd like to be able to run all Priority=P0 test cases from the command line. This means I need a way to specify the Priority attribute for each test case.

best: command not found

Following the instruction in the README, running best --interactions 3 from best/examples/simple_benchmark/ gives a best: command not found error. This is after running yarn install in the best home dir.

[git] Add pull request template

Test

Yes?

Testing issues

Foo
Bar

Parallelize best-build process

Right now build all the tests in one core, we should utilize as many CPUs as possible.

Typescript

As a proponent of typescript, I need to ask the question. While the codebase is fairly small, does it worth migrating the existing codebase to typescript?

Feature request: Use checks instead of comments for reporting results

I started to play with the Github actions and Github checks APIs. Here is the kind of report I would like to get from Best: https://github.com/pmdartus/github-api-sandbox/pull/3/checks?check_run_id=64567214

Using checks instead of comments has the following benefits:

• checks doesn't pollute the conversations on PRs 🎉
• checks are attached to each commit, while comments are attached to the PR.
  • You can publish the performance results also on all the commits regardless if a branch has a PR associated with or not.
  • You can keep track on the performance numbers even after the PR is closed. This is really usefull for the master branch.
• checks provide a text and a detail field (both support markdown), to publish more structured results.

Add `.github/CODEOWNERS` file

People with admin or owner permissions can set up a CODEOWNERS file in a repository. The people you choose as code owners must have write permissions for the repository.

Code owners are automatically requested for review when someone opens a pull request that modifies code that they own.

Ref / See also: https://help.github.com/en/articles/about-code-owners

Benchmark run fails. Possibly related to puppeteer

Chrome version 80
Recently, one of our tests started failing when it is attempting to run the benchmarks. The error is:
RUNNING (benchmark) .../testTree.benchmark.html
ERROR (benchmark) .../testTree.benchmark.html

ERROR Error: Error running benchmark Error: Error running benchmark Error: Evaluation failed: undefined
at RunnerRemote._triggerBenchmarkError (/mnt/best/node_modules/@best/runner-remote/build/runner-remote.js:170:59)
at RunnerRemote.agent_rejection (/mnt/best/node_modules/@best/runner-remote/build/runner-remote.js:76:14)
at Socket.Emitter.emit (/mnt/best/node_modules/component-emitter/index.js:133:20)
at Socket.onevent (/mnt/best/node_modules/socket.io-client/lib/socket.js:278:10)
at Socket.onpacket (/mnt/best/node_modules/socket.io-client/lib/socket.js:236:12)
at Manager. (/mnt/best/node_modules/component-bind/index.js:21:15)
at Manager.Emitter.emit (/mnt/best/node_modules/component-emitter/index.js:133:20)
at Manager.ondecoded (/mnt/best/node_modules/socket.io-client/lib/manager.js:345:8)
at Decoder. (/mnt/best/node_modules/component-bind/index.js:21:15)
at Decoder.Emitter.emit (/mnt/best/node_modules/component-emitter/index.js:133:20)
error Command failed with exit code 1.

Unexpected result metrics for simple benchmarks

Observations

When running the fibonacci example on the getting started page locally the CLI outputs the following table:

┌─────────────────┬─────────────┬─────┬───────────────┬───────────────┐
│ Benchmark name  │ Metric (ms) │ N   │ Mean ± StdDev │ Median ± MAD  │
├─────────────────┼─────────────┼─────┼───────────────┼───────────────┤
│ js-execution    │ -           │ -   │ -             │ -             │
├─────────────────┼─────────────┼─────┼───────────────┼───────────────┤
│ └─ fibonacci 15 │ script      │ 237 │ 0.097 ± 12.6% │ 0.095 ± 10.5% │
├─────────────────┼─────────────┼─────┼───────────────┼───────────────┤
│ └─ fibonacci 15 │ aggregate   │ 236 │ 0.872 ± 16.2% │ 0.903 ± 5.5%  │
├─────────────────┼─────────────┼─────┼───────────────┼───────────────┤
│ └─ fibonacci 15 │ paint       │ 230 │ 0.032 ± 10.4% │ 0.031 ± 3.2%  │
├─────────────────┼─────────────┼─────┼───────────────┼───────────────┤
│ └─ fibonacci 38 │ script      │ 248 │ 0.077 ± 10.9% │ 0.080 ± 6.3%  │
├─────────────────┼─────────────┼─────┼───────────────┼───────────────┤
│ └─ fibonacci 38 │ aggregate   │ 238 │ 0.306 ± 7.0%  │ 0.305 ± 4.9%  │
└─────────────────┴─────────────┴─────┴───────────────┴───────────────┘

In the case of fibonacci example, the paint and aggregate metrics doesn't make much sense.

It would be great to pass to the benchmark block or the describe block the set of metrics the benchmark if interested by instead of gathering all the metrics by default.

Versions

• node: 10.16.0
• best: 4.0.0-alpha4

Allow regexp benchmark matching

Equivalent to jest regexp matching:
https://facebook.github.io/jest/docs/en/cli.html#jest-regexfortestfiles