mmcloughlin / goperf Goto Github PK

With the current terraform-based deployment, it's annoying that the hash of the archive keeps on changing.

Use Stackdriver Monitoring to define some basic metrics, for example:

• Number of tasks in each state
• Coverage: percentage of commits with at least one result, broken down by env/module

Ingest job is taking nearly 5 minutes.

It's written in the least efficient way possible right now. Use batch inserts to speed it up.

Library code that will run benchmarks on a given:

• go commit (pulled down from snapshots)
• package at a given commit
• go test flags

For each module there will be a minimum go commit that it works for. Any earlier commits are bound to fail, so the work scheduler should be able to detect when this has happened and stop scheduling work before that point.

Change queries to use points table instead of results.

Related #38

Make a nice diagram 😄

https://diagrams.mingrammer.com/

• Use only one core (currently using all but one core)
• Final review of which config keys are considered perf-critical

Investigate charges for "Network Inter Zone Egress".

Suspect this is because the SQL database is in a different zone to the cloud functions.

Workers currently write results files to an artifacts directory which isn't cleaned up. This directory could grow indefinitely.

We should ensure that files are cleaned up after they are needed.

Are there any ideas in this paper we can apply?

https://www.ifi.uzh.ch/dam/jcr:ccf1399a-2d57-4ff9-a3b0-59d69616d5d3/msr18-author-version.pdf

Via anp/lolbench#67

Add support for testing the special-case std module.

Use "Stackdriver Trace" for tracing.

https://cloud.google.com/trace/docs/setup/go

Provide a way to view recent tasks in the dashboard UI.

Need a way to expose significant changes in the dashboard UI.

Related #25

Cloud function triggered on new commits added to firestore to enqueue benchmark jobs on pubsub topic.

For some reason a small number of tasks are stuck in the results_uploaded state. At the moment these trigger the stale timeout and are re-processed. This is fine but it might be better to have a secondary process that ingests data files from these tasks.

Currently queried by date, meaning that we get a different number of commits each time. Also the graphs for different environments don't line up.

Make an index page. Highlight top changes.

Provide a view for comparing results from two commits.

Use benchstat for the delta statistics: https://pkg.go.dev/github.com/golang/perf/benchstat

Google analytics for usage stats.

Auto-scaling group of workers subscribed to a pubsub topic, executing benchmarks and writing to storage bucket.

Related #1 #2

There are some clear false positives, or attribution to incorrect commits. Example:

https://goperf.org/bench/778e1e4b-e3b3-550a-83fe-0ac8ba0e13fd?c=42304

The change detector attributed this to commit 42304 whereas the regression is clearly the prior commit.

Make it easy to turn examples like this into test cases for the change package.

We need a way to identify significant changes in benchmark result timeseries.

Currently the logs from watch just say upserted commit <sha> so we don't know whether the object was added or not. It would be good to determine from the firestore response whether an insert happened.

Daniel Marti's "What's coming in Go 1.15" talk highlights known performance improvements in Go 1.15. These serve as excellent examples of known diffs which would hopefully be detectable by continuous benchmarking.

https://docs.google.com/presentation/d/1veyF0y6Ynr6AFzd9gXi4foaURlgbMxM-tmB4StDrdAM/edit#slide=id.g8419a86179_0_37

Inspect these diffs manually and see if performance changes were picked up, and if not why not.

Implement CPU shielding something like: https://github.com/lpechacek/cpuset

Or just use the Ubuntu package.

Display improvement direction on significant changes view.

To support other features I think it's finally necessary to use a commit index, rather than date.

Create and regularly update a commit position table.

• increase cache control max-age
• temporary larger database instance
• increase database max_connections
• turn off workers
• rerun changes job
• crawl the site

Dynamic pages still don't change that often, so we should configure some short-term caching.

The benchmark view pages load very slowly. Investigate why and make it faster.

Package summary page is awful right now. Need to display more information about the benchmarks, since right now you can't even tell which one is which.

The system is detecting clearly spurious changes, for example at a commit that adds a name to the authors file. This is likely paranoia but one potential explanation is that for some reason the computation of commit index is not stable. For my sanity, verify this is not the case.

Graphs such as below suggest that timeseries ordering is not correct.

https://goperf.org/bench/9cdae867-65b9-5e52-96a2-af34ef9c62be

This graph should have a step change in it, but the scatter plot shows the lower "step" continues and overlaps the higher one. Inspecting some of the points suggests that the lower step actually comes from the dev.link branch, which presumably still has an older version of the code. Meanwhile master contains the performance regression, and development is happening concurrently on both branches.

This obscures what's really going on and makes it far harder to pinpoint exact commits responsible for a performance change. We need a more nuanced approach to ordering commits by time.

Figure out which GCP instance type to use.

If a worker process doesn't shut down cleanly, the sys and shield cpusets can be left around. This causes any subsequent attempts to setup a CPU shield to fail.

The shield logic could be made to handle the case where the cpusets already exist.

More descriptive <title> tags.

Need to collect a benchmark suite. Places to look:

• Subrepos
• https://github.com/dr2chase/bent
• https://pkg.go.dev/golang.org/x/benchmarks
• https://twitter.com/davecheney/status/1232431596415803392
• "An Evaluation of Open-Source Software Microbenchmark Suites for Continuous Performance Assessment"
• BigQuery Github Extracts (blog post)

Don't want to accidentally spend thousands of dollars on this.

https://www.terraform.io/docs/providers/google/r/billing_budget.html

Need to fetch environment information and report it as configuration lines in the output.

https://github.com/golang/proposal/blob/master/design/14313-benchmark-format.md

commit: 7cd9055
commit-time: 2016-02-11T13:25:45-0500
goos: darwin
goarch: amd64
cpu: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz
cpu-count: 8
cpu-physical-count: 4
os: Mac OS X 10.11.3
mem: 16 GB

Currently the runner just uses performance scaling governor. Consider pinning frequency instead, similar to perflock.

This project has evolved a lot and some approaches have been abandoned so it makes sense to delete the code. This issue just records which packages have been deleted (or should be soon) in case they are useful again later.

• app/model
• app/mapper
• app/obj
• firestore dependency
• fn/enqueue
• app/consumer
• app/launch
• pubsub dependency
• app/cmd/launcher
• app/gce
• empty firestore db
• delete old buckets
• script/dist.sh
• fn/perfdataupload

Basic github actions CI job.

Module summary page is boring. Perhaps include some more metadata about each package.

Make a basic about page.

• Describe methodology
• Pics of the worker machines?

• nav: "Go Performance Dashboard" heading?
• changes: gap between table and heading is cramped
• changes: friendly units
• changes: top 5 for each
• changes: false positives warning
• modules: sort list
• module view: "Version:" is empty for std
• package view: show module version
• bench graph: default tooltip value: 42,261, 7.11 is confusing
• result view: metadata view partially hidden on overflow
• result view: excess precision
• commit view: show other metadata: author, committer, commit time, ...
• commit view: linkify
• file view: widen