tweag / stackage-head

Stackage builds based on GHC HEAD

Stackage builds based on GHC HEAD

This project is an effort to build Stackage with arbitrary GHC development versions and, in particular, with the development HEAD. This allows us to detect regressions during GHC development much faster.

How it works

The process is currently run on Circle CI 4 times a day and can be described as the following progression of steps:

1. Start with the docker container that is used to build Stackage Nightly, snoyberg/stackage:nightly.

2. To avoid re-compiling GHC every time we use build artifacts and some associated metadata provided by ghc-artifact-collector.

3. Download Stackage curator that is used to execute build plans.

4. Reuse a plan from the stackage-nightly repository. These plans are known to build fine, so they are OK for us in most cases, and even if a couple of packages cannot be built it's not a big deal and can be detected as usual (see below).

5. Update, if necessary, the downloaded plan setting source URLs in case we need to use not-yet-released versions of some packages.

6. Execute the chosen build plan and save the build log.

7. Parse the build log and turn it into a build report, store it for future runs.

8. Compare two most recent build reports and detect regressions. Fail if there are suspicious changes (which are necessarily due to some changes in GHC, because we build always with the same build plan, only changing GHC commits).

9. GHC team is notified if the build fails.

Updating the snapshot

The Stackage snapshot used for the builds is updated manually. The reason is that we don't want extra noise and volatility associated with changing snapshots.

Updating the snapshot is usually as simple as editing .circleci/config.yml and changing the line

  BUILD_PLAN: nightly-2018-10-23

to something else.

Occasinally, our docker image also needs to be rebuilt on top of the latest snoyberg/stackage:nightly image—for instance, if a newly added package needs an extra system dependency. The corresponding Dockerfile is at .circleci/images/primary/Dockerfile. The image then needs to be uploaded to Docker Hub and the following line updated in .circleci/config.yml:

  docker:
    - image: rctwg/stackage-head:0.3.2

You can see when snoyberg/stackage:nightly was last updated here.

Blog posts and talks

• The original blog post introducing the initiative
• A talk from HIW'17

License

Copyright © 2018 Tweag I/O

Distributed under BSD 3 clause license.

stackage-head's Issues

URL files that store URLs to CircleCI builds are not truncated

These should be truncated just like CSV files.

Adjust CI workflows to have different jobs for triggered and scheduled builds

We currently use the same setup for triggered and scheduled builds and it's not right. When we trigger a build it should run relatively quickly (10 minutes maybe) and indicate state of a branch (green/red). The scheduled runs should on the other hand actually perform build of a build plan and upload static website with results.

The dashboard should show the date/time when the build was done

Dashboard: split up the Build column

Currently the build column contains the ghc commit hash and the stackage snapshot, but that is not obvious from looking at it. That column should be split into two columns for the commit and snapshot to make it clearer.

Not sure where to put the link to the build info — a separate column?

Font awesome icons are not showed on the site

This is quite minor, but wtf.

If a package is unreachable, the reason should be given

If you are interested in a particular package and it is unreachable, currently the dashboard just says "The build is unreachable.". It would be more informative if the reason was given, e.g. unreachable because of the dependency X.

Succseeding builds are sometimes detected as unreachable

For some reason sometimes succeeding builds with passing test suites are marked as unreachable. This seems to be non-deterministic. Needs investigation. Example:

http://stackage-head.s3-website-us-east-1.amazonaws.com/diff/nightly-2018-04-26-cf35ab9ac7e0f33e39af6af16ecf850e24c2cb79-vs-nightly-2018-04-26-6132d7c5e6404936ef281a6f3be333fea780906e.html

(This is not a permalink though, because we expire old history entries...)

Some builds are perceived as unreachable, incorrectly

For example Early here:

http://stackage-head.s3-website-us-east-1.amazonaws.com/build/nightly-2018-06-14-3606075104e3aa3a3fd9f3a5ca9d314587b03821/Earley/overview.html

Save CSV files with build results as build artifacts for debugging

Sometimes we need to debug Stackage HEAD. To do this currently I need to re-run a build with SSH and then go and copy some files via scp to re-create a situation locally to debug it.

This

1. takes relatively long time
2. requires too much manual intervention to obtain the files of interest.

We can solve this by storing CSV files with Stackage HEAD build results as build artifacts on CircleCI.

Stackage head dashboard has not been updated since Nov 23 last year

Document the steps to update the snapshot

Currently the stackage snapshot needs to be updated manually. We need to automate, simplify, and document this process as much as possible so that we can track the recent nighly snapshots and benefit from the updated/fixed packages.

Also consider always running with the most recent nightly snapshot instead of updating it manually.

Move the site to a domain with more human-readable name

This would be good. This way other people (GHC team for example) can check on Stackage HEAD.

Switch to newer Stackage nightly build plan

I think it's about time to switch to a newer build plan.

Dashboard should show detailed failure information

I.e. the specific compilation error or which dependency is blocking it.

Incorrect detection of status changes for failing packages

Currently failing packages are detected as changed if the number of packages they are blocking changes between two versions of a package (older/newer). This is because we just use a derived Eq instance to compare for change. Even though such changes are considered "innocent" by the system and don't trigger alerts, obviously it is not quite correct.

We also need proper testing of this portion of code. Since it showed up in actual app, this proves that it's a part of the application that should be better tested.

cuda fails intermittently

The cuda package occasionally fails, and when it does, there are no errors in the output.

> /tmp/stackage-build2454$ stack unpack cuda-0.10.0.0@sha256:28b310fe829371a51e48ba4eba26a27b864146d46c96147d824ff3ed2251e71c
Unpacked cuda-0.10.0.0 to /tmp/stackage-build2454/cuda-0.10.0.0/
> /tmp/stackage-build2454/cuda-0.10.0.0$ ghc -clear-package-db -global-package-db -package-db=/home/circleci/project/builds/nightly/pkgdb -hide-all-packages -package=Cabal -package=base -package=directory -package=filepath Setup
[1 of 1] Compiling Main             ( Setup.hs, Setup.o )
Linking Setup ...
> /tmp/stackage-build2454/cuda-0.10.0.0$ ./Setup configure --package-db=clear --package-db=global --package-db=/home/circleci/project/builds/nightly/pkgdb --libdir=/home/circleci/project/builds/nightly/lib --bindir=/home/circleci/project/builds/nightly/bin --datadir=/home/circleci/project/builds/nightly/share --libexecdir=/home/circleci/project/builds/nightly/libexec --sysconfdir=/home/circleci/project/builds/nightly/etc --docdir=/home/circleci/project/builds/nightly/doc/cuda-0.10.0.0 --htmldir=/home/circleci/project/builds/nightly/doc/cuda-0.10.0.0 --haddockdir=/home/circleci/project/builds/nightly/doc/cuda-0.10.0.0
Configuring cuda-0.10.0.0...
Found CUDA toolkit at: /usr/local/cuda-8.0
Storing parameters to cuda.buildinfo.generated
Using build information from 'cuda.buildinfo.generated'.
Provide a 'cuda.buildinfo' file to override this behaviour.
> /tmp/stackage-build2454/cuda-0.10.0.0$ ghc -clear-package-db -global-package-db -package-db=/home/circleci/project/builds/nightly/pkgdb -hide-all-packages -package=Cabal -package=base -package=directory -package=filepath Setup
> /tmp/stackage-build2454/cuda-0.10.0.0$ ./Setup build
Using build information from 'cuda.buildinfo.generated'.
Provide a 'cuda.buildinfo' file to override this behaviour.
Preprocessing library for cuda-0.10.0.0..

Do not execute build plan when the commit is already in history

Oftentimes we run a build for a commit that is already in history. This happens when between two Stackage HEAD builds no new commits appeared in GHC master. Obviously this sort of double-checking cannot provide us with any useful information (it may although give a different result if we have some packages with flaky test suites, but it's not valuable info anyway).

We should first check history file and skip execution of build plan for commits that are already in history .

Security lock error

http://stackage-head.s3-website-us-east-1.amazonaws.com/build/nightly-2018-09-26-2e1df7c1556fc2fb6f7449ae07a382c57a0a536c/th-data-compat/overview.html

user error (hTryLock: lock already exists: /home/circleci/.stack/indices/Hackage/hackage-security-lock)

Implement a dashboard webapp

The webapp should track and expose to the community what is working and what is not and why.

The diff page should have links to older/newer package pages with build/test logs

Currently we have links to entire Stackage HEAD builds. With this design to check on a package I need to go to the list of all packages and then find the package of interest using Ctrl+F and only then I can click on it and read its logs.

Dashboard should allow filtering

E.g. show only the failing packages

Add a build epoch

At the moment stackage-head assumes that our builds are completely determined by the combination of the stackage snapshot and the ghc commit hash, and will ignore the duplicate combinations.

This assumption is sometimes violated: for instance, when using custom package versions (#27) or fixing bugs/changing the algorithm in stackage-head itself.

I propose to add the epoch number — an integer that will be increased every time there's a change not captured by the snapshot or ghc version to let stackage-head know that this is not a duplicate. This is somewhat analogous to the epoch used in debian and rpm packages.

Transition from "build succeeded" to "build unreachable" is considered innocent

In case of stm-delay at the bottom of this diff. The transition from "build succeeded" to "build unreachable" is considered innocent, while it should be detected as suspicious. I'm about to get the data to reproduce this.

Regression test should be added once we fix this.

Add integration tests

These should check that various commands of stackage-head program work as intended.

Refresh Docker image and use fixed stackage-curator

We need to refresh the docker image and in particular use current version of stackage-curator with my fix applied. We currently pull stackage-curator in binary from from some FP Complete-owned bucket I guess. Instead we should build it from source ourselves to be sure which version of the program we'll be using.

Some core packages are failing

At the moment some core packages are failing, including:

• stm (blocking 84)
• primitive (blocking 60)
• memory (blocking 10)

They may be masking some other failures.

We need to investigate why they are failing, whether they are fixed in a newer snapshot or in their git repos. Then make them succeed by either updating the snapshot or using a custom source url.