ipfs-inactive / dev-team-enablement Goto Github PK

Now, with sharness in it's own repository, we need to figure out a development flow that works for everyone. The goal is to allow developers to make changes in sharness without having to update the implementations until they are ready. The idea is similar to how you decouple DB migrations and backend deployments from each other so they can be run when appropriate.

So the current thinking is some flow like this:

• New Feature

  • Add new feature tests in sharness
  • New go-ipfs PRs will run sharness as usual via triggering a build after go-ipfs builds
  • If a test that never passed before is not passing, don't treat this as a test failure[0]
  • If the test have passed before, but is not passing now, it's a failure

• Bugfix

  • If something was fixed as a bugfix (maybe the asserted hash was wrong), we should be able to
    fix it async from other fixes, as to not stop development. Changed tests needs to be marked somehow that they can now fail, but don't treat it as such

• [0] This is to avoid new changes in sharness to now start failing bunch of PRs that are not related to the change at all.

Goal of this issue is to come forward with a process that works for the teams involved (mainly go-ipfs, js-ipfs and rs-ipfs (when that happens))

cc @chriscool @travisperson @diasdavid @whyrusleeping

During the last js-ipfs Weekly All Hands -- ipfs/js-ipfs#1179 (comment) --, there were a lots of question about Yarn vs NPM (https://youtu.be/CNAPR9HMjR4?t=26m57s).

@victorbjelkholm @travisperson can you expand on the rationale behind it and if it still holds true with the latest npm vs yarn?

Ref: ipfs/community#276

Continue discussion here

https://github.com/ipfs/testing/projects/1#card-9340002

The recent refactor to interface-ipfs-core did a lot to help with improving test isolation. There is currently only a single test which fails to run in isolation from the others (from js-ipfs-api under go-ipfs v0.4.15)

https://github.com/ipfs/interface-ipfs-core/issues/320

It also appears that this is more of an issue with js-ipfs-api as well, it just snuck through as valid because go-ipfs was rejecting the type change of the config key.

https://ci.ipfs.team/blue/organizations/jenkins/IPFS%2Fjs.ipfs.io/detail/master/5/pipeline

Seems we're getting the organization/repo names wrong and we're getting a 404 when trying to add the status to GitHub

Just a list with all repositories touching QA/Testing

Utilities

• https://github.com/ipfs/kubernetes-ipfs
• https://github.com/ipfs/test-lab
• https://github.com/ipfs/project-repos

Aegir

• https://github.com/ipfs/aegir
• https://github.com/ipfs/aegir-test-repo
• https://github.com/ipfs/jest-environment-aegir
• https://github.com/ipfs/eslint-config-aegir

Benchmarks/Profiling

• https://github.com/ipfs/benchmark-js.ipfs.io
• https://github.com/ipfs/ipfs-performance-profiling

CI

• https://github.com/ipfs/jenkins-libs
• https://github.com/ipfs/jenkins
• https://github.com/ipfs/ci-helpers
• https://github.com/ipfs/ci-sync

Interop

• https://github.com/ipfs/pdd
• https://github.com/ipfs/interop
• https://github.com/ipfs/interface-ipfs-core

From ipfs-inactive/docs#68 (comment)

Currently, websites using jenkins to deploy to ipfs cannot chose which hugo version to use.

What we want to enable, is that the website() function in the ipfs/jenkins-libs repository, should accept a hugo_version argument, where projects can specify exactly which hugo version to use.

For this to work, we should tag our ci-websites image with the version from hugo, build and push those to Docker Hub.

As per convo with @lidel.

@victorbjelkholm could you add the label set we use for js-ipfs to the repos that the IPFS in Web Browsers WG looks over? You can find a list at https://waffle.io/ipfs/in-web-browsers

The following projects should be checked and observed why they are failing, fixing any issue that is on CI's side rather than on the project itself.

• https://github.com/ipfs/js-datastore-level
• https://github.com/ipfs/js-ipfs
• https://github.com/ipfs/js-ipfs-api
• https://github.com/ipfs/js-ipfs-bitswap
• https://github.com/ipfs/js-ipfs-block-service
• https://github.com/ipfs/js-ipfs-http-response
• https://github.com/ipfs/js-ipfs-mfs
• https://github.com/ipfs/js-ipfs-unixfs
• https://github.com/ipfs/js-ipfs-unixfs-engine
• https://github.com/ipfs/js-ipfsd-ctl
• https://github.com/ipfs/js-ipns
• https://github.com/ipfs/js-level-pull-blob-store
• https://github.com/ipld/js-ipld
• https://github.com/ipld/js-ipld-bitcoin
• https://github.com/ipld/js-ipld-dag-cbor
• https://github.com/ipld/js-ipld-dag-pb
• https://github.com/ipld/js-ipld-git
• https://github.com/ipld/js-ipld-graph-builder
• https://github.com/ipld/js-ipld-raw
• https://github.com/ipld/js-ipld-selector
• https://github.com/ipld/js-ipld-zcash
• https://github.com/libp2p/js-libp2p-crypto
• https://github.com/libp2p/js-libp2p-crypto-secp256k1
• https://github.com/libp2p/js-libp2p-delegated-content-routing
• https://github.com/libp2p/js-libp2p-delegated-peer-routing
• https://github.com/libp2p/js-libp2p-floodsub
• https://github.com/libp2p/js-libp2p-kad-dht
• https://github.com/libp2p/js-libp2p-keychain
• https://github.com/libp2p/js-libp2p-spdy
• https://github.com/libp2p/js-libp2p-tcp
• https://github.com/libp2p/js-libp2p-utp
• https://github.com/libp2p/js-libp2p-webrtc-direct
• https://github.com/libp2p/js-libp2p-webrtc-star
• https://github.com/multiformats/js-multibase

From ipld/js-ipld-dag-pb#78 (comment)

They should not appear by default

From ipfs/ipfs-webui#728 (comment)

Node.js 10 has been released and it is on track to become LTS.

The changelog contains some notable new additions, for example N-API is no longer experimental. Full changelog here https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V10.md#10.0.0

Shall we update all our testing to use Node.js 10 from now on and enable devs to use it as a baseline for features?

• Label projects with tag for code language
• All those projects are tested

ref: ipfs/community#241

Seems that there were a bunch of scripts at https://github.com/ipfs/community/tree/master/dev, one of those scripts is now handled by https://github.com/ipfs/pm/tree/master/tools/sync-labels. We need to track better these.

Linked with #25

• Setup process for how to deal with a build failure (not test failure)
• Setup notifications for QA team when build failure happens

Does anyone know of a better name / the actual name for this?

This relates to #45, interface-ipfs-core, as well as other interface testing projects.

The goal of, what I'm calling here at the moment, progressive testing, is to enable a way to run a test suite that may contain failing tests and still get value out of the results. It also aims to remove implementation specific workarounds inside the interface tests themselves, which current exists quite extensively insight of interface-ipfs-core.

The basic idea is to allow failing tests to not effect the outcome of a test suite as long as they have not been previously proven to have passed.

I think this is important because IPFS and libp2p are build around the idea of interfaces and protocols. These ideas enable a powerful way to build and compose software, but I think it's important to also make it easy to consume and work with the tests.

libp2p

• https://github.com/libp2p/interface-transport
• https://github.com/libp2p/interface-record-store
• https://github.com/libp2p/interface-connection
• https://github.com/libp2p/interface-stream-muxer
• https://github.com/libp2p/interface-peer-routing

ipfs

• https://github.com/ipfs/interface-ipfs-core
• https://github.com/ipfs/interface-datastore
• https://github.com/ipfs/interface-pull-blob-store

Protocol Driven Development https://github.com/ipfs/pdd

The issue at the moment is due to the simplicity of must test runner and reporters. A suite of tests is consider to have failed if any single test fails. Generally communicated through the runners exit code.

When starting to work on a new project this means you are in a continue state of failure till a full implementation has been completed. To get around this issue at the moment we rely on making modifications to the tests themselves for each implementation (see interface-ipfs-core).

One possible solution is to define a new kind of reporter. One that has previous knowledge of which tests have passed prior. Any previously passing test failing results in a failure of the suite. If a test has never passed it is considers skipped.

This enables a project to introduce the entire test suite and incrementally move towards 100% test completion.

This does have a drawback though, as it does require tracking previous state of a tests.

@alanshaw has made some progress with this by allowing more finally controlled execution of tests by breaking things down into smaller sections.

ipfs-inactive/interface-js-ipfs-core#290

This is a great step in the right direction, as it enables projects to incrementally build and introduce subsystems. However, we still run into the issue of a new test being added and breaking implementations that have yet to introduce the new feature.

Would it be possible to remove the continuous-integration from the line to make it smaller?

The npm registry is still having issues serving packages, returning 404 whenever it feels like it. We should run our own package cache that automatically retries to get packages that gets returned a 404, at least a couple of times.

Would make build times faster but most importantly, make builds more stable.

Might be useful to start having sync meetings so we can ensure that we're making progress on the right things.

@travisperson for now it'll just be us (and interested people), but in the future more people so might be good that we start with it right away. What you think?

Is there a way to know if a request was acknowledge and what is the expectation to have it done?

We need Jenkins CI badges so that we can replace the Jenkins, Circle and Appveyor on README

With the move to Codecov and Jenkins, pretty much every repo has now their badges wrong misinforming users. Can we get a script that does an update like this -- https://github.com/ipfs/js-ipfs-unixfs/pull/31/files -- to all the repos?

https://github.com/ipfs/testing/projects/1#card-9340136

A large refactor of interface-ipfs-core project (ipfs-inactive/interface-js-ipfs-core#290) was merged last week. The new organization of tests make it easier to find test for any given command, as the test structure mimics the js-ipfs / js-ipfs-api command layouts.

I recently opened an issue (https://github.com/ipfs/interface-ipfs-core/issues/313) to build on top of this refactor to help address some of the patterns and conventions around writing tests to help reduce code duplication and code complexity.

Currently seeking feedback before moving forward.

@victorbjelkholm could you please add a blocked label to all the repos as it was requested during the js-ipfs weekly -- ipfs/js-ipfs#1179 (comment)?

As a first step with sharness, we should add it to Jenkins now when it's under the IPFS organization.

Plan is to:

• Have Jenkins find the latest successful build of go-ipfs
• Clone sharness
• Put the go-ipfs binary in the right place
• Run sharness
• Save the tests results
• Figure out how to set Commit Status on other projects (for when sharness will be triggered from go-ipfs PR, we need to set cross-project Commit Status)

cc @chriscool

Linked with #26

• When build failure (not test failure) happens, publish post-mortem about build failure

WIP issue to organize information obtained from team leads of the rest of the WGs

ipfs-cluster (Hector)

• Jenkins needs to handle secrets for code coverage uploading
• kubernetes-ipfs might need support for automation. Also handle secrets
• test-lab, we're both unsure if this falls under DX WG

Can we get this -- ipfs/js-ipfs#1179 -- automated?

Once coming across PRs that contains changes for tests, make sure to review and educate on how to write better tests.

We have some tests designed to guarantee interoperability between the js and go implementations (also these ones to a degree although I don't believe it's their primary purpose).

Are they run as part of CI or the release process anywhere? They should really prevent releases if they fail.