ipfs-inactive / dev-team-enablement

[ARCHIVED] Dev Team Enablement Working Group

[ARCHIVED] Dev Team Enablement Working Group

NOTICE

The Dev Team Enablement Working Group is currently dismantled. Further team dev related issues should go into https://github.com/ipfs/infra

---

---

Dashboards

• Repository Overview: http://ci-sync.cloud.ipfs.team/
• Metrics: https://protocollabs.grafana.net/d/di64ECFiz/ci-ipfs-team?orgId=1

Issue Reporting

All issues can be viewed here: https://github.com/ipfs/testing/issues

The current labels are outlined here: https://github.com/ipfs/testing/labels

You can track the current work by looking at the Dashboard Project: https://github.com/ipfs/testing/projects/2

If you want to open a issue, please try to follow the templates if applicable. You can create a new issue here: https://github.com/ipfs/testing/issues/new/choose

Repositories

Utilities

• https://github.com/ipfs/kubernetes-ipfs - Full Kubernetes deployment containing IPFS
• https://github.com/ipfs/test-lab - InterPlanetary Test Lab
• https://github.com/ipfs/project-repos - Project Health Metrics

Aegir

• https://github.com/ipfs/aegir - Automated JavaScript project building
• https://github.com/ipfs/aegir-test-repo - Repository for testing AEgir changes

Benchmarks/Profiling

• https://github.com/ipfs/benchmark-js.ipfs.io - Benchmarking interface for js-ipfs
• https://github.com/ipfs/ipfs-performance-profiling - Benchmarking tests for js-ipfs, using go-ipfs as a baseline

CI

• https://github.com/ipfs/jenkins-libs - Reusable Jenkins Pipelines
• https://github.com/ipfs/jenkins - Jenkins Infrastructure Setup
• https://github.com/ipfs/ci-helpers - Helper Scripts for CI
• https://github.com/ipfs/ci-sync - Dashboard for Repositories and their CI status

Interop

• https://github.com/ipfs/pdd - Protocol Driven Development
• https://github.com/ipfs/interop - Interoperability tests for IPFS Implementations (Protocol Interop)
• https://github.com/ipfs/interface-ipfs-core - A test suite and interface you can use to implement a IPFS core interface

Contribute

Feel free to join in. All welcome. Open an issue!

This repository and all of our sprint calls fall under the IPFS Code of Conduct.

License

CC-BY-SA

dev-team-enablement's Issues

Setup Jenkinsfile for sharness to pull down go-ipfs to run with sharness

As a first step with sharness, we should add it to Jenkins now when it's under the IPFS organization.

Plan is to:

• Have Jenkins find the latest successful build of go-ipfs
• Clone sharness
• Put the go-ipfs binary in the right place
• Run sharness
• Save the tests results
• Figure out how to set Commit Status on other projects (for when sharness will be triggered from go-ipfs PR, we need to set cross-project Commit Status)

cc @chriscool

Setup a "Golden Standard" for module creation and maintaince

Update all the badges in repos

With the move to Codecov and Jenkins, pretty much every repo has now their badges wrong misinforming users. Can we get a script that does an update like this -- https://github.com/ipfs/js-ipfs-unixfs/pull/31/files -- to all the repos?

Consolidate dev tools from different repos

Seems that there were a bunch of scripts at https://github.com/ipfs/community/tree/master/dev, one of those scripts is now handled by https://github.com/ipfs/pm/tree/master/tools/sync-labels. We need to track better these.

Educate and have guide how to write better tests (Review changes to tests and new additions)

Once coming across PRs that contains changes for tests, make sure to review and educate on how to write better tests.

multiformats compliance tests via cli, language independent

Website deployment preview URL broken

https://ci.ipfs.team/blue/organizations/jenkins/IPFS%2Fjs.ipfs.io/detail/master/5/pipeline

Seems we're getting the organization/repo names wrong and we're getting a 404 when trying to add the status to GitHub

Jenkins supports having secret variables, for code signing and auth tokens

Investigate and reach out to five other projects using IPFS for inclusion in our CI

Move to Node.js 10?

Node.js 10 has been released and it is on track to become LTS.

The changelog contains some notable new additions, for example N-API is no longer experimental. Full changelog here https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V10.md#10.0.0

Shall we update all our testing to use Node.js 10 from now on and enable devs to use it as a baseline for features?

Interop tests

We have some tests designed to guarantee interoperability between the js and go implementations (also these ones to a degree although I don't believe it's their primary purpose).

Are they run as part of CI or the release process anywhere? They should really prevent releases if they fail.

every false-positive build in production tests is tracked

Linked with #25

• Setup process for how to deal with a build failure (not test failure)
• Setup notifications for QA team when build failure happens

Custom npm registry running with jenkins master

The npm registry is still having issues serving packages, returning 404 whenever it feels like it. We should run our own package cache that automatically retries to get packages that gets returned a 404, at least a couple of times.

Would make build times faster but most importantly, make builds more stable.

Currently failing JS projects

The following projects should be checked and observed why they are failing, fixing any issue that is on CI's side rather than on the project itself.

• https://github.com/ipfs/js-datastore-level
• https://github.com/ipfs/js-ipfs
• https://github.com/ipfs/js-ipfs-api
• https://github.com/ipfs/js-ipfs-bitswap
• https://github.com/ipfs/js-ipfs-block-service
• https://github.com/ipfs/js-ipfs-http-response
• https://github.com/ipfs/js-ipfs-mfs
• https://github.com/ipfs/js-ipfs-unixfs
• https://github.com/ipfs/js-ipfs-unixfs-engine
• https://github.com/ipfs/js-ipfsd-ctl
• https://github.com/ipfs/js-ipns
• https://github.com/ipfs/js-level-pull-blob-store
• https://github.com/ipld/js-ipld
• https://github.com/ipld/js-ipld-bitcoin
• https://github.com/ipld/js-ipld-dag-cbor
• https://github.com/ipld/js-ipld-dag-pb
• https://github.com/ipld/js-ipld-git
• https://github.com/ipld/js-ipld-graph-builder
• https://github.com/ipld/js-ipld-raw
• https://github.com/ipld/js-ipld-selector
• https://github.com/ipld/js-ipld-zcash
• https://github.com/libp2p/js-libp2p-crypto
• https://github.com/libp2p/js-libp2p-crypto-secp256k1
• https://github.com/libp2p/js-libp2p-delegated-content-routing
• https://github.com/libp2p/js-libp2p-delegated-peer-routing
• https://github.com/libp2p/js-libp2p-floodsub
• https://github.com/libp2p/js-libp2p-kad-dht
• https://github.com/libp2p/js-libp2p-keychain
• https://github.com/libp2p/js-libp2p-spdy
• https://github.com/libp2p/js-libp2p-tcp
• https://github.com/libp2p/js-libp2p-utp
• https://github.com/libp2p/js-libp2p-webrtc-direct
• https://github.com/libp2p/js-libp2p-webrtc-star
• https://github.com/multiformats/js-multibase

Tracking OKRs 2018 Q2

WIP issue to organize information obtained from team leads of the rest of the WGs

ipfs-cluster (Hector)

• Jenkins needs to handle secrets for code coverage uploading
• kubernetes-ipfs might need support for automation. Also handle secrets
• test-lab, we're both unsure if this falls under DX WG

Allow website jobs to choose hugo version

From ipfs-inactive/docs#68 (comment)

Currently, websites using jenkins to deploy to ipfs cannot chose which hugo version to use.

What we want to enable, is that the website() function in the ipfs/jenkins-libs repository, should accept a hugo_version argument, where projects can specify exactly which hugo version to use.

For this to work, we should tag our ci-websites image with the version from hugo, build and push those to Docker Hub.

every tracked false-positive has a post-mortem published within two weeks

Linked with #26

• When build failure (not test failure) happens, publish post-mortem about build failure

Development flow/process with sharness in it's own repository

Now, with sharness in it's own repository, we need to figure out a development flow that works for everyone. The goal is to allow developers to make changes in sharness without having to update the implementations until they are ready. The idea is similar to how you decouple DB migrations and backend deployments from each other so they can be run when appropriate.

So the current thinking is some flow like this:

• New Feature

  • Add new feature tests in sharness
  • New go-ipfs PRs will run sharness as usual via triggering a build after go-ipfs builds
  • If a test that never passed before is not passing, don't treat this as a test failure[0]
  • If the test have passed before, but is not passing now, it's a failure

• Bugfix

  • If something was fixed as a bugfix (maybe the asserted hash was wrong), we should be able to
    fix it async from other fixes, as to not stop development. Changed tests needs to be marked somehow that they can now fail, but don't treat it as such

• [0] This is to avoid new changes in sharness to now start failing bunch of PRs that are not related to the change at all.

Goal of this issue is to come forward with a process that works for the teams involved (mainly go-ipfs, js-ipfs and rs-ipfs (when that happens))

cc @chriscool @travisperson @diasdavid @whyrusleeping

runs tests for all repositories with code in all orgs

• Label projects with tag for code language
• All those projects are tested

Tracking requests

Is there a way to know if a request was acknowledge and what is the expectation to have it done?

Add the label set to the IPFS in Web Browsers WG repos

As per convo with @lidel.

@victorbjelkholm could you add the label set we use for js-ipfs to the repos that the IPFS in Web Browsers WG looks over? You can find a list at https://waffle.io/ipfs/in-web-browsers

OKR: Test isolation (`interface-ipfs-core`)

https://github.com/ipfs/testing/projects/1#card-9340002

The recent refactor to interface-ipfs-core did a lot to help with improving test isolation. There is currently only a single test which fails to run in isolation from the others (from js-ipfs-api under go-ipfs v0.4.15)

https://github.com/ipfs/interface-ipfs-core/issues/320

It also appears that this is more of an issue with js-ipfs-api as well, it just snuck through as valid because go-ipfs was rejecting the type change of the config key.

CI status badge

We need Jenkins CI badges so that we can replace the Jenkins, Circle and Appveyor on README

Make sure CodeCov comments are disabled

From ipld/js-ipld-dag-pb#78 (comment)

They should not appear by default

all ipfs-api clients are running tests with master/$commit go/js-ipfs

Regular weekly sync?

Might be useful to start having sync meetings so we can ensure that we're making progress on the right things.

@travisperson for now it'll just be us (and interested people), but in the future more people so might be good that we start with it right away. What you think?

Have a transparant process for developers to report time sinks and helping define a plan for reducing them

Better CI context titles

Would it be possible to remove the continuous-integration from the line to make it smaller?

Sharness is abstracted into own project that can run for either go-ipfs or js-ipfs

ipfs-station, ipfs-cluster, peerpad, y.js-connector, pubsub-room are included in integration tests for js-ipfs and go-ipfs

expose runtime support status (badge which includes each platform/config)

go-ipfs and js-ipfs tested on all supported platforms (osx, linux, windows, freebsd, openbsd, 32/64bit, arm)

code coverage everywhere (js repos uses coveralls rather than codecov)

have harness for race condition tests, with four different tests implemented by four different people

AEgir blockers are dealt with in a timely manner (define timely)

Each change in a project runs upstream and downstream integration tests to ensure compatibility

Move proper to here: https://github.com/ipfs/testing

Repository listings

Just a list with all repositories touching QA/Testing

Utilities

• https://github.com/ipfs/kubernetes-ipfs
• https://github.com/ipfs/test-lab
• https://github.com/ipfs/project-repos

Aegir

• https://github.com/ipfs/aegir
• https://github.com/ipfs/aegir-test-repo
• https://github.com/ipfs/jest-environment-aegir
• https://github.com/ipfs/eslint-config-aegir

Benchmarks/Profiling

• https://github.com/ipfs/benchmark-js.ipfs.io
• https://github.com/ipfs/ipfs-performance-profiling

CI

• https://github.com/ipfs/jenkins-libs
• https://github.com/ipfs/jenkins
• https://github.com/ipfs/ci-helpers
• https://github.com/ipfs/ci-sync

Interop

• https://github.com/ipfs/pdd
• https://github.com/ipfs/interop
• https://github.com/ipfs/interface-ipfs-core

Reboot and unify InterPlanetary Contributing Guidelines

Qualitative PDD tests are run automatically for go-ipfs and js-ipfs

Standarized Makefile for go projects

Ref: ipfs/community#276

Continue discussion here

JavaScript pipelines (for coverage, commitlint and codelint) to run nodejs 8.11.3

From ipfs/ipfs-webui#728 (comment)

Add CONTRIBUTING.md to every repo

ref: ipfs/community#241

Please add a `blocked` label as well to all the repos

@victorbjelkholm could you please add a blocked label to all the repos as it was requested during the js-ipfs weekly -- ipfs/js-ipfs#1179 (comment)?

OKR: Reduce complexity (interface-ipfs-core)

https://github.com/ipfs/testing/projects/1#card-9340136

A large refactor of interface-ipfs-core project (ipfs-inactive/interface-js-ipfs-core#290) was merged last week. The new organization of tests make it easier to find test for any given command, as the test structure mimics the js-ipfs / js-ipfs-api command layouts.

I recently opened an issue (https://github.com/ipfs/interface-ipfs-core/issues/313) to build on top of this refactor to help address some of the patterns and conventions around writing tests to help reduce code duplication and code complexity.

Currently seeking feedback before moving forward.

Awaiting sync with @lgierth

Have a plan for upgrading the current package management process

Automate the creation of weekly sync notes

Can we get this -- ipfs/js-ipfs#1179 -- automated?

Yarn vs NPM

During the last js-ipfs Weekly All Hands -- ipfs/js-ipfs#1179 (comment) --, there were a lots of question about Yarn vs NPM (https://youtu.be/CNAPR9HMjR4?t=26m57s).

@victorbjelkholm @travisperson can you expand on the rationale behind it and if it still holds true with the latest npm vs yarn?

Progressive Testing

Does anyone know of a better name / the actual name for this?

---

This relates to #45, interface-ipfs-core, as well as other interface testing projects.

The goal of, what I'm calling here at the moment, progressive testing, is to enable a way to run a test suite that may contain failing tests and still get value out of the results. It also aims to remove implementation specific workarounds inside the interface tests themselves, which current exists quite extensively insight of interface-ipfs-core.

The basic idea is to allow failing tests to not effect the outcome of a test suite as long as they have not been previously proven to have passed.

I think this is important because IPFS and libp2p are build around the idea of interfaces and protocols. These ideas enable a powerful way to build and compose software, but I think it's important to also make it easy to consume and work with the tests.

libp2p

• https://github.com/libp2p/interface-transport
• https://github.com/libp2p/interface-record-store
• https://github.com/libp2p/interface-connection
• https://github.com/libp2p/interface-stream-muxer
• https://github.com/libp2p/interface-peer-routing

ipfs

• https://github.com/ipfs/interface-ipfs-core
• https://github.com/ipfs/interface-datastore
• https://github.com/ipfs/interface-pull-blob-store

Protocol Driven Development https://github.com/ipfs/pdd

The issue at the moment is due to the simplicity of must test runner and reporters. A suite of tests is consider to have failed if any single test fails. Generally communicated through the runners exit code.

When starting to work on a new project this means you are in a continue state of failure till a full implementation has been completed. To get around this issue at the moment we rely on making modifications to the tests themselves for each implementation (see interface-ipfs-core).

One possible solution is to define a new kind of reporter. One that has previous knowledge of which tests have passed prior. Any previously passing test failing results in a failure of the suite. If a test has never passed it is considers skipped.

This enables a project to introduce the entire test suite and incrementally move towards 100% test completion.

This does have a drawback though, as it does require tracking previous state of a tests.

@alanshaw has made some progress with this by allowing more finally controlled execution of tests by breaking things down into smaller sections.

ipfs-inactive/interface-js-ipfs-core#290

This is a great step in the right direction, as it enables projects to incrementally build and introduce subsystems. However, we still run into the issue of a new test being added and breaking implementations that have yet to introduce the new feature.