paritytech / zombienet-sdk

ZombieNet SDK

🚧⚠️ [WIP] ZombieNet SDK ⚠️🚧

Rust Docs

The Vision

This issue will track the progress of the new ZombieNet SDK.

We want to create a new SDK for ZombieNet that allow users to build more complex use cases and interact with the network in a more flexible and programatic way. The SDK will provide a set of building blocks that users can combine in order to spawn and interact (test/query/etc) with the network providing a fluent api to craft different topologies and assertions to the running network. The new SDK will support the same range of providers and configurations that can be created in the current version (v1).

We also want to continue supporting the CLI interface but should be updated to use the SDK under the hood.

The Plan

We plan to divide the work phases to. ensure we cover all the requirement and inside each phase in small tasks, covering one of the building blocks and the interaction between them.

Prototype building blocks

Prototype each building block with a clear interface and how to interact with it

• Building block Network #2
• Building block Node #3
• Building block NodeGroup #4
• Building block Parachain #5
• Building block Collator #6
• Building block CollatorGroup #7
• Building block Assertion #8

Integrate, test interactions and document

We want to integrate the interactions for all building blocks and document the way that they work together.

• Spawning Integration #9
• Assertion Integration #10
• Documentation #11

Refactor CLI and ensure backwards compatibility

Refactor the CLI module to use the new SDK under the hood.

• Refactor CLI #12
• Ensure that spawning from toml works #13
• Ensure that test-runner from DSL works #14

ROADMAP

Infra

• Chaos testing, add examples and explore possibilities in native and podman provider
• Add docker provider
• Add nomad provider
• Create helm chart to allow other use zombienet in k8s
• Auth system to not use k8s users
• Create GitHub Action and publish in NPM marketplace (Completed)
• Rename @paritytech/zombienet npm package to zombienet. Keep all zombienet modules under @zombienet/* org (Completed)

Internal teams

• Add more teams (wip)

Registry

• Create decorators registry and allow override by paras (wip)
• Explore how to get info from paras.

Functional tasks

• Add subxt integration, allow to compile/run on the fly
• Move parser to pest (wip)
• Detach phases and use JSON to communicate instead of paths
• Add relative values assertions (for metrics/scripts)
• Allow to define nodes that are not started in the launching phase and can be started by the test-runner
• Allow to define race assertions
• Rust integration -> Create multiples libs (crates)
• Explore backchannel use case
• Add support to run test agains a running network (wip)
• Add more CLI subcommands
• Add js/subxt snippets ready to use in assertions (e.g transfers)
• Add XCM support in built-in assertions
• Add ink! smart contract support
• Add support to start from a live network (fork-off) [check subalfred]
• Create "default configuration" - (if zombieconfig.json exists in same dir with zombienet then the config applied in it will override the default configuration of zombienet. E.G if user wants to have as default native instead of k8s he can add to

UI

• Create UI to create .zndls and network files.
• Improve VSCode extension (grammar/snippets/syntax highlighting/file validations) (repo)
• Create UI app (desktop) to run zombienet without the need of terminal.

zombienet-sdk's Issues

Shared (`./crates/configuration/shared`)

Component of Configuration Crate

Building block `CollatorGroup`

Docker (`./crates/providers/docker` - features = "docker")

Component of Provider Crate

Test Runner Crate

Responsible for executing tests and reporting their results (./crates/test-runner)
Note: Depends on Support, Configuration crates

• #44
• #45
• #46

[Docker] Add integration test in GHA

Depends on #38

We want then to add some smoke test in GHA for this provider.

Ability to test against bad network conditions

In the future, it would be nice for zombienet to provide an API for injecting network issues.

Zombienet clusters and internal testnets are usually spawned within a single data center, where network speeds are often close to optimal. Unfortunately, in the real world things are not so ideal. Therefore, I'd propose to extend zombienet functionality w/ ability to simulate bad & adversary network conditions. Sort of how Jepsen provides an API for splitting the network into halves (or majorities).

Here is the list of things I think we want in the first approximation:

1. introduce latency, packet loss & jitter (tc & netem on Linux) for all connections of the given peer
2. isolate the single peer from the rest of the network for the given period (iptables on Linux)
3. partition the network into two groups (halves, majority & minority) (iptables on Linux)

(1) can be simplified into multiple predefined options: "slow", "50_loss", etc.

Slightly unrelated, but killing random nodes (chaos engineering; using tools like https://github.com/asobti/kube-monkey) also sound like a good idea to me.

Curious to hear your thoughts on this 🙏

How hard would it be to implement something like this? Are there existing tools for Kubernetes that give you control over network? What zombienet API would look like?

---

UPDATE:

Adding individual issues to track progress:

• #16
• #18
• #17

Support Crate

Responsible for additional capabilities - not specific to our problem space (./crates/support)

• #28
• #29
• #30

Zombienet SDK [Umbrella issue]

The Vision

This issue will track the progress of the new ZombieNet SDK.

We want to create a new SDK for ZombieNet that allow users to build more complex use cases and interact with the network in a more flexible and programatic way.
The SDK will provide a set of building blocks that users can combine in order to spawn and interact (test/query/etc) with the network providing a fluent api to craft different topologies and assertions to the running network. The new SDK will support the same range of providers and configurations that can be created in the current version (v1).

We also want to continue supporting the CLI interface but should be updated to use the SDK under the hood.

The Plan

We plan to divide the work phases to. ensure we cover all the requirement and inside each phase in small tasks, covering one of the building blocks and the interaction between them.

Architecture Proposal

After [team's discussion] (#22) the following architecture was agreed to be followed as a good-initial plan:

## Prototype building blocks
Prototype each building block with a clear interface and how to interact with it

• #2
• #3
• #4
• #5
• #6
• #7
• #8

Building blocks are replaced with the structure depicted in the image above with crates and crate components;
All issues of Prototype building blocks are closed and replaced with the ones mentioned in the next paragraph;

Crates

• #23 : Responsible for additional capabilities - not specific to our problem space (./crates/support)
• #24 : Responsible for creating the configuration and updating it (./crates/configuration)
• #25 : Responsible for the interaction between different providers (./crates/providers)
• #26: Responsible for bringing the state of the running network to the expected configuration and managing it in real-time (./crates/orchestrator)
• #27: Responsible for executing tests and reporting their results (./crates/test-runner)

Integrate, test interactions and document

We want to integrate the interactions for all building blocks and document the way that they work together.

• #9
• #10
• #11

Refactor CLI and ensure backwards compatibility

Refactor the CLI module to use the new SDK under the hood.

• #12
• #13
• #14

Setup `github pipeline`

Ensure that spawning from `toml` works

Proposal of architecture for the SDK

This schema is a proposal of an architecture that could be done for the SDK:

• It favors isolation of components which make testing of units easier
• Definition of abstract/concrete boundaries between crates depending on needs
• Use of features when not all dependencies of the given crate needed
• Easily extendable

Open for your suggestions !

Spawning Integration

Building block `NodeGroup`

Logging (`./crates/support/logging` - features = "logging")

Component of Support Crate

Provisioning the auth service helm chart

In order to automate the package installation for this new service we will turn it into helm charts that easily can be deployed to whatever cluster we want

Partition the network into two groups (halves, majority & minority).

• Add support for partition the network into two groups (halves, majority & minority)

Relaychain (`./crates/orchestrator/relaychain`)

Component of Orchestrator Crate

Ensure that test-runner from `DSL` works

Setup `rust-clippy` for repo

Add Clippy for all crates

• #23
• #24
• #25
• #26
• #27

Kubernetes (`./crates/providers/kubernetes` - features = "kubernetes")

Component of Provider Crate

Building block `Collator`

Parachain (`./crates/configuration/parachain`)

Component of Configuration Crate

[Setup] make `Docker` the default provider for setup

Ones we have docker as provider (#38), we should change the setup to use it as default since is the easiest way to start using zombienet (in both mac/linux).

Podman (`./crates/providers/podman` - features = "podman")

Component of Provider Crate

Assertions (`./crates/test-runner/assertions`)

Component of Test Runner Crate

Reporting (`./crates/test-runner/reporting`)

Component of Test Runner Crate

Runner (`./crates/test-runner/runner`)

Component of Test Runner Crate

Networking (`./crates/support/networking` - features = "networking")

Component of Support Crate

Refactor `CLI`

Configuration Crate

Responsible for creating the configuration to be used by the orchestrator (./crates/configuration)
Note: Depends on Support crate

• #31
• #32
• #33
• #34

Filesystem (`./crates/support/filesystem` - features = "filesystem")

Component of Support Crate

Shared (`./crates/providers/shared`)

Component of Provider Crate

Zombienet SDK PoC (native provider)

The Vision

As part of the building of the new SDK (#1), we want to create a PoC with a subset of features (spawning only) to collect feedback from the internal teams.

The Plan

The subset of features we want to achieve in this Poc:

• #24 : Responsible for creating the configuration (./crates/configuration)
• #26: Responsible for bringing the state of the running network to the expected configuration and managing it in real-time (./crates/orchestrator)
• #37 (provide basic functionality for spawn a network)
• Port some of the examples from https://github.com/paritytech/zombienet/tree/main/examples
• #80

Tasks

Beta Give feedback

1. Configuration Crate #24
   crate +1
2. Orchestrator Crate #26
   crate +1
3. Native (./crates/providers/native - features = "native") #37
   crate component native provider +3
   
4. Port some of the examples from https://github.com/paritytech/zombienet/tree/main/examples
5. [Configuration] API documentation #80
   configuration documentation +2
   

Building block `Network`

Network high-level specs:

• We should allow to use a builder pattern to construct the network and the build method should create all the needed configuration to spawn the network.

• spawn method should check that all the needed artifacts are ready, and if not we should build internally.

• info method should return all the running config info (endpoints/chain-spec paths/ log paths/etc)

• destroy method should tear-down all the network and clean-up all the resources. (we want to allow to save the logs/database for a later analysis)

• rebuild should allow to rebuild the network instance from the same json we get from info.

Provider Crate

Responsible for the interaction between different providers (./crates/providers)
Note: Depends on Support crate

• #35
• #36
• #37
• #38
• #39

Building block `Parachain`

Documentation

Orchestrator Crate

Responsible for bringing the state of the running network to the expected configuration and managing it in real-time (./crates/orchestrator)
Note: Depends on Support, Configuration, Provider crates;

• #40
• #41
• #42
• #43

Simulate bad network conditions.

• Add support to set:

• latency

• packet loss

• jitter

We should support in the available providers:
k8s through chaos-mesh CRD (installed in the cluster)
native ( tc & netem on Linux)
podman (need exploration).

Relaychain (`./crates/configuration/relaychain`)

Component of Configuration Crate

Network (`./crates/configuration/network`)

Component of Configuration Crate

Initial setup for all crates

Create initial setup for all crates

• #23
• #24
• #25
• #26
• #27

Parachain (`./crates/orchestrator/parachain`)

Component of Orchestrator Crate

Native (`./crates/providers/native` - features = "native")

Component of Provider Crate

Shared (`./crates/orchestrator/shared`)

Component of Orchestrator Crate

Building block `Assertion`

Add network `isolation` config.

Allow to isolate the single peer (or a group of peers) from the rest of the network for the given period

Building block `Node`

Represent an individual node in the network topology, inherit from the default_* config of the network (iif is attached to one) and we should have nice defaults to allow run a single node easily.

Assertion Integration

Network (`./crates/orchestrator/network`)

Component of Orchestrator Crate