strangelove-ventures / interchaintest

e2e testing framework for the interchain

interchaintest

Formerly known as ibctest.

🌌 Why use interchaintest?

In order to ship production-grade software for the Interchain, we needed sophisticated developer tooling...but IBC and Web3 have a lot of moving parts, which can lead to a steep learning curve and all sorts of pain. Recognize any of these?

• repeatedly building repo-specific, Docker- and shell-based testing solutions,
• duplication of effort, and
• difficulty in repurposing existing testing harnesses for new problem domains.

We built interchaintest to extract patterns and create a generic test harness: a use-case-agnostic framework for generating repeatable, diagnostic tests for every aspect of IBC.

Read more at the Announcing interchaintest blog post.

🌌🌌 Who benefits from interchaintest?

interchaintest is for developers who expect top-shelf testing tools when working on blockchain protocols such as Cosmos or Ethereum.

🌌🌌🌌 What does interchaintest do?

interchaintest is a framework for testing blockchain functionality and interoperability between chains, primarily with the Inter-Blockchain Communication (IBC) protocol.

Want to quickly spin up custom testnets and dev environments to test IBC, Relayer setup, chain infrastructure, smart contracts, etc.? interchaintest orchestrates Go tests that utilize Docker containers for multiple IBC-compatible blockchains.

🌌🌌🌌🌌 How do I use it?

As a Module

Most people choose to import interchaintest as a module.

• Often, teams will integrate interchaintest with a github CI/CD pipeline.
• Most teams will write their own suite. Here's a tutorial on Writing Custom Tests.
• You can also utilize our suite of built-in Conformance Tests that exercise high-level IBC compatibility.

As a Binary

There's also an option to build and run interchaintest as a binary (which might be preferable, e.g., with custom chain sets). You can still run Conformance Tests.

References

• Environment Variable Options
• Retaining Data on Failed Tests

🌌🌌🌌🌌🌌 Extras

Maintained Branches

Branch Name	IBC-Go	Cosmos-sdk
main	v8	v0.50
v7	v7	v0.47

Deprecated Branches

These are branches that we no longer actively update or maintain but may be of use if a chain is running older versions of the Cosmos SDK or IBC Go. Please see the Backport Policy below.

Branch Name	IBC-Go	Cosmos-sdk	Deprecated Date
v6	v6	v0.46	Sept 5 2023
v5	v5	v0.46	Aug 11 2023
v4	v4	v0.45	Aug 11 2023
v4-ics	v4	v0.45.x-ics	Aug 11 2023
v3	v3	v0.45	June 25 2023
v3-ics	v3	v0.45.11-ics	April 24 2023

Backport Policy:

Strangelove maintains n and n - 1 branches of interchaintest, n being current main.

We strive to keep interchaintest inline with the latest from the ibc-go and cosmos sdk teams. Once an alpha versions of the next major ibc-go version is released, we will discontinue n - 1 and branch off a new n.

Recommendation: Even if your chain uses an older version of ibc-go, try importing from main. This should work unless you are decoding transactions that require a specific ibc-go version.

If there is a feature you would like backported to an older branch, make an issue! We are happy to work with you.

Contributing

Contributing is encouraged.

Please read the logging style guide.

Trophies

Significant bugs that were more easily fixed with interchaintest:

• Juno network halt reproduction
• Juno network halt fix confirmation

interchaintest's Issues

Requirements from Quasar team

Multi-chain setup

At this moment, the test framework setup takes a pair of chains (source and destination chains).
This is enough most of the times but becomes a blocker for more complex scenarios.

An example involving 3 chains (A, B, C): an IBC denom, originally from ChainA, needs to be transferred from ChainB to ChainC. The ChainA needs to be used as pass-through using the packet-forwarder feature so that the denon is valid in the destination ChainC.

Ideally, the ibc-test-framework should take an arbitrary number of blockchains as an input for the test setup.
All chains will be started, same as now, and it is for the developer to orchestrate the communication between these chains and test accordingly.

Note that we could also imagine future scenarios where multiple relayers are needed.

Changes

The main change is in the test setup.

ibctest/test_setup.go

func StartChainsAndRelayerFromFactory(
	t *testing.T,
	ctx context.Context,
	pool *dockertest.Pool,
	networkID string,
	home string,
	srcChain, dstChain ibc.Chain,
	f RelayerFactory,
	preRelayerStart func([]ibc.ChannelOutput, User, User) error,
) (ibc.Relayer, []ibc.ChannelOutput, *User, *User, error)

It currently takes two chains but should accept an arbitrary number, or a wrapping structure containing the various chains to be started.

Interchain Accounts support

The Interchain accounts feature is currently missing in the ibc-test-framework. Only the SendIBCTransfer is available on the type Chain interface.

The test framework should allow the developer to configure their ICA keepers accordingly and access the connections IDs necessary to start sending ICA packets.

Changes

A way to access the IBC connection IDs is needed to send ICA packets.
A new IBCConnection structure could be introduced containing more than what is currently in the ChannelOutput.
The appropriate genesis params will have to be set on the host chain (like the allow_list for the ICA messages).

Other

• Ability to use locally built docker image

Add tests to assert height timeout behavior

With height timeout, you do not get an acknowledgement message and therefore, cannot query for acks.

Instead, the relayer writes a different tx message: /ibc.core.channel.v1.MsgTimeout

Fee middleware should have assertions.

See cosmos/ibc-go#396 for ibc-go's manual QA test plan.

As of this writing, this story is likely blocked because:

• Still in active development for ibc-go.
• cosmos/relayer support

Assert inclusion proof of packet data commitment in the outgoing chain's state

From the IBC spec on receiving packets: https://github.com/cosmos/ibc/blob/master/spec/core/ics-004-channel-and-packet-semantics/README.md#receiving-packets

Checks the inclusion proof of packet data commitment in the outgoing chain's state

Namely, the chain should protect itself from a malicious relayer.

This may be related to "misbehavior" as present in the IBC spec. However, as of this writing, misbehavior is not implemented in the go relayer.

Interchaintest should have a test for multi-hop.

cosmos/ibc#548

All chains from Chain-Reg should be added to the test matrix.

• bitcanna
• cosmoshub
• firmachain
• likecoin
• persistence
• stargaze
• bitsong
• cronos
• gravitybridge
• lumnetwork
• provenance
• starname
• bostrom
• cryptoorgchain
• impacthub
• microtick
• regen
• terra
• agoric
• carbon
• decentr
• injective
• nomic
• rizon
• akash
• cerberus
• desmos
• irisnet
• odin
• secretnetwork
• thorchain
• arkh
• dig
• juno
• oraichain
• sentinel
• umee
• cheqd
• emoney
• kava
• osmosis
• shentu
• vidulum
• axelar
• chihuahua
• evmos
• kichain
• panacea
• sifchain
• bandchain
• comdex
• fetchhub
• konstellation
• sommelier

Add tests for interchain security

Intermittent panic when block height is zero

WaitForBlocks

panic: height cannot be zero

goroutine 1866 [running]:
github.com/strangelove-ventures/ibctest/test.(*height).WaitForDelta(0xc00216bef8, {0x1051e6aa8, 0xc0006b0300}, 0x5)
	/Users/hh/go/pkg/mod/github.com/strangelove-ventures/[email protected]/test/wait_for_blocks.go:44 +0x160
github.com/strangelove-ventures/ibctest/test.WaitForBlocks.func1()
	/Users/hh/go/pkg/mod/github.com/strangelove-ventures/[email protected]/test/wait_for_blocks.go:24 +0x80
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/Users/hh/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:74 +0x70
created by golang.org/x/sync/errgroup.(*Group).Go
	/Users/hh/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:71 +0x110
[originating from goroutine 282]:
golang.org/x/sync/errgroup.(*Group).Go(...)
	/Users/hh/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:83 +0x110
github.com/strangelove-ventures/ibctest/test.WaitForBlocks(...)
	/Users/hh/go/pkg/mod/github.com/strangelove-ventures/[email protected]/test/wait_for_blocks.go:20 +0x108
github.com/strangelove-ventures/ibctest/chain/cosmos.(*CosmosChain).Start(...)
	/Users/hh/go/pkg/mod/github.com/strangelove-ventures/[email protected]/chain/cosmos/cosmos_chain.go:564 +0xf44
github.com/strangelove-ventures/ibctest.chainSet.Start.func1(...)
	/Users/hh/go/pkg/mod/github.com/strangelove-ventures/[email protected]/chainset.go:97 +0xc0
golang.org/x/sync/errgroup.(*Group).Go.func1(...)
	/Users/hh/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:74 +0x70
created by golang.org/x/sync/errgroup.(*Group).Go
	/Users/hh/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:71 +0x110
[originating from goroutine 8]:
golang.org/x/sync/errgroup.(*Group).Go(...)
	/Users/hh/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:83 +0x110
github.com/strangelove-ventures/ibctest.chainSet.Start(...)
	/Users/hh/go/pkg/mod/github.com/strangelove-ventures/[email protected]/chainset.go:94 +0x114
github.com/strangelove-ventures/ibctest.(*Interchain).Build(...)
	/Users/hh/go/pkg/mod/github.com/strangelove-ventures/[email protected]/interchain.go:195 +0x280
github.com/cosmos/relayer/v2/ibctest_test.TestRelayMany_WIP(...)
	/Users/hh/go/src/github.com/cosmos/relayer/ibctest/relaymany_test.go:71 +0xce0
testing.tRunner(...)
	/Users/hh/gotip/src/github.com/golang/go/src/testing/testing.go:1449 +0x188
created by testing.(*T).Run
	/Users/hh/gotip/src/github.com/golang/go/src/testing/testing.go:1493 +0x55c

Probably retry when height is 0.

Expose relayer account information

This is necessary for making assertions for new features such as ibc-go's fee middleware.

Per ibc-go team

oh one more thing, I wanted to to do some assertions on relayer accounts. I didn't see a way of getting the relayer addresses

Audit and debug blocks and transactions

Logging blocks was a first pass at being able to see the blocks and txs. However, the logs are verbose and different chain logs are mixed in together. It becomes very difficult to see the order of blocks and txs per chain.

Override entry point in `Exec` function

Currently, the entry point is not overridden when creating a container to run a job. I encountered the following error

    reporter.go:226:
        	Error Trace:	testsuite.go:191
        	            				fee_middleware_test.go:287
        	Error:      	Received unexpected error:
        	            	failed to create faucet accounts: failed to create common account with name faucet: failed to create key with name "faucet" on chain simapp-a: exit code 1:  Error: unknown command "simd" for "simd"
        	            	Run 'simd --help' for usage.

The entrypoint to this specific container is ["simd"], as a result the command the ibctest builds is appended to this entrypoint.

I believe this should be solvable by just specifying an empty array for the entrypoint.

	Config: &docker.Config{
		Entrypoint: []string{},
                 // other fields
	},

Interchaintest should have tests for timeouts on ordered channels.

Currently have tests to cover unordered with and without both height timeout and timestamp timeout.
See #5 and #12

Need to add tests for timeouts on ordered channels.
Some context here cosmos/relayer#321

More context from @jtieri:
Timeouts on ordered channels work a little different than unordered timeouts because in ordered channels packets must be processed in FIFO order so if a packet with sequence 2 times out then every packet with sequence n > 2 cannot be processed. What happens in ordered channels is if a packet times out the channel is closed
In the relayer we don't currently differentiate between the two, ordered channels require a proof of nextpacketrecv and unorderd channels require a proof of non existence to prove they have not been relayed

Interchaintest test suite should test the upgrade/update clients flow for unfreezing IBC light clients.

Testing the upgrade/update clients flow for unfreezing IBC light clients.

For example in #8 we required a hack in the relayer to update clients after the juno-1 phoenix genesis to use a pre-fork rpc to query heights before the new genesis height to successfully update the clients to a new height.

Allow the exposed docker ports to be configurable/accessible from the test host

In our tests, we would like to use the grpc endpoints exposed by the chains. They are exposed, but the current exposed ports are not configurable, I believe the are randomly being assigned.

The existing functions which extract addresses e.g. GetGRPCAddress are not resolvable on the machine running the test.

It looks like it can be configured like this. It would be nice to be able to specify ports, or to have accessor functions which return addresses accessible to the host running the test.

Alternatively, ibctest could provide a mechanism to return clients that are wired up with a given chain. e.g. chainA.GetRPCClient(), client.InitGRPC(chainA, TestRunner) or something to that effect.

Break up conformance tests

Problem

ibc-go team wants to use ibctest as part of their CICD. They primarily want to test IBC transfers. In other words, is the IBC module working? They are probably not interested in relayer tests like relayer flush and setup.

It was not obvious to them to use the conformance package which handles a lot of boilerplate for them. They were duplicating a lot of what ibctest does internally.

This, in turn, means they have to handle timing issues and races. The conformance package does that for you.

Options

• Better document conformance tests.
• Examples in ibctest.
• Rename some conformance tests for clarity?
• Break apart some conformance tests?

Create examples for making fine grained assertions

Problem

The conformance tests are not enough, albeit a great starting point.

Per cian on ibc-go team, they struggled with failing tests using the Interchain API. To be clear, the API worked well, but there's missed nuance in when to wait for blocks or other state.

I don't want to test all of these all the time, but the things I think would be useful are creating arbitrary wallets, being able to assert things are in escrow at certain times and being able to make sure that the correct wallet was refunded the correct amount etc.

I think that maybe some of these tests could belong in the conformance test suite, but I think there are quite a few fine grained assertions that we want to be able to make on the state of certain wallets/chains etc.

ibctest.User/ibctest.RelayerWallet types could maybe be combined

the difference is seems to be only in use case of the type and not the type itself.

for example, currently, we have to do things like this chainAWallet.Bech32Address(chainB.Config().Bech32Prefix) to get the bech32 address from a User type, but the RelayerWallet type offers it out of the box. just a little confusing as from the chain/SDK POV these could be the same account address types right?

Consider using ioutil.TempDir instead of t.TempDir to create home directories

t.TempDir removes the home directory upon test passing or failing. This means that if a test fails, it is difficult to exec into the containers to debug as the home dir has been removed. ioutil.TempDir leaves it to the caller to remove the directory.

I propose something along the lines of

home, err := ioutil.TempDir("", t.Name())
// handle err

defer func(){
   if !t.Failed(){
     // remove home dir
  }
}

This way we can still exec into the containers and debug upon test failure.

ibctest fails with HTTP error on Arch Linux

I am running on the latest commit, d906ad7.

After cloning, I ran make ibctest and then ./bin/ibctest. I ran the second command twice, and both times the default tests failed with an HTTP error. Here is the console log for each.

First run: run1.txt
Second run: run2.txt

I understand this project is still in active development, I'm filing this issue in hopes of learning more. Maybe this is a known ongoing bug, or something unexpected that we can hopefully fix. Thank you!

Support duplicate chains on Interchain

There are some test cases where it is helpful to run two independent but otherwise identical chains, e.g. two gaia instances. Right now, if you attempt to do this, the Interchain will reject it due to a duplicate name. Then if you allow the Interchain to accept it, the Docker setup will reference the chain.Config().Name from a few places, so the fix is more subtle than just adjusting the Interchain.

Add ability to focus on specific tests.

Proposal

This is a proposal and up for debate.

It is useful to focus on a single test, such as RelayPacket. The benefit is quicker feedback cycles vs. testing everything.

Essentially, support something akin to ibctest -test.run=RelayPacket.

Out of Scope

Focusing on a single chain pair and/or relayers is already handled by the matrix file.

Workaround

My current workaround is to download source code, comment out irrelevant tests, and run go test manually.

Feature Request: Add functionality to wait for events

Currently we have functionality to wait for a number of blocks as a mechanism to account for delay of certain operations to be seen by the test.

It would be useful to be able to listen for events and wait specific ones to be emitted.

// incentivise packet 

err := events.WaitFor(chainA, feetypes.EventTypeIncentivizedPacket, timeOut)
require.NoError(t, err)

// continue with test

Configuration should be less confusing.

Problem

I know we've sunk a lot of time into this already, but I still think configuration is confusing.

See DavidNix/ibc-go#1 as an example. I gave this PR to the ibc-go team to help them integrate ibctest into their CICD.

Namely:

cf := ibctest.NewBuiltinChainFactory(logger, []*ibctest.ChainSpec{
 		{Name: "gaia", ChainConfig: setup.NewSimappConfig("simapp-a", "chain-a", "atoma")},
 		{Name: "gaia", ChainConfig: setup.NewSimappConfig("simapp-b", "chain-b", "atomb")},
 	})

ChainSpec and ChainConfig duplicate a lot of the same fields like Name, GasAdjustment, NoHostMount, etc.

In the above they want to test a simapp. I feel there should be a more obvious way to hand over a complete ChainConfig or select a default one we provide, perhaps with optional builder methods to override.

I'm also wondering if we can use the Interchain here as well and perhaps even forgo factories entirely. (That's a large hypothesis though).

Add test to assert timestamp timeout behavior

This one is currently blocked because we skip that test for the go relayer. The relayer may not have it implemented yet.

Extend chain interface to run arbitrary commands and return an output.

Problem

Per ibc-go's team:

I was thinking maybe when iterating on tests it would be nice to have some hooks that allow for arbitrary command execution, since as we develop the features the commands might change before they're done. Ideally the interfaces get updated once the feature is released (but that's a bit of a chicken and egg scenario!)

So they're trying to test unreleased, new features. We wouldn't update the chain interface until those features are finalized. As cian mentions, those commands could change a lot before the final release.

Periodically run tests for unstable chains.

Problem

Chains beyond our control (like Penumbra) tend to block PRs when the tests fail. In turn, this causes toil because the failure is not related to the PR changes.

But we still want automated tests for these chains.

PRs, typically, should have stable tests in order to protect developer speed.

Options

• Run periodically (once per day)?
• (Fancier) Only run unstable chain tests on certain file changes?
• How to be alerted if upstream fails?

Add tests for wasm

example contract for sending cw20 across IBC https://github.com/CosmWasm/cw-plus/tree/main/contracts/cw20-ics20
example reflect test https://github.com/notional-labs/wasmd/blob/bounty/x/wasm/ibc_reflect_test.go

Add tests for interchain accounts

Related: #14 #225

Hermes support

Great initiative with the work here, love it!

I just wanted to quickly drop a couple of resources here detailing our current process of automatically creating docker images with every release of Hermes. Hoping these hints could help understanding how to support Hermes in the testing framework.

• Docker tags uploaded here: https://hub.docker.com/r/informalsystems/hermes/tags
• Images are based on this docker file: https://github.com/informalsystems/ibc-rs/blob/master/ci/hermes.Dockerfile

Enable a RestoreKey function for the chains

Since the relayer address has been exposed, I ran into a new issue where I needed to run an sdk command and sign a transaction using the relayer address. The relayer address is not in the keychain (as the relayer address/mnemonic is generated before the chain nodes exist)

I propose a function that looks something like this

func (c *CosmosChain) RecoverKey(ctx context.Context, name, mnemonic string) error {
	tn := c.ChainNodes[0]

	cmd := []string{
		"bash",
		"-c",
		fmt.Sprintf(`echo "%s" | %s keys add %s --recover --keyring-backend %s --home %s`, mnemonic, c.Config().Bin, name, keyring.BackendTest, tn.NodeHome()),
	}

	exitCode, stdout, stderr, err := tn.NodeJob(ctx, cmd)
	if err != nil {
		return dockerutil.HandleNodeJobError(exitCode, stdout, stderr, err)
	}

	return nil
}

I don't know if there is a more elegant way of piping values using NodeJob or maybe there is a command that doesn't require it.

Feature Request: Support broadcasting messages as test users from the test host

From the tests, I would like to be able to broadcast arbitrary messages as users created in the tests themselves.

In issue #172 I found myself needing the private key of the users in order to sign transactions, however since then I found out that it should be possible to broadcast messages using this function.

If we use this function, I would no longer need private keys exposed (although I suspect it would still be useful in certain situations)

When attempting to utilize this function, I ran into a few issues.

• BroadCastTx signs the transactions by reading from the specified keyring, which means the test host needs access to valid keyring for this to work.
• A tx.Factory needs to be configured based on the chains.
• A client.Context needs to be configured based on user/chain/test environment.

Remove dependency on strangelove docker repo

There is currently a dependency on strangeloves docker repo, here:

Repository: fmt.Sprintf("ghcr.io/strangelove-ventures/heighliner/%s", name),

It would be great if a full docker image path (rather than image:version) is passed in, it doesn't get prefixed.

Can't run tests when ibc-go verison is different from ibctest's version

I ran into the following issue

In the ibc-go repo, we have upgraded the module i.e.

module github.com/cosmos/ibc-go/v4

In the go.mod file. In the ibctest repo, we have

github.com/cosmos/ibc-go/v3 v3.0.0

I'm attempting to write a test that imports a type from ibc-go/v4, and I run into the following error

panic: error with code 2 is already registered: "invalid identifier"

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/types/errors.Register({0x1010d4d20, 0x4}, 0x2, {0x1010e6d4b, 0x12})
	/Users/chatton/go/pkg/mod/github.com/cosmos/[email protected]/types/errors/errors.go:163 +0x268
github.com/cosmos/ibc-go/v4/modules/core/24-host.init()
	/Users/chatton/checkouts/ibc-go/modules/core/24-host/errors.go:12 +0x3c
FAIL	github.com/cosmos/ibc-go/v4/e2e	0.367s
FAIL

This ended up being due to the fact that there is an init function that is running twice (one in v3 and one in v4) and the second one fails.

I upgraded a local copy of ibctest to v4.0.0-rc0 and everything worked as expected.

I don't know what the best solution to this problem is. I'm guessing that ibctest will not ever need to explicitly test an older version of ibc-go.

In order for us to run tests right now, we need to update ibctest's version of ibc-go to v4.

Rename repo from ibc-test-framework to ibctest

If ever exposed as a Go package, ibc-test-framework is quite long and I suspect most package consumers will end up aliasing the package name.

Allow the ability to provide your own `builtinChainConfigs`

Currently there is a hard coded map of built in chain configs

var builtinChainConfigs = map[string]ibc.ChainConfig{
	"gaia":     cosmos.NewCosmosHeighlinerChainConfig("gaia", "gaiad", "cosmos", "uatom", "0.01uatom", 1.3, "504h", false),
	"osmosis":  cosmos.NewCosmosHeighlinerChainConfig("osmosis", "osmosisd", "osmo", "uosmo", "0.0uosmo", 1.3, "336h", false),
	"juno":     cosmos.NewCosmosHeighlinerChainConfig("juno", "junod", "juno", "ujuno", "0.0025ujuno", 1.3, "672h", false),
	"agoric":   cosmos.NewCosmosHeighlinerChainConfig("agoric", "agd", "agoric", "urun", "0.01urun", 1.3, "672h", true),
	"icad":     cosmos.NewCosmosHeighlinerChainConfig("icad", "icad", "cosmos", "photon", "0.00photon", 1.2, "504h", false),
	"penumbra": penumbra.NewPenumbraChainConfig(),
}

We are able to use NewBuiltinChainFactory which makes creating a ChainFactory very easy, but it would be nice to be able to specify our own defaults, something like

func NewChainFactory(log *zap.Logger, specs []*ChainSpec, builtIns map[string]ibc.ChainConfig) *BuiltinChainFactory

to provide the easy of use and also our set of configs that are specific to our repo and our registries, and maybe don't belong in this particular builtins map.

testing webhook

Add Composable chain to test framework

The composable chain is implementing IBC and we need to implement a spin up for it. The following dockerfile contains how they spin their chain up. We can also ask questions in their telegram about how to break this out into separate pieces.

https://github.com/ComposableFi/composable/blob/main/docker/composable-sandbox.dockerfile

Intermittent test failure for locked sqlite database

=== CONT  TestMessagesView/initiate_transfer
[1253](https://github.com/strangelove-ventures/ibctest/runs/6835042621?check_suite_focus=true#step:5:1254)
    messages_view_test.go:265: 
[1254](https://github.com/strangelove-ventures/ibctest/runs/6835042621?check_suite_focus=true#step:5:1255)
        	Error Trace:	messages_view_test.go:265
[1255](https://github.com/strangelove-ventures/ibctest/runs/6835042621?check_suite_focus=true#step:5:1256)
        	Error:      	Received unexpected error:
[1256](https://github.com/strangelove-ventures/ibctest/runs/6835042621?check_suite_focus=true#step:5:1257)
        	            	database is locked (5) (SQLITE_BUSY)
[1257](https://github.com/strangelove-ventures/ibctest/runs/6835042621?check_suite_focus=true#step:5:1258)
        	Test:       	TestMessagesView/initiate_transfer

This is odd because the database is :memory:.

Large test matrix should be runnable on a schedule.

Related to #27, we should run ibctest against that large matrix on a schedule. We don't want the large matrix to gate ibctest CI, because some tests will likely fail on a large enough set of chains.

This might run on GitHub Actions or it might run somewhere else. We should also consider where this will archive the test report output.

Use docker volumes for chain containers

Following up on #195, use Docker volumes -- not host mounts -- for chain containers. There is a chance that this will address the macOS Docker engine hang, but even if it doesn't, this will allow using ibctest against a remote Docker engine.

Add penumbra chain

https://github.com/strangelove-ventures/heighliner/pkgs/container/heighliner%2Fpenumbra
https://github.com/penumbra-zone/penumbra
https://guide.penumbra.zone/main/pd.html

Add tests for multiple channels on a single connection

Height timeout test is flaky

=== CONT  TestConformance/chain_pairs/[email protected][email protected]/[email protected]/height_timeout
    reporter.go:226:
            Error Trace:    test.go:487
                                        test.go:301
            Error:          Not equal:
                            expected: 10000000000
                            actual  : 9999000000

Personally, I've seen this ocassionally locally but not on CI yet.

Add tests for submit misbehavior

https://github.com/cosmos/ibc/blob/master/spec/client/ics-007-tendermint-client/README.md#misbehaviour

Test that the light clients are frozen when relayers submit misbehavior, for a situation such as if two headers could be considered valid for the same height

Add test(s) to assert acknowledgements are relayed

Expose user private keys

Feedback from ibc-go team:

I'm trying to add a test which creates an signs a transaction using a grpc client. In order to do this I need the private key of a user. Can you think of a relatively straight forward way of doing this?

I see that the generated private keys go into some db somewhere? But maybe we could use a preset fixture of a mnemonic/public/private key for certain tests that need access to this. What do you think?

Create channel with specific version

From cian, ibc-go team:

a channel needs to be created with a specific version for it to be enabled, I don't know is that configurable with ibctest? This command is using hermes but I think it should probably work in a similar way with go relayer

hermes -c ./network/hermes/config.toml create channel test-1 connection-0 \
--port-a transfer \
--port-b transfer \
-v "{\"fee_version\":\"ics29-1\",\"app_version\":\"ics20-1\"}"

in our tests it would be nice to test behaviour when one side does not have it enabled etc.
looking at the code, I'm guessing this would be done through CreateChannelOptions.Version

---

Note: This may have relayer dependencies.

Execute arbitrary commands for relayer.

Similar to #158 but for the relayer instead of the chain.

Chain state should be cached on startup (opt-in)

Problem

The test setup can take a long time (> 60s) to create clients, connections, channels, faucet accounts, test accounts, etc.

Proposal

Allow a developer (or perhaps an end user) to optionally specify a starting state that already has the above mentioned transactions. That way, we can skip the setup and quickly get into more interesting aspects of IBC such as transfering funds from one chain to another.

This option could be part of the matrix file or command line flags.

Goals

• MVP supports cosmos chains only.

Add a seed node test