matrix-org / patience

Full stack integration testing for Matrix clients and servers

Patience

Full stack integration testing for Matrix clients and servers

Features

• Any permutation of supported clients can be tested together
• Client specifics are hidden by default (but still accessible) so tests can target many clients without modification
• Tests written in TypeScript
• Report mode gives a quick summary of results in your terminal
• Interactive mode allows you to try out and debug the clients in your browser
• Builds on top of Complement's homerunner API
• Test harness displays all clients together on one page

Supported environments

Patience aims to support testing different combinations of Matrix clients and servers in a unified environment.

Element Web and Hydrogen are the currently supported clients. Anything that fits in an iframe should be easy to add. Tools such as Appetize could be used to add mobile clients.

Synapse and Dendrite are the currently supported homeservers.

Usage

Setup is currently a bit manual, as this project is just getting started. If you have suggestions on how to improve setup, please file an issue.

To get started with Patience in your project, first collect the following bits and bobs:

• Docker
• Go
• Node.js
• homerunner from Complement
  • go install github.com/matrix-org/complement/cmd/homerunner@latest
• One or more Complement-ready homeserver images
• Chrome
  • We plan to switch to Playwright in the future to support additional browsers

Create a directory to hold your tests and add Patience:

npm add @matrix-org/patience --save-dev

Add a test, perhaps by copying hello.ts. At a minimum, you should call orchestrate to set up servers, clients, and rooms for your test. Most likely you'll want to actually test something too.

To run your tests in reporting mode:

npx patience '*.ts'

You should see:

Finished running tests, all tests passed! 🎉

To run your tests in interactive mode:

npx patience '*.ts' -- --manual

This will start a server at localhost:8000 which you can navigate to in your browser. Click on one of listed test files to watch the test run. You can interact with the clients, timeline, etc. The clients remain after for exploration until you stop the server.

patience's Issues

Add Cinny as a testable client

As it says in the title, adding support for Cinny would be interesting to see.

Add client configuration

At the moment, choosing the clients to use for each test is very manual. We should instead add some kind of config format (perhaps using https://github.com/davidtheclark/cosmiconfig) so that you can specify a list of clients to pair up as desired.

Patience would then run all tests for all pairs of clients listed.

Specific tests may override this configuration as needed, in case e.g. only certain clients will work etc.

Add room DAG visualizations

During monthly wrap-up demos today, I heard that @jryans had plans to add the room DAG visualization to the patience test harness interface 😍 This is exactly what I have been aspiring to add as well!

Would be cool to make a generalized solution to make a TARDIS visualization for any test run in Complement (after hook to save a file which we can view in TARDIS)

-- matrix-org/complement#68

Patience already automates hooking up Element to the homeserver which I was doing manually and added docs for, https://github.com/matrix-org/complement/blob/master/ONBOARDING.md#how-do-i-hook-up-a-matrix-client-like-element-to-the-homeservers-spun-up-by-complement-after-a-test-runs

So it sounds like a perfect opportunity to combine some efforts 🤠. My use case is a little different as I am after testing the homeserver vs testing a client but there is so much overlap here.

For reference on the DAG visualizations I have been generating so far, I adapted TARDIS to work with Synapse, matrix-org/tardis#1

Element Web storage confusion when running multiple tests

For some reason, when running multiple tests, Element Web's account storage does not seem to get cleared properly between test runs, so tests after the first one get confused and fail.

It's a bit odd, since manual runs where you click each test file one by one do seem to work.

Add test writer documentation

Once the APIs and interfaces have firmed up a bit more, we should more docs on how to write tests.

Using develop.element.io for Element Web causes frames to nest the harness

While polishing up Patience yesterday, I encountered one of the stranger web behaviours I've seen in a while...

The clients are loaded into iframes inside the Patience test harness page. When Element Web tries to adjust the URL for going to a room (location.assign("#/room/!1234:matrix.org")), somehow the browser decides it should use the parent document (the test harness page) as the base URL, and nests a second copy of the test harness inside itself... 🤯

This was not happening a few months ago, so... that's going to be a strange one to sort out. 😵‍💫

It seems to reproduce consistently in Chrome, but only sometimes in Firefox.

It only seems to happen when using https://develop.element.io as the Element Web deployment. If I instead run Element Web locally, it works fine. So far, I've tried comparing HTTP headers and Element Web configs, but nothing seems to cause my local copy to behave like develop in this way.

Hydrogen service worker confuses multiple client instances

The Hydrogen service worker causes general confusion when running multiple instances at the same time like Patience does, since all instances hear what the others are doing.

We'll need to do at least one of:

• Isolate each client on its own domain (blocks direct window access)
• Run a modified Hydrogen instance with service workers disabled
• Craft some hack to make it seem like the browser doesn't support service workers

We are already attempting the hack approach:

patience/core/api/adapters/hydrogen.ts

Lines 155 to 170 in a255dce

	// Inject a helper script to disable service workers, as the multiple
	// frames on same domain otherwise affect each other via the service
	// worker.
	const helperScript = frameDoc.createElement("script");
	helperScript.textContent = "(" + function() {
	// We can't delete navigator.serviceWorker, so instead we hide it
	// with a proxy.
	const navigatorProxy = new Proxy({}, {
	has: (target, key) => key !== "serviceWorker" && key in target,
	});
	Object.defineProperty(window, "navigator", {
	value: navigatorProxy,
	});
	console.log("Service worker support disabled");
	} + ")()";
	frameDoc.head.prepend(helperScript);

...but it seems to only work in Firefox. Recent Chrome appears to run the frame's document before the hack has a chance to mess with things.

Mobile device integration testing

patience looks exactly like an idea I wrote down a while back, I think it'll work well if support for integration testing of mobile-based matrix apps is also added.

I see Appetize mentioned in the readme, however, I think a locally-hostable alternative should also be considered.