progval / irctest

Semi-automated conformance checking for IRC implementations (RFC1459/RFC2812, the "modern spec" by ircdocs, and IRCv3)

irctest

This project aims at testing interoperability of software using the IRC protocol, by running them against common test suites.

It is also used while editing the "Modern" specification to check behavior of a large selection of servers at once.

The big picture

This project contains:

• IRC protocol test cases, primarily checking conformance to the "Modern" specification and IRCv3 extensions, but also RFC 1459 and RFC 2812. Most of them are for servers but also some for clients. Only the client-server protocol is tested; server-server protocols are out of scope.
• Small wrappers around existing software to run tests on them. So far this is restricted to headless software (servers, service packages, and clients bots).

Wrappers run software in temporary directories, so running irctest should have no side effect.

Test results for the latest version of each supported software, and respective logs, are published daily.

Prerequisites

Install irctest and dependencies:

sudo apt install faketime  # Optional, but greatly speeds up irctest/server_tests/list.py
cd ~
git clone https://github.com/progval/irctest.git
cd irctest
pip3 install --user -r requirements.txt

Add ~/.local/bin/ (and/or ~/go/bin/ for Ergo) to your PATH if it is not.

export PATH=$HOME/.local/bin/:$HOME/go/bin/:$PATH

Using pytest

irctest is invoked using the pytest test runner / CLI.

You can usually invoke it with python3 -m pytest command; which can often be called by the pytest or pytest-3 commands (if not, alias them if you are planning to use them often).

After installing pytest-xdist, you can also pass pytest the -n 10 option to run 10 tests in parallel.

The rest of this README assumes pytest works.

Test selection

A major feature of pytest that irctest heavily relies on is test selection. Using the -k option, you can select and deselect tests based on their names For example, you can run LUSERS-related tests with -k lusers.

Using the -m option, you can select and deselect and them based on their markers (listed in pytest.ini). For example, you can run only tests based on RFC1459 with -m rfc1459.

By default, all tests run; even niche ones. So you probably always want to use these options: -m 'not Ergo and not deprecated and not strict. This excludes:

• Ergo-specific tests (included as Ergo uses irctest as its official integration test suite)
• tests for deprecated specifications, such as the IRCv3 METADATA specification
• tests that check for a strict interpretation of a specification, when the specification is ambiguous.

Running tests

This list is non-exhaustive, see workflows.yml for software not listed here. If software you want to test is not listed their either, please open an issue or pull request to add support for it.

Servers

Ergo:

cd /tmp/
git clone https://github.com/ergochat/ergo.git
cd ergo/
make install
cd ~/irctest
pytest --controller irctest.controllers.ergo -k 'not deprecated'

Solanum:

cd /tmp/
git clone https://github.com/solanum-ircd/solanum.git
cd solanum
./autogen.sh
./configure --prefix=$HOME/.local/
make -j 4
make install
pytest --controller irctest.controllers.solanum -k 'not Ergo and not deprecated and not strict'

InspIRCd:

cd /tmp/
git clone https://github.com/inspircd/inspircd.git
cd inspircd

# Optional, makes tests run considerably faster. Pick one depending on the InspIRCd version:
# on Insp3 <= 3.16.0 and Insp4 <= 4.0.0a21:
patch src/inspircd.cpp < ~/irctest/patches/inspircd_mainloop.patch
# on Insp3 >= 3.17.0 and Insp4 >= 4.0.0a22:
export CXXFLAGS=-DINSPIRCD_UNLIMITED_MAINLOOP

./configure --prefix=$HOME/.local/ --development
make -j 4
make install
cd ~/irctest
pytest --controller irctest.controllers.inspircd -k 'not Ergo and not deprecated and not strict'

UnrealIRCd:

cd /tmp/
git clone https://github.com/unrealircd/unrealircd.git
cd unrealircd
./Config  # This will ask a few questions, answer them.
make -j 4
make install
cd ~/irctest
pytest --controller irctest.controllers.unreal -k 'not Ergo and not deprecated and not strict'

Servers with services

Besides Ergo (that has built-in services) and Sable (that ships its own services), most server controllers can optionally run service packages.

Atheme:

You can install it with

sudo apt install atheme-services

and add this to the pytest call:

--services-controller irctest.controllers.atheme_services

Anope:

Build with:

cd /tmp/
git clone https://github.com/anope/anope.git
cd anope
./Config  # This will ask a few questions, answer them.
make -C build -j 4
make -C build install

and add this to the pytest call:

--services-controller irctest.controllers.anope_services

Clients

Limnoria:

pip3 install --user limnoria pyxmpp2-scram
cd ~/irctest
pytest --controller  irctest.controllers.limnoria

Sopel:

pip3 install --user sopel
mkdir ~/.sopel/
cd ~/irctest
pytest --controller irctest.controllers.sopel

What irctest is not

A formal proof that a given software follows any of the IRC specification, or anything near that.

At best, irctest can help you find issues in your software, but it may still have false positives (because it does not implement itself a full-featured client/server, so it supports only “usual” behavior). Bug reports for false positives are welcome.

irctest's Issues

chathistory batch parameter is not validated

I accidentally switched the batch type (chathistory) and batch parameter (canonical name of the message target) in Ergo. This is the fix for reference: ergochat/ergo#2070

This is a MUST in the spec, so we should have a test for it:

If the batch capability was negotiated, the server MUST reply to a successful CHATHISTORY command using a batch. For subcommands that return message history (i.e. all subcommands other than TARGETS), the batch MUST have type chathistory and take a single additional parameter, the canonical name of the target being queried.

Server controllers: allow defining user accounts

This is needed to test SASL with servers.

irctest for draft/multiline's max-lines and max-bytes limits

Factorize client and server tests

It would be nice to have a way to write tests that work for both peers.
The problem is tests would have to be both loose (what they accept) and exact (what they send) for each event.

Make it possible to run only a part of the tests.

test handling the forbidden octets

Test handling of \x00 and \r in messages (I don't think we test this currently?)

Not supporting ECDSA is seen as a failure in all cases

Since IRC ECDSA SASL isn't a standard, I don't think it should be seen as a failure for clients to not support it in all cases.

I've got a (dodgy) attempt at a fix here, but maybe it would be nice to allow controllers to set a variable like:

self.supported_sasl_mechs = ['PLAIN']

And if that list exists and the given mechanism isn't in that list, just skip the testing of it?

irctest dashboard

[Just dumping some thoughts here]

In the last week I added CI to the repo + made it so GH Actions exports the pytest results as junit XML.

So far this isn't very useful, because the only UI we have is the shitty bot comments on PRs, and a list of successful/failed builds, without per-test details.

So I'd like to make a dashboard (probably just a cron generating XHTML files) that reads these junit XML files, and produces large support tables, highlighting what tests pass for each ircd, and what tests are automatically skipped (with the reason) based on these XML files, and what tests are manually excluded from the collection (also with the reason) somehow (parsing comments in the makefile?)

Ideally this should be readable to people unfamiliar with irctest (eg. by including each test's docstring) so they can use it to see exactly what is implemented by each IRCd.

decrease minimum Python version requirement

https://github.com/ProgVal/irctest/blob/4ded96fbbacc9b9085a8dbe58e6469dacbf49cbd/irctest/irc_utils/junkdrawer.py#L11-L15

This does not work with Python 3.6, and does work with 3.8 (I'm not sure about 3.7).

There may be other, similar issues.

default assertion failure messages have become unhelpful

I broke one of my ZNC tests in development and:

        self.sendLine("viewer", "PRIVMSG *playback :play * %d" % (early_time,))
        messages = extract_playback_privmsgs(self.getMessages("viewer"))
>       self.assertEqual(set(messages), set([dm] + echo_messages))
E       AssertionError: None

irctest/server_tests/znc_playback.py:95: AssertionError

It looks like there's no sane default for msg here?

irctest/irctest/cases.py

Line 226 in 5e4ae7c

msg: Any = None,

tests for the bot mode spec

ircv3/ircv3-specifications#439 specifies the bot mode; Oragono's master (0db7f886374ce616 and later) implements the spec as written, including the new draft/bot tag. It would be neat to get some tests for this (they can be fully ircd-agnostic, since the value of the bot mode is published in 005).

Add tests for draft/metadata-2

Spec: ircv3/ircv3-specifications#501

WIP tests in this branch: https://github.com/progval/irctest/compare/metadata2?expand=1 (no PR because it would be noisy)

• SUB before connect
• UNSUB
• SYNC
• rate-limiting
• quote a spec paragraph in every docstring
• test Unreal module on Github CI (pull https://github.com/progval/unrealircd-contrib/blob/metadata2/files/metadata2.c )