aaronabramov / k9

Rust testing library

K9 - Rust Testing Library

Snapshot testing + better assertions

Available test macros

• snapshot
• assert_equal
• assert_greater_than
• assert_greater_than_or_equal
• assert_lesser_than
• assert_lesser_than_or_equal
• assert_matches_regex
• assert_err_matches_regex
• assert_matches_snapshot
• assert_matches_inline_snapshot
• assert_ok
• assert_err

See https://docs.rs/k9 for API documentation

snapshot!() macro

Snapshot macro provides the functionality to capture the Debug representation of any value and make sure it does not change over time.

If it does change, the test will fail and print the difference between "old" and "new" values.

If the change is expected and valid, running cargo test with K9_UPDATE_SNAPSHOTS=1 env variable set will automatically take the new value and insert it into the test source code file as a second argument, after which all subsequent test runs should start passing again.

assert_equal!() macro

Rust already provides a good built-in test runner and a set of assertion macros like assert! and assert_eq!. They work great for for quick unit tests, but once the codebase and test suites grows to a certain point it gets harder and harder to test things and keep tests readable.

For example, when testing that two structs are equal using assert_eq! macro the output does not provide a lot of help in understanding why exactly this test failed.

#[derive(PartialEq, Debug)]
struct Person {
    name: &'static str,
    age: usize,
}

#[test]
fn test_eq() {
    let person1 = Person {name: "Bob", age: 12 };
    let person2 = Person {name: "Alice", age: 20 };
    assert_eq!(person1, person2, "These two must be the same person!");
}

All we get is usually a wall of text collapsed into a single line and you have to find the difference between two structs yourself. Which becomes very time consuming when structs are 10+ fields.

---- eq::test_eq stdout ----
thread 'eq::test_eq' panicked at 'assertion failed: `(left == right)`
  left: `Person { name: "Bob", age: 12 }`,
 right: `Person { name: "Alice", age: 20 }`: These two must be the same person!', src/eq.rs:13:5

using k9::assert_equal macro improves this output and prints the difference between two structs:

use k9::assert_equal;
assert_equal!(person1, person2, "These two must be the same person!");

Non-equality based assertions

Testing equality is very simple and can definitely work for most of the cases, but one of the disadvantages of only using assert! and assert_eq! is the error messages when something fails. For example, if you're testing that your code produces valid URL

let url = generate_some_url();
assert_eq!(URL_REGEX.is_match(url), true);

What you get is

thread 'eq::test_eq3' panicked at 'assertion failed: `(left == right)`
  left: `false`,
 right: `true`', src/eq.rs:19:5

Which doesn't help much. Especially, if you're new to the code base, seeing things like expected 'true' but got 'false' will make you go and look at the code before you even know what the problem can be, which can be very time consuming.

What we probably want to see is:

Which gives us enough context on what the problem is and how to fix it without for us having to go and run/debug the test first.

k9's Issues

`assert_eq!(left, right, format, args...)` parity

The standard assertion macros generally pass the error message and arguments to the panic! macro.
k9 just seems to allow for a lone string without formatting, which means that migrating this example from the bottom of the docs:

https://doc.rust-lang.org/std/macro.assert_eq.html

assert_eq!(a, b, "we are testing addition with {} and {}", a, b);

to k9 needs to done like this:

assert_equal!(a, b, format!("we are testing addition with {} and {}", a, b));

https://docs.rs/k9/latest/k9/ has version 0.11.6 rather than 0.12

Is there a doc build issue, or is something else wonky?

I tend to use docs.rs as a quick way to sanity check the current version so I was surprised to find that there's a new k9 available!

assert_ok! and assert_err!

assert_equal does not handle comparing vectors to arrays

This is a parity issue with assert_eq!

// This works:
assert_eq!(vec![1, 2, 3], [1, 2, 3]);

// This does not:
assert_equal!(vec![1, 2, 3], [1, 2, 3]);

error[E0308]: mismatched types
   --> src/sampler.rs:429:38
    |
429 |         assert_equal!(vec![1, 2, 3], [1, 2, 3]);
    |                                      ^^^^^^^^^
    |                                      |
    |                                      expected struct `std::vec::Vec`, found array `[{integer}; 3]`
    |                                      help: try using a conversion method: `[1, 2, 3].to_vec()`
    |
    = note: expected struct `std::vec::Vec<{integer}>`
                found array `[{integer}; 3]`

Asserting more than one snapshot per test

right now we're storing the snapshot in the file with the name of the test.

e.g.

// my_test.rs
#[test]
fn some_test() {
  assert_matches_snapshot!("111");
}

will save its snapshot under
__k9_snapshots__/my_test/some_test.snap

If we use the macro more than once, we'll technically be overwriting the same file

// my_test.rs
#[test]
fn some_test() {
  assert_matches_snapshot!("111");
  assert_matches_snapshot!("111");
}

We'll need to come up with some serialization format for snapshots that relies on

1. order that snapshots were asserted in
2. name of the snapshot (if provided, doesn't exist yet)

Inline snapshots TODO

• Transaction support for file writes (concurrently updated source files, updating multiple snapshots in the same file while its source is changisg)
• properly escape all characters that can mess up string literals ("#)
• Remove all .unwrap() and .expect() and replace with Result with nice error messages and error chains
• Write a good test suite that create a temporary projects and runs/updates snapshots in it
• reformat source file (rustfmt) after updating an inline snapshot in it
• adjust inline snapshot failure message to not include inline snapshot literal twice
• k9::assert_matches_inline_snapshot fails to locate macro location

better error message for snapshot failure (when no snapshot present)

assert_panics!

// success if the closure inside panics, panics if the closure doesn't panic
assert_panics!(|| {
  panic!("sup y'all");
});

see https://doc.rust-lang.org/std/panic/fn.catch_unwind.html

Make the list of assertions in README.md links to rust docs

Also need to polish all API docs and add a lot of good examples

Snapshots won't work with cargo workspaces

assert_err_matches_inline_snapshot!()

also need to implement some kind of regex based set of sanitizers, to parse out dynamic things like timestamps, iso8601, ipv4, ipv6, etc.

Add docs for crate features

Regex feature is undocumented currently

k9/Cargo.toml

Lines 12 to 13 in f979a89

	[features]
	default = ["regex"]

doc assets

assert_contains!

basic prototype of the fn

https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=a9a307bab62baa5bc6e6823ebc60b15c

also makes me thing that we'll need assert_contains_value! and assert_contains_key! macros for maps!

@TheWebDevel

Cant run program

running a cargo run
returns : error: a bin target must be available for cargo run

assertion panic backtrace should point to the test source's as the first line

display only relevant parts in diff string

when comparing large serialized strings (assert_equal or snapshots)
the diffs can get pretty large, with only 2-3 relevant lines.
we need to find a way to collapse the irrelevant parts, e.g.

Expected AAA to match snapshot:


... 100 lines

   value: 1
+ value: None
- value: Some(2),
  other_value: 2,

... 100 lines

not sure what the best way of collapsing, probably worth looking into other libraries that do that, like git output?

assert_matches_snapshot needs Display impl

It seems that assert_matches_snapshot!(T) needs T to implement Display. This is a pretty restrictive requirement IMO, and Debug should suffice.

The current workaround id to format! T :assert_matches_snapshot!(format!("{:?}")), but this should not be necessary.

What do you think?

k9::assert_matches_inline_snapshot!("blah") fails to find AST node

assert_matches_inline_snapshot!()

Trailing space after a backspace in a string will lead to a wrong snapshot

Hey,
I found a small bug. When the input for the snapshot macro has a \ followed by a space, that backspace will be duplicated in the generated snapshot. That bug is not a problem, because it happens everytime. But the snapshot looks wrong uppon inspection. Here is an example:

#[cfg(test)]
mod tests {
    #[test]
    fn test_foo() {
        k9::snapshot!(r#"\ "#, r#"\\ "#);
    }
}

☂️ DX Discussion

These are my notes and thoughts after trying out k9.

• wish it showed PASS like Jest
• show docs on hover in IDE (like other crates)
• better error message for snapshot failure (when no snapshot present)
• couldn't get snapshot assertion to work with example from docs (i blame myself)
• also, could explain the _r assertions better and when you might want to use one

README

• could be nice to have a quick walkthrough like jest
• add installation instructions

adjust inline snapshot failure message to not include inline snapshot literal twice

only use `r##" "##` escaping when needed

in inline snapshots literals

codeframe in assertion failures

we have line! and column! macros. we can technically take out a pice of code with some context around it where assertion failed and add it to the error message
e.g.

Assertion Failure!

43 | fn do something() {
44 |   let stuff = 1
45 |   assert_equal!(stuff, 2);
46 |  }

Trailing comma in multiline macros

    assert_err_matches_regex!(
        conn.query_raw("SELECT 1").await,
        "time out"
    );

works, but

    assert_err_matches_regex!(
        conn.query_raw("SELECT 1").await,
        "time out",
    );

doesn't

replace all .unwrap() and .expect() with `?`

add

#![warn(clippy::unwrap_used)]
#![warn(clippy::expect_used)]

to lib.rs to enable linter that will show all usages of unprawe() and expect()

Warnings in Rust 1.51

warning: panic message is not a string literal
   --> term/src/test/mod.rs:366:5
    |
366 | /     k9::snapshot!(
367 | |         term.get_semantic_zones().unwrap(),
368 | |         "
369 | | [
...   |
378 | | "
379 | |     );
    | |______^
    |
    = note: `#[warn(non_fmt_panic)]` on by default
    = note: this is no longer accepted in Rust 2021
    = note: this warning originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

and similarly for the assert_equals macro, which I've aliased here:

warning: panic message is not a string literal
   --> wezterm-font/src/shaper/harfbuzz.rs:512:13
    |
512 | /             assert_eq!(
513 | |                 info,
514 | |                 vec![GlyphInfo {
515 | |                     cluster: 0,
...   |
526 | |                 },]
527 | |             );
    | |______________^
    |
    = note: this is no longer accepted in Rust 2021
    = note: this warning originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

[WIP] Preliminary prototype for collapsed output

Usecase: Difference between two structs.
Assertion: assert_equal

Current Output:

Collapsed Output:

Method: String Iteration

Implementation:

1. Computed the index of each occurrence with reference to (- or +). // [4, 5, 16, 17, 20, 21]
2. I have a cur_index and a peek_index.
3. If the difference between cur_index and peek_index is <= 5, I'll have those index and the intermediate values in the final collapsed output else I have a .... // [4, 5, 16, 17, 18, 19, 20, 21]