verifytests / verify.quibble

Extends Verify to allow comparison of text via Quibble.

Verify.Quibble

Extends Verify to allow comparison of text via Quibble.

See Milestones for release notes.

NuGet package

https://nuget.org/packages/Verify.Quibble/

Usage

Initialize

[ModuleInitializer]
public static void Init()
{
    VerifierSettings.UseStrictJson();
    VerifyQuibble.Initialize();
}

^{snippet source | anchor}

UseStrictJson is required since Verify by default uses a variant of json which Quibble cannot parse.

Verify

Given an existing verified file:

{
  "Property1": "ValueA",
  "Property2": "ValueB"
}

And a test:

[Test]
public async Task Sample()
{
    var target = new Target(
        Property1: "ValueC",
        Property2: "ValueD");
    await Verifier.Verify(target);
}

Diff results

When the comparison fails, the resulting differences will be included in the test result displayed to the user.

Results do not match.
Use DiffEngineTray to verify files.
Differences:
Received: Tests.Sample.received.json
Verified: Tests.Sample.verified.json
Compare Result:
String value difference at $.Property1: ValueC vs ValueA.
String value difference at $.Property2: ValueD vs ValueB.

Icon

Argument designed by Vinence Studio from The Noun Project.

verify.quibble's Issues

System.Text.Json.JsonReaderException when model changes and contains an empty string

Description

Running a test with Verify.Quibble throws a System.Text.Json.JsonReaderException when the JSON model the contains an empty string and received is different from verified.

How to reproduce

1. Create a new xUnit test project.
2. Create a test class equal to Verify.Quibble/src/Tests/Tests
3. Run the test the first time to get the verified file created
4. Run it again and the test passes
5. On the Sample test, change Property1 on the Target model from ValueC to an empty string
6. Run the test again.

I would expect that in the test result Verify now detects a change between the received and verified since we changed the ValueC to empty string for the Property1.

What happens is that we get the following exception:

Message: 
System.ArgumentException : Failed to parse the value passed for leftJsonString as a JSON value. See inner exception for details. (Parameter 'leftJsonString')
---- System.Text.Json.JsonReaderException : ',' is an invalid start of a value. LineNumber: 1 | BytePositionInLine: 15.

Stack Trace: 
JsonStrings.parse(String s, String paramName)
JsonStrings.diff(String leftJsonString, String rightJsonString)
JsonStrings.TextDiff(String leftJsonString, String rightJsonString)
<>c.<Initialize>b__0_0(String received, String verified, IReadOnlyDictionary`2 _) line 22
Comparer.CompareStrings(String extension, String received, String verified, VerifySettings settings) line 38
Comparer.Text(FilePair filePair, String receivedText, VerifySettings settings) line 13
VerifyEngine.GetResult(VerifySettings settings, FilePair file, Target target, Boolean previousTextFailed) line 39
VerifyEngine.HandleResults(List`1 targetList) line 60
InnerVerifier.VerifyInner(Object target, Func`1 cleanup, IEnumerable`1 targets) line 25
InnerVerifier.Verify[T](T target) line 80
<<Verify>b__0>d.MoveNext() line 31
--- End of stack trace from previous location ---
Tests.Sample() line 20
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
Utf8JsonReader.ConsumeValue(Byte marker)
Utf8JsonReader.ReadSingleSegment()
Utf8JsonReader.Read()
JsonDocument.Parse(ReadOnlySpan`1 utf8JsonSpan, JsonReaderOptions readerOptions, MetadataDb& database, StackRowStack& stack)
JsonDocument.Parse(ReadOnlyMemory`1 utf8Json, JsonReaderOptions readerOptions, Byte[] extraRentedArrayPoolBytes, PooledByteBufferWriter extraPooledByteBufferWriter)
JsonDocument.Parse(ReadOnlyMemory`1 json, JsonDocumentOptions options)
JsonDocument.Parse(String json, JsonDocumentOptions options)
JsonStrings.parse(String s, String paramName)

Hypothesis

I think this might be happening because even though we are using VerifierSettings.UseStrictJson(); the verified file that gets created when we set Property1 to an empty string looks like this:

{
  "Property1": ,
  "Property2": "ValueD"
}

As you can see, even though we want to use strict json, the value for the key Property1 doesn't have "". The above json is what we get if we don't have any verified file and we run the test but start with an empty string.

System

This happens when using the following NuGet package versions:

<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.3.0" />
<PackageReference Include="Verify.Quibble" Version="1.2.0" />
<PackageReference Include="Verify.Xunit" Version="17.10.2" />
<PackageReference Include="xunit" Version="2.4.2" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.5">
<PackageReference Include="coverlet.collector" Version="3.1.2">

in the following system:

.NET SDK (reflecting any global.json):
 Version:   6.0.400
 Commit:    7771abd614

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19044
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\6.0.400\

Note

Please let me know if I need to provide more information or if this issue somehow lacking.