Can't handle anchored lists about yamldotnet

[hughesjs]

Following some advice on SO I tried parsing with a MergingParser:

MergingParser               parser       = new(new Parser(new StringReader(_aghYml)));
FluidApiMethodDefinition[]? res          = deserializer.Deserialize<FluidApiMethodDefinition[]>(parser);

But now I'm getting this exception:

System.Collections.Generic.KeyNotFoundException: The given key 'AxisSettingsCircular' was not present in the dictionary.

System.Collections.Generic.KeyNotFoundException
The given key 'AxisSettingsCircular' was not present in the dictionary.
at System.Collections.Generic.Dictionary2.get_Item(TKey key) at YamlDotNet.Core.MergingParser.ParsingEventCollection.FromAnchor(AnchorName anchor) at YamlDotNet.Core.MergingParser.GetMappingEvents(AnchorName anchor) at YamlDotNet.Core.MergingParser.HandleAnchorAlias(LinkedListNode1 node, LinkedListNode1 anchorNode, AnchorAlias anchorAlias) at YamlDotNet.Core.MergingParser.HandleMerge(LinkedListNode1 node)
at YamlDotNet.Core.MergingParser.Merge()
at YamlDotNet.Core.MergingParser.MoveNext()
at YamlDotNet.Core.ParserExtensions.Accept[T](IParser parser, T& event)
at YamlDotNet.Core.ParserExtensions.TryConsume[T](IParser parser, T& event)
at YamlDotNet.Serialization.Deserializer.Deserialize(IParser parser, Type type)
at YamlDotNet.Serialization.Deserializer.Deserialize[T](IParser parser)
at SuperFluid.Tests.SourceGenerators.FluidGeneratorServiceTests.Agh() in /home/james/repos/SuperFluid/src/SuperFluid.Tests/SourceGenerators/FluidGeneratorServiceTests.cs:line 32
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

[perlpunk]

If you simply want to reuse the list, just do this:

CanTransitionTo: *AxisSettingsCircular

The << merge key is for merging mappings.

[hughesjs]

Ideally I need to support both cases but if I do what you suggest, I get this exception:

YamlDotNet.Core.YamlException: Expected 'SequenceStart', got 'MappingStart' (at Line: 1, Col: 1, Idx: 0).

YamlDotNet.Core.YamlException
Expected 'SequenceStart', got 'MappingStart' (at Line: 1, Col: 1, Idx: 0).
at YamlDotNet.Core.ParserExtensions.Require[T](IParser parser)
at YamlDotNet.Core.ParserExtensions.Consume[T](IParser parser)
at YamlDotNet.Serialization.NodeDeserializers.CollectionNodeDeserializer.DeserializeHelper(Type tItem, IParser parser, Func3 nestedObjectDeserializer, IList result, Boolean canUpdate) at YamlDotNet.Serialization.NodeDeserializers.ArrayNodeDeserializer.Deserialize(IParser parser, Type expectedType, Func3 nestedObjectDeserializer, Object& value)
at YamlDotNet.Serialization.ValueDeserializers.NodeValueDeserializer.DeserializeValue(IParser parser, Type expectedType, SerializerState state, IValueDeserializer nestedObjectDeserializer)
at YamlDotNet.Serialization.ValueDeserializers.AliasValueDeserializer.DeserializeValue(IParser parser, Type expectedType, SerializerState state, IValueDeserializer nestedObjectDeserializer)
at YamlDotNet.Serialization.Deserializer.Deserialize(IParser parser, Type type)
at YamlDotNet.Serialization.Deserializer.Deserialize[T](IParser parser)
at SuperFluid.Tests.SourceGenerators.FluidGeneratorServiceTests.Agh() in /home/james/repos/SuperFluid/src/SuperFluid.Tests/SourceGenerators/FluidGeneratorServiceTests.cs:line 32
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

[EdwardCooke]

There probably is a bug where << is referenced as something other than a scalar, which is what it should be seen as, Which would fail in your instance anyways, because << is not a property on your class, which would be hard to do, if at all possible in C#. However, the following does work:

using System.Diagnostics;
using YamlDotNet.Serialization;

var yaml = @"
Methods:
  - Name: ""AddPgfPlotWithAxes""
    Arguments:
      - Name: ""axisType""
        Type: ""AxisType""
      - Name: ""options""
        Type: ""AxisOptions?""
    CanTransitionTo: &AxisSettingsCircular
      - ""AddPlot""
      - ""SetXLabel""
      - ""SetYLabel""
      - ""SetXMin""
      - ""SetYMin""
      - ""SetXMax""
      - ""SetYMax""
      - ""SetMinorXTickNumber""
      - ""SetMinorYTickNumber""
      - ""SetXTicks""
      - ""SetYTicks""
      - ""SetGrid""
  - Name: ""SetXLabel""
    Arguments:
      - Name: ""label""
        Type: ""string?""
    CanTransitionTo:
      *AxisSettingsCircular";
var deserializer = new DeserializerBuilder().Build();
var o = deserializer.Deserialize<X>(yaml);
var serializer = new SerializerBuilder().Build();
var newYaml = serializer.Serialize(o);
Console.WriteLine(newYaml);

record FluidApiArgumentDefinition
{
    public string Name { get; set; }
    public string Type { get; set; }
}

[DebuggerDisplay("{Name}")]
 record FluidApiMethodDefinition
{
    public string Name { get; init; }
    public string? ReturnType { get; init; }
    public List<string> CanTransitionTo { get; init; } = new();

    public List<FluidApiArgumentDefinition> Arguments { get; init; } = new();
}

record X
{
    public List<FluidApiMethodDefinition> Methods { get; set; }
}

Notice the difference, the removal of <<: from the yaml.
The reference parser for your yaml is here: http://ben-kiki.org/ypaste/data/77643/index.html
Notice that the << is a scalar, shouldn't have any real meaning to the parser itself.

[hughesjs]

See, my issue here is that I don't think that's valid YAML at that point, at least in terms of what I'm trying to accomplish. If it was just for my own usage, that would be fine, but I'm writing a library for third-party consumption so it needs to consume proper YAML.

I don't know how far I'll get with it, because honestly I suck at parsers, but I'll fork and see if I can get this working if you think it's a bug.

I'm also somewhat confused by you saying that << is just a scalar, I was under the impression that this is the merge key?

As a side note, I don't know if you've seen C#11 raw strings? They make embedding stuff like this a lot easier since you don't have to worry about all the escaping and left-aligning everything.

[hughesjs]

So I've written this test case:

public class MergeListTest
{
    private const string YAML = """
                                Collection:
                                  - Name: "List 1"
                                    InnerList: &List1
                                      - "Item 1"
                                      - "Item 2"
                                  - Name: "List 2"
                                    InnerList:
                                      <<: *List1                                          
                                """;

    [Fact]
    public void CanMergeAnchoredList()
    {
        MergingParser parser = new(new Parser(new StringReader(YAML)));
        var deserializer = new DeserializerBuilder().Build();
        var res = deserializer.Deserialize<TestClass[]>(parser);

        res.Length.Should().Be(2);
        res[0].Name.Should().Be("List 1");
        res[1].Name.Should().Be("List 2");
        res[0].InnerList.ShouldBeEquivalentTo(res[1].InnerList);
    }


    private class TestClass
    {
        public string Name { get; set; }
        public string[] InnerList { get; set; }
    }
}

Stepping through the MergingParser, I get to here:

public IEnumerable<LinkedListNode<ParsingEvent>> FromAnchor(AnchorName anchor)
{
    var node = references[anchor].Next;
    return Enumerate(node);
}

Where references is actually empty, so I think the issue lies wherever in the code is meant to have added the reference when it encountered &List1 whilst parsing the first item...

I'll keep looking

[hughesjs]

Seems as though it only handles mapping merges not sequence merges:

  private void AddReference(ParsingEvent item, LinkedListNode<ParsingEvent> node)
  {
      if (item is MappingStart mappingStart)
      {
          var anchor = mappingStart.Anchor;
          if (!anchor.IsEmpty)
          {
              references[anchor] = node;
          }
      }
  }
}

The ParsingEvent for List1 is:

Sequence start [anchor = List1, tag = ?, isImplicit = True, style = Block]

[perlpunk]

Seems as though it only handles mapping merges not sequence merges:

That's what I wrote earlier, maybe you overlooked what I wrote.
The << merge key is for merging mappings, and only mappings. Here is the specification: http://yaml.org/type/merge.html

It can take a sequence as a value, as you can see in that example, but that's for merging multiple mappings, not merging sequences.

[hughesjs]

Is that YAML parser just parsing undefined syntax then and totally misleading me..?

I have, however, just retried your example syntax and that seems to work on that online parser if I remove the linebreak:

Collection:
  - Name: "List 1"
    InnerList: &List1
      - "Item 1"
      - "Item 2"
  - Name: "List 2"
    InnerList: *List1     

And this modified testcase based on yours seems to work too:

    public class MergeListTest
    {
        private const string YAML = """
                                    Collection:
                                      - Name: "List 1"
                                        InnerList: &List1
                                          - "Item 1"
                                          - "Item 2"
                                      - Name: "List 2"
                                        InnerList: *List1                                          
                                    """;

        [Fact]
        public void CanMergeAnchoredList()
        {
            var deserializer = new DeserializerBuilder().Build();
            var res = deserializer.Deserialize<X>(YAML);

            res.Collection.Length.Should().Be(2);
            res.Collection[0].Name.Should().Be("List 1");
            res.Collection[1].Name.Should().Be("List 2");
            res.Collection[0].InnerList.ShouldBeEquivalentTo(res.Collection[1].InnerList);
        }


        private class TestClass
        {
            public string Name { get; set; }
            public string[] InnerList { get; set; }
        }


        private class X
        {
            public TestClass[] Collection { get; set; }
        }
    }

Sorry if I'm going round in circles a bit here, my knowledge of YAML's different components isn't the strongest.

Seems a bit of oversight in the spec to not be able to merge different sequences, that could be really useful.

I guess my last question would be, do I need that wrapper class, or is there a way to deserialize straight to a collection?

Thanks for your help here btw.

[perlpunk]

Is that YAML parser just parsing undefined syntax then and totally misleading me..?

I don't know which parser that is using. It is showing a behaviour that's not specified here.

and it is behaving very strange if I add another key/value pair at the bottom, e.g.

# ...
    CanTransitionTo:
      <<: *AxisSettingsCircular
      a: b

The result doesn't look like something that's possible, but I'm not too familiar with that syntax on the right side.

CanTransitionTo {13}
0: AddPlot
...
10: SetYTicks
11: SetGrid
a: b

So is that supposed to be an array or a dictionary?

Second problematic behaviour: When the string << is quoted, then it should be treated as a simple string and not as a merge key.

    CanTransitionTo:
      "<<": *AxisSettingsCircular

https://codebeautify.org/ however doesn't make a difference, it just ignores the quotes. That's bad. YAML is a data serialization language, it's perfectly possible to have the string << as a mapping key.

I have, however, just retried your example syntax and that seems to work if I remove the linebreak:

It should also work with a line break.

a: *alias
b:
  * alias

should both work.

Seems a bit of oversight in the spec to not be able to merge different sequences, that could be really useful

The syntax <<: *some_sequence with a sequence as a value is already reserved for merging multiple mappings.
Concatenating sequences (calling it merging does not make much sense for sequences) would have to get a different syntax.
It's not an oversight. Inventing a special string like << for just one programmatic feature has already been a mistake, although I agree that it's useful and I wouldn't want to miss it. But it's a pain to implement for a full featured YAML processor that's allowing also custom constructors. I implemented such a processor. I also contribute to PyYAML, where the implementation of the << feature currently prevents detecting duplicate keys in mappings. A refactoring of the code would be necessary.
YAML is not a programming language.
And concatenating sequences is not the only thing that people ask for.
What would also be useful is a deep merge. << only does a top level merge. We can't just define a special string for each of these features

[hughesjs]

Yeah fair one pal, thanks for taking the time to explain all of that for me. Seems as though that site has just made me completely misunderstand what's supported with YAML and isn't an issue here after-all so I'm going to close this.

Thanks again!