Question about the tests about iarna-toml

[iarna]

I'm gonna move the test runner over to https://github.com/iarna/toml-spec-tests so maybe we can do further dev there.

[iarna]

@sgarciac Hey, I'd love to get these more accurate, this is how I test bombadil currently:

https://github.com/iarna/iarna-toml/blob/latest/spec-compliance/local-spec-test.js#L29-L35

    parse: str => {
      // this is assuming that readToml should never throw
      const reader = new bombadil.TomlReader()
      reader.readToml(str)
      if (reader.result === null) throw new BombadilError(reader.errors)
      return reader.result
    }

That is, I build an interface around it that I believe works the same as the other TOML parsers, throwing on errors.

[iarna]

My raw test data is here, if you want to use it:

https://github.com/iarna/toml-spec-tests/tree/0.5.0

(They're YAML which works like the yaml/toml pairs in the spec, and JSON which is the same format as BurntSushi's tests, with the addition of date, time and datetime-local types.)

[sgarciac]

@iarna Great, I'll verify what is it that is making Bombadil fail for most of the invalid toml tests when running your test scripts, I imagine that in some cases even when Bombadil detect errors, it is returning something other than null in result. (according to my own execution of toml-test, Bombadil passes all of BurntSushi's tests except for the big integers one).

I already had a go at your tests for 0.5 and catched a few bugs that I need to fix, thanks!

EDIT: ouch, contrary to what the README states, Bombadil actually returns undefined, not null, for invalid TOML documents.

[iarna]

oh! ok, lemme fix that!

[iarna]

I've updated the results:

https://shared.by.re-becca.org/misc/TOML-SPEC-SUPPORT.html

Much more green!

[iarna]

@sgarciac The reason BurntSushi 0.4.0: should throw: string-bad-codepoint is failing is because bombadil is throwing instead of filling in errors. If throwing is appropriate there, I'm happy to change that assertion.

[iarna]

Oh, while I've got your ear, I've started to add more QA-style tests which you can see in the results I posted. Right now they're just scaling type things, though I have a bit more to do there (I was prompted to write this toml parser due to a crash in toml-v0.4 that I haven't reproed in a test yet).

But I intend to add more black-boxy tests around corners that are likely trick up more lax parsers.

[sgarciac]

@iarna great! Bombadil should not throw any errors when parsing, so string-bad-codepoint is indeed a legitime test fail. I'll be looking forward to the new tests. I already started fixing the bugs in bombadil found by your tests and I'll PR to update bombadil's version in your test scripts when I release the fixes.

It is a shame that BurntSushi's toml-test project is sort of dead, it is practical to have a shared set of tests.

[sgarciac]

Hello @iarna, I'm gonna make a pull request to toml-test, including some of your tests, if you don't mind. (it's practical for me to have them on toml-test, it makes it easier to test both of my parsers).

(In my PR, I'm changing the format specification to add types for 'local-datetime', 'local-date', and 'local-time' (offset datetimes will still be labeled 'datetime' to keep old tests compatible).)

[iarna]

@sgarciac I noticed that because I was testing from your branch of the toml-test suite! It broke things because of a misalignment on those types. I've been using:

• datetime-local - A datetime without a timezone. Floating.
• date - A date without any time component
• time - A time without any date component

I'm not terribly attached though.

[sgarciac]

Those make more sense than mine, I'll use them instead