W3C test suite about rdf4h

[robstewart57]

This library finally has 100% compliance with the W3c tests in rdf-tests for parsing NTriples, Turtle and RDF/XML 👍

All 1099 tests passed (1.47s)

https://travis-ci.org/robstewart57/rdf4h

Thanks to @wismill for the Turtle parser and XML parser bug fixes!

This compliance is reflected in release 4.0.0 of this library.

[cordawyn]

@robstewart57 I've just committed an update to Makefile to get and "deploy" all W3C test files for the tests (cordawyn@75d67ca). You can finally try things out at your place. I've also updated README.md to mention that special make task. Feel free to adjust Makefile, as it's just hastily concocted to get things running asap.

[cordawyn]

Update on RDF/XML tests (as of b84de4a):

         Test Cases    Total
 Passed  1             1
 Failed  161           161
 Total   162           162

Uh-oh :)

[cordawyn]

But we're doing slightly better on Turtle tests, with my latest fixes:

         Test Cases    Total
 Passed  186           186
 Failed  105           105
 Total   291           291

[robstewart57]

So a 0.6% pass rate for the XML tests leaves room for improvement :-) A 63% pass rate for the Turtle tests is a lot more respectable. It's a great job that we've picked up the W3C tests. I'm travelling for the next two weeks, but after that I would like to tackle some of these bugs, probably starting with the failing Turtle tests given its simpler syntax.

[cordawyn]

Btw, I got a reply from @gkellogg saying they're going to fix those bugs in W3C test files. That's very nice. I'll start compiling a list of the bugs, as we find more.

[cordawyn]

NTriples test (added in 3b1e916) fails for now - it stumbles upon some test file where it starts consuming all of RAM, then dies. I'm going to find out where exactly it fails, but it's definitely not the first file ;-)

UPDATE. The list of culprits:

    <#nt-syntax-bad-esc-01>
    <#nt-syntax-bad-esc-02>
    <#nt-syntax-bad-esc-03>
    <#nt-syntax-bad-string-01>
    <#nt-syntax-bad-string-06>
    <#literal>
    <#literal_ascii_boundaries>
    <#literal_all_controls>
    <#literal_all_punctuation>
    <#literal_with_BACKSPACE>
    <#literal_with_FORM_FEED>
    <#literal_with_UTF8_boundaries>

With the above tests disabled, I'm getting:

         Test Cases   Total       
 Passed  39           39          
 Failed  17           17          
 Total   56           56          

[cordawyn]

@robstewart57 Whenever you're going to start fixing TurtleParser, could you start with #15 ? This would enable me to load "manifest.ttl" of "RDF Entailment" test suite and possibly get some test results from it. Since we don't handle TriG and N-Quads in RDF4H, this should be the last W3C test suite. (Although I don't think we're handling RDF entailment at all. But let's have the tests for it, for completeness sake).

[robstewart57]

For the sake of documentation, #15 is fixed.

[robstewart57]

At the moment when I try testing, I get an unknown pattern PositiveEntailmentTest.

$ cabal test
Building rdf4h-1.2.7...
Preprocessing library rdf4h-1.2.7...
In-place registering rdf4h-1.2.7...
Preprocessing executable 'rdf4h' for rdf4h-1.2.7...
Preprocessing test suite 'test-rdf4h' for rdf4h-1.2.7...
Running 1 test suites...
Test suite test-rdf4h: RUNNING...
test-rdf4h: unknown TestEntry pattern in mfEntryToTest: PositiveEntailmentTest {name = "xmlsch-02-whitespace-facet-4", comment = "\n    Ill-formed datatyped literals now are inconsistent.\n    Used to be negative entailment.\n  ", approval = UNode("http://www.w3.org/ns/rdftest#Approved"), action = UNode("data/w3c/turtle/xmlsch-02/test002.ttl"), result = LNode(TypedL(false,"http://www.w3.org/2001/XMLSchema#boolean")), entailmentRegime = "RDFS", recognizedDatatypes = [UNode("http://www.w3.org/2001/XMLSchema#int")], unrecognizedDatatypes = []}

Is this what I should be seeing at this point?

[cordawyn]

No, please do not touch RDF Entailment tests yet. You can ignore their results. Since we do not handle any kind of inference in RDF4H, I just left those tests half-way through and switched to #12 . I'll get back, of course, but I think we should finish with that issue first.

You can either comment those tests out or add some temporal stub, if that's bothering you.

The rest of W3C tests should be working, though.

[robstewart57]

How do I disable the entailment tests in order to test the rest of the W3C tests?

[cordawyn]

I've just tried my "w3test" branch, no local changes - tests run just fine. The HEAD is "a07888b Merge branch 'master' into w3tests". (It appears I did not commit RDF entailment tests themselves, so the issue that you've mentioned should not happen ;) ). Prehaps you need to update your local branch?

(Btw, you may need to uncomment "W3CNTripleTest.tests" in Test.hs - they are going to enter an infinite loop, as I described above).

[robstewart57]

I've just tried this again, having merged my w3tests with the current head of my master ( dd936aa ). I've just re-ran make test/w3c/fetch and then tried running the tests. The error I still get is:

Test suite test-rdf4h: RUNNING...
test-rdf4h: unknown TestEntry pattern in mfEntryToTest: NegativeEntailmentTest {name = "datatypes-intensional-xsd-integer-decimal-compatible", comment = "\n    The claim that xsd:integer is a subClassOF xsd:decimal is not\n    incompatible with using the intensional semantics for\n    datatypes.\n  ", approval = UNode("http://www.w3.org/ns/rdftest#Approved"), action = UNode("data/w3c/turtle/datatypes-intensional/test001.nt"), result = LNode(TypedL(false,"http://www.w3.org/2001/XMLSchema#boolean")), entailmentRegime = "RDFS", recognizedDatatypes = [UNode("http://www.w3.org/2001/XMLSchema#decimal"),UNode("http://www.w3.org/2001/XMLSchema#integer")], unrecognizedDatatypes = []}
Test suite test-rdf4h: FAIL

@cordawyn I don't understand why you do not also see this error, apart from if your data/w3c/turtle/manifest.ttl has been manually edited to comment out any entries for http://www.w3.org/2001/sw/DataAccess/tests/test-manifest#NegativeEntailmentTest ?

[cordawyn]

@robstewart57 I've just pulled the latest changes, re-downloaded W3C test files afresh and run the tests. No, I'm not getting that error :-/
I get warnings about incomplete pattern matches during compilation, but that's OK for now -- please ignore that, I'll take care of it later (when we start passing W3C tests). Hopefully, it's not the cause of your error. Could you investigate that issue on your side, since I cannot reproduce it?

(Speaking of W3C tests - they seem to have updated the files (hopefully, thanks to our input), but the turtle tests are now packed in a directory, so they get unpacked to "data/w3c/turtle/TurtleTests/". You'll have to update the path in TurtleTest.hs. And it appears they've deleted NQuads (N4) tests too).

[robstewart57]

It doesn't happen with W3CRdfXmlTest.tests. If I comment out the other W3C tests in Test.hs, then I get a report:

         Test Cases    Total        
 Passed  68            68           
 Failed  94            94           
 Total   162           162

If I comment back in the W3CTurtleTest.tests line, then I see the unknown TestEntry pattern in mfEntryToTest ... error. My guess:

1. TurtleTest.hs loads the manifest.ttl tile from data/w3c/turtle/.
2. In triplesToTestEntry, the the UNode http://www.w3.org/2001/sw/DataAccess/tests/test-manifest#NegativeEntailmentTest is pattern matched to call mkNegativeEntailmentTest ts rdf.
3. In manifest.ttl, there are numerous NegativeEntailmentTest entries. An example is below.
4. Thus, numerous TestEntry values with the NegativeEntailmentTest constructor are passed to mfEntryToTest in TurtleTest.hs .
5. The NegativeEntailmentTest pattern is not matched in mfEntryToTest.

The example:

<#datatypes-intensional-xsd-integer-decimal-compatible> a mf:NegativeEntailmentTest;
  mf:name "datatypes-intensional-xsd-integer-decimal-compatible";
  rdfs:comment """
    The claim that xsd:integer is a subClassOF xsd:decimal is not
    incompatible with using the intensional semantics for
    datatypes.
  """;
  rdfs:approval rdft:Approved;
  mf:entailmentRegime "RDFS" ;
  mf:recognizedDatatypes ( xsd:decimal xsd:integer ) ;
  mf:unrecognizedDatatypes ( ) ;
  mf:action <datatypes-intensional/test001.nt>;
  mf:result false .

[cordawyn]

Only the tests in the manifest of "rdf-mt", that we do not use, have NegativeEntailmentTest entries, I double-checked that. So that UNode... shouldn't even match anywhere. Perhaps your manifest.ttl files are messed up or something? Could you remove "data/w3c" and download the latest W3C tests?

[robstewart57]

Aha, thanks! Yes, somehow the manifest.ttl file had incorrect content. I might even admit to wgeting it manually a few weeks ago from the wrong URL :-) My bad. It works nicely now, which gives me something to work on...

         Test Cases    Total        
 Passed  185           185          
 Failed  106           106          
 Total   291           291

[robstewart57]

As of deb4632 :

         Test Cases    Total        
 Passed  190           190          
 Failed  101           101          
 Total   291           291

About half the parsec combinators follow the W3C grammar rules for Turtle. The other combinators are combinations of numerous grammar rules, so are less easy to correspond (and therefore to debug). This perhaps reflects the accepted proposoals in the change log added after the first TurtleParser implementation. http://www.w3.org/TR/turtle/#sec-changelog

[robstewart57]

As of f891ac5 :

         Test Cases    Total        
 Passed  203           203          
 Failed  88            88           
 Total   291           291

(I'm now out of action for a couple of weeks)

[ddssff]

RDF newbie here - how concerned should I be about these failures?

[robstewart57]

Hi @ddssff

These failing cases are corner cases for the N3, TTL and RDF/XML parsers. This unit test suite is the one specified by the W3C RDF specification. It's on my TODO list to squish these parser bugs in the next month or so.

As a side point, I notice that you clones from amccausl/RDF4H . The upstream version of RDF4H is this repository at https://github.com/robstewart57/rdf4h .

Do please create Github issues to track any bugs you encounter with RDF4H.

[gkellogg]

When I can carve out some time, I'd like to get these added to our copies of the various language test suites.

[ddssff]

Thanks, I found the newer repo a few minutes later. I will be in touch.

[cordawyn]

@robstewart57 Rob, I'm trying the latest stuff from master branch, but the tests (stack test) seem to fail, because the turtle parser fails to parse the manifest files. I tried this in ghci:

*Data.RDF> let f = parseFile (TurtleParser (Just $ BaseUrl "") Nothing) "data/w3c/turtle/TurtleTests/manifest.ttl" :: IO (Either ParseFailure TriplesGraph)
*Data.RDF> f
Left (ParseFailure "(line 18, column 2):\nunexpected Invalid URI in Turtle parser: \"\"\nexpecting subject resource")

So it seems to stumble while reading the manifest file:

...
<>  rdf:type mf:Manifest ;
...

If, however, I provide some base URI for the parser, even more terrible things happen:

*Data.RDF> let f = parseFile (TurtleParser (Just $ BaseUrl "http://example.org/") Nothing) "data/w3c/turtle/TurtleTests/manifest.ttl" :: IO (Either ParseFailure TriplesGraph)
*Data.RDF> f
*** Exception: data/w3c/turtle/TurtleTests/manifest.ttl: hGetContents: invalid argument (invalid byte sequence)
*Data.RDF> *** Error in `/home/vagrant/.stack/programs/x86_64-linux/ghc-7.10.2/lib/ghc-7.10.2/bin/ghc': double free or corruption (out): 0x00007f8970008a50 ***
Aborted

[robstewart57]

An update.. For e62e714 across all tests, i.e. API property and test cases and the W3C test cases:

         Properties   Test Cases    Total        
 Passed  61           482           543          
 Failed  2            122           124          
 Total   63           604           667

Which is down from about 180 failing test cases a few months ago, prior to the current round of test driven development, so progress is good if a little steady.

[robstewart57]

The current W3C test results are:

• NTriples

  stack test --test-arguments="--pattern /parser-w3c-tests-ntriples/"

All 136 tests passed (0.14s)

• Turtle

  stack test --test-arguments="--pattern /parser-w3c-tests-turtle/"

All 596 tests passed (1.17s)

• RDF/XML

  stack test --test-arguments="--pattern /parser-w3c-tests-xml/"

47 out of 162 tests failed (1.03s)

I will close this issue ticket once all RDF/XML parser bugs have been fixed.