4.05 compatibility about camomile

[gasche]

(After more discussion upstream, it is clear that the Bigarray problem will be resolved soon, so that no change on the Camomile side is required.)

[gasche]

@michipili, @seliopou: I am worried about the "Assertion failed" problem that may be a 4.05 regression, but I don't know the Camomile codebase enough to understand what is going on. Any help from you to identify the root of the issue (and whether it looks like an ocaml-side bug) would be warmly welcome.

[yoriyuki]

Did "Assertion failed" appear in early versions? The problem is caused by someone trying to compare two characters in Quaternary level precision using AbsCe. This should not happen, because weights in Quaternary level precision are algorithmically derived from weights in other precisions.

If this error happens in earlier versions of OCaml, this could be a bug in Camomile (but I have no idea when the bug enter the code). Otherwise, this could be a serious bug in OCaml, possibly related to polymorphic variants or parser/lexer.

[gasche]

Thanks! I believe that the error is indeed new to 4.05 -- I can give it another try in case it was build environment thing. Because it occurs during the normal build process, I suppose that it would have been caught before.

[gasche]

I did more tests. I can confirm that the problem is new to 4.05 -- the build, including this test, works fine in 4.04.
What I also noticed is that while tools/camomilelocaledef.opt --file locales/test_1.txt locales fails, the same thing with tools/camomilelocaledef.byte works correctly, even under 4.05. So there is a failure with the native-compiled tool, but not with the bytecode-compiled tool.

I also tested the code under the trunk version of OCaml (4.05 is the release branch that has less changes), and it also fails there. (I previously wrote here that this ruled out MPR#7346 as guilty of this change, because I believed that it was in trunk only. The corresponding change is in 4.05 as well, so it may still be responsible. I will do more tests.)

Note that the existence of a difference in evaluation order between byte- and native-compiled programs is not a sure sign of a bug in the compiler. It may be that the program is wrongly relying on an unspecified evaluation order. That said, even if that was the case, a previously-unknown difference in behavior between byte- and native-compiled program would be worthy of an issue report and possibly of an upstream change.

To summarize:

• the program works correctly under 4.04, under either byte or native compilation
• the program fails under 4.05 or trunk under native compilation, but work (under 4.05 or trunk) with the bytecode compiler.

(These tests were done without flambda.)

[gasche]

I have tested that reverting either GPR#966 or GPR#967 (the two obvious evaluation-order-related changes in 4.05) does not fix the build failure. I will try a more classic bisection of 4.05 changes.

[yoriyuki]

I don't see how the issue is related to evaluation order. The assertion states that Quaternary polymorphic variant somehow enters into AbsCe.prev, which should not happen. (I agree that it should not be checked by runtime, because it can be checked in compile time. But this is another issue.)

IIRC, polymorphic variants is converted to integers by hash. Do we find finally a hash collision?

[gasche]

The reason why I test for evaluation order is that it is the classic source of difference in behavior between the bytecode and native compiler. I don't know how the Quaternary variant is produced by Camomile, it could have been the result of a computation that does side-effects in some order, and might return something else with a different order. (You seem to say that is not the case).

Re hash conflicts: I think that there is a test at compile-time that distinct variants all get hashed differently, so I don't think that user program can be bitten by this without a compilation failure.

[yoriyuki]

I don't know how the Quaternary variant is produced by Camomile, it could have been the result of a computation that does side-effects in some order, and might return something else with a different order. (You seem to say that is not the case).

It is theoretically possible, but the Quaternary variant is not used for a complex computation. It is just used as an optional argument to change the behavior of functions.

I quickly searched the source and I could not find any place where the Quaternary variant can enter AbsCe.prev. It is only used in toolslib/colParser.mly, and in all calls to AbsCe.prev, it checks that the argument is not Quaternary.

One possibility is to change the type signature of AbsCe.prev to see where Quaternary variant enters this function.

[yoriyuki]

One possibility is to change the type signature of AbsCe.prev to see where Quaternary variant enters this function.

I did. I changed the type signature of AbsCe.prev from UCol.prev to [Primary | Secondary | Tertiary] . It compiles and runs fine. (I use ocaml 4.02.1) So there is no possibility to enter Quaternary into AbsCe.prev.

[gasche]

A value that the type-checker can rule out! You have convinced me that this is a compiler bug. I am bisecting to find the guilty change and will report upstream (and here). Thanks a lot for your help.

[gasche]

Indeed, the issue came from a faulty optimization acting on the polymorphic variants. We will get a fix in 4.05 upstream.

[gasche]

@yoriyuki given the various investigations, I believe that the pull request #20 is the only change that is required on Camomile's side for 4.05 compatibility (we will see about the Unix dependency thing, it may or may not require a change later). I would thus recommend merging it.

My plan is to apply the change as a patch to Camomile's released OPAM package, so that, as soon as the two other issues are fixed upstream, the existing Camomile version can be built under 4.05, and the packages that depend on it can themselves be tested under 4.05 before the release.

[yoriyuki]

Yes. now I'm checking that it builds safely in my environment.

[yoriyuki]

#20 Merged. I close the issue. Please reopen if you find other changes are necessary to 4.05 compatibility.

[gasche]

Thanks! I don't expect to find any further issue (with all the fixes in my experimental branch, I was able to build Camomile). I will reopen if the Unix dependency is not reverted upstream, to add it to Camomile.