FIXME - java about bison

[bokrenok]

Hello, @bitPogo!
I've just run into this while investigating the way syntax errors are printed.
And I would like to share what I've found.
(BTW my project is for C++, but the whole idea is the same and source code is very similar).

TL;DR;
When the mentioned line is reached by the code execution, it means that there were found some expected tokens, but the allowed limit yyargn has been reached and the execution of the function getExpectedTokens would result in 'no expected tokens found'... which could be quite surprising.

Full story:
So, I was investigating why in some cases syntax errors display expecting tokens like this:

syntax error, unexpected =, expecting byte or le16 or be16

and in other cases there are no expecting tokens displayed like this:

syntax error, unexpected =

It turned out that the function getExpectedTokens (or expected_tokens in C++) is intended to return no more than given yyargn possible tokens.
Whenever there are more possible tokens - the function returns no tokens at all.
The line you are referred to is exactly about it: if there are more than yyargn tokens possible - none of them are returned at all.

And the yyargn value is set by TOKENMAX = 5 constant value, which is defined right before getExpectedTokens call.
I suppose that for java it is kind of legacy from original C/C++, which can be rewritten in a better way.

But anyway there is kind of a dilemma: what should be done in case of lots of possible tokens?
Given that the list of them is to be printed in a syntax error message for user.

1. should it return a huge list of alternatives (ending up whith quite a long enumeration of values ' ... or ... or ... or ... ')
2. should it return at least yyargn number of them (and lose info about other possibilities, basically restricting the list only to fixed amout of data)
3. should it return nothing at all (leaving the user without any key to correct values which are usable here)
   and this is what it does right now and ever before.

As I sought through the repo I found that for the first time the constant YYERROR_VERBOSE_ARGS_MAXIMUM was introduced in this commit: a0ffc17
and then rewritten with additional comments here d2060f0
And earlier it was just if (count < 5)

[bitPogo]

Thx for the explanation and the pointers!

Yeah well...the dilemma is tricky in terms of usability. 1) is maybe ugly but gives you the most possible information and is open to refinement I suppose (like the famous did you mean XXX or at least nice formatting), while 3) is in the end no help at all in my eyes. That said, I think for Bison itself there should be a discussion before committing any thing, how to deal with it.

In general the Java Source could use quite some polishing even while Bison is maybe not so popular in realm of Java for some reasons. Also that could benefit its flexibility in terms of modification (which I really adore after spending some time with it aside from M4). And I bet that needs discussion as well upfront.

For my current use case I figured out (unfortunately), Bison is not right hammer for that nail. But what is maybe good to know - with light modification in the current Java Skeleton and some post processing once the source is converted to Kotlin it is possible to use it for KMP. (Of course only with a certain degree of certainty, since I did not made exhausted tests)

All that said I close the issue for now.