To all Ren'Py users! We need your help! - Full Ren'Py grammar definition for 'intellisense' about vscode-language-renpy

[Gouvernathor]

Some notes about those :

• what we consider to be a string doesn't always follow python rules. For example, we allow backquotes as delimiters for dialogue lines, line breaking behavior is... let's say weird (we allow it for not-triple strings), and we usually don't support f, r and u as prepended. All that, at least for dialogue lines, I'm not sure if it's specific or if shared among several statements parsing.
• the python block is not exactly as you describe : "hide" and "in x" are mutually exclusive, and I doubt "early" takes either of the two (gotta check)
• "end translate" is something we implicitly generate in AST, but it's not represented with characters in the file : the end translate of a non-None language is a block ending, and of a None translate (implicit too), it's just the end of a dialogue line. So it should probably not be in your list.
• del is not a renpy statement, it's a python statement. Not sure why it ended there.
• the SCREEN_ section looks wrong

Ill increment this list.

[renpytom]

Question - where is the grammar located? I'd be interested in taking a look at it, and seeing what I can do.

[duckdoom4]

Something to look into:
https://code.visualstudio.com/api/extension-guides/virtual-documents
https://code.visualstudio.com/api/language-extensions/embedded-languages

Basically we could already add python's language server features by building a virtual document of the python code and then adding the results to our own providers. This would be incomplete though, since we'd be missing a lot of the python context. It might still be nice enough to be worth adding already

[duckdoom4]

In any case I guess that would require warning you when we make changes to the grammar, preferably before release. It will require some discipline on our end.

Maybe we could build an official specification together? That would help on both ends I suppose.

[Gouvernathor]

I think LABEL_NAME should be [[NAME], "."], NAME. Assuming I understand the syntax correctly, there can't be label .name.be.

image_specifier seems wrong in that it seems to allow several behind_expression (for example) to be parsed in a row, as different image_specifier_clause. It needs to allow at most one of each, in any order. I'm not sure how to specify that, though.
Same with menu_block, possibly. If not it's our fault enabling set and caption to come before choices. But in any case there can be only one caption/say in a given menu statement.
Same with style_clause and probably others. Maybe I don't understand the bnf syntax though.

say_attributes should be {["-"], IMAGE_NAME_COMPONENT_NO_DASH}, I believe. The dashes can only be at the beginning of image attributes, and there can be any number of negated attributes.

The layeredimage statement is missing, as well as a number of statements defined in renpy/common : the audio statements (some of them at least) are part of those I think. They are UserStatements, technically, but they're also built-in.

It's just cosmetic maybe, but I wouldn't put the if_expression in common between the if clause and the menu statement, for me it's just a common keyword just as the in operator and the for x in y syntax in python. But it's just my view of things, it works the same.

rpy_python is just "rpy", "python", "3". Until python 4 comes out, I guess. Though I may want a rpy python annotations, but we'll see.
testcase is still work in progress, not merged yet and the semantics and grammar may still evolve.

[Gouvernathor]

It came up some time ago for some reason and the answer was no, I doubt it has been done since. There were some bugs (in renpy/renpy#3991 for example) where a grammar would certainly have helped flagging them, but that example also means we are changing it occasionally. And it's possibly a bag of knots already, though I'm not a connaisseur of BNF so I may be wrong as compared to other languages. Our parser code is not that long and quite regular, so who knows.

In any case I guess that would require warning you when we make changes to the grammar, preferably before release. It will require some discipline on our end.

[Gouvernathor]

The init $ is probably intended, sort of. And archaic. The others, probably not.

[duckdoom4]

Can I put in a request as well, kinda like typescript has or even pylint allowing @ignore lines? When working on mods the undefined storemodule stuff, while it's technically correct, in practice isn't due to the presence of base scripts (inside .rpa files)

As such it leads to unnecessary warnings, and it'd be great to kinda have a # @renpy-ignore-next-line or some more fine tuned diagnostic rules, if not just # @renpy-expect-error <description> i.e. TS

Yeah we can implement this feature when the other features are implemented. I think this is a nice to have

[duckdoom4]

@renpytom / @Gouvernathor does such a specification already exist? That would be very helpfull :)

[duckdoom4]

@Gouvernathor, I used Bing Chat (🤣) to extract some EBNF definitions from the official Ren'Py parser source. It wasn't 100% accurate, so after it had processed most of the statements I manually went over them.
I've uploaded the very rough draft here.

The entries at the bottom of the file (after # Renpy Expressions) are the ones I manually reviewed and improved based on the source code. A review would be appreciated, but feel free to do so later :). I will continue to work on this for the next few days

[duckdoom4]

Ha, yes. I see it did a really bad job on the python statement. There is a lot more to it then this. Good catch

[duckdoom4]

@Gouvernathor Do I read correctly that the current parser will allow any statement to start with init?
https://github.com/renpy/renpy/blob/a33f891d65d3fd2b59a533c365800a59f8e15f0c/renpy/parser.py#L1167

So that would mean things like:

init $ <python_expression>
init init label
init return
init init init init ...

Would all be valid? (Note that this is not an init block!)

Bug? Yes it is :) renpy/renpy#4713

[duckdoom4]

The init $ is probably intended, sort of. And archaic. The others, probably not.

Yeah, kinda makes sense.

I've updated the grammar file again. With the exception of screen and translate, all 'normal' (meaning non screen- or ATL blocks) are now in there. Another review would be appreciated :)

(You can ignore the bits below (* GENERATED *), I'm gonna manually convert the code anyway)

[duckdoom4]

Question - where is the grammar located? I'd be interested in taking a look at it, and seeing what I can do.

I have the grammars located on this branch
(renpy-grammar-specification/grammars/).

Thanks in advance for taking the time to help :)

[multimokia]

Can I put in a request as well, kinda like typescript has or even pylint allowing @ignore lines? When working on mods the undefined storemodule stuff, while it's technically correct, in practice isn't due to the presence of base scripts (inside .rpa files)

As such it leads to unnecessary warnings, and it'd be great to kinda have a # @renpy-ignore-next-line or some more fine tuned diagnostic rules, if not just # @renpy-expect-error <description> i.e. TS