Fix colors for OCaml syntax highlighting about vscode-reasonml

[Enzojz]

I have strings systematically un-highlighted, any explain to this?

This is what I got with old OCaml extension:

And this is with ReasonMl extension:

I am using Bimbo theme, but it seems the same under other themes. Under Windows and MSVC built OCaml and Merlin

[maelvls]

I went through the ocaml.ts. The idea of "abstracting" the grammar using typescript seems promising!

I have some questions though:

• don't you think using CSS/styling inside the syntax definition file is going to make the syntax coloring inconsistent between two languages (using the same theme)? E.g., strings colored differently
• maybe the fine-grained highlighting (like let being underlined) should be inside the theme itself? (I know it is not ideal at all)

Imagine that let has the name keyword.other.function-definition.ocaml (example taken from vscode-ocaml). Most themes will highlight let using keyword.other; but if you want to have underlining for this specific case in Dark+, we can add a rule in the theme that uses keyword.other.function-definition?

Now that I think of that, these naming conventions are really limiting because these "conventions" only covers the basic language elements; beyond these, everyone has his own recipe 😞 making themes and syntax definition less and less compatible with each other.

Update: for example, the standard themes only have variables.other to highlight all variables Impossible to highlight differently object variables or static variables (if only we could distinguish them semantically, because textmate only highlights syntactically; for that, we will have to wait for semantic coloring in vscode-language-server).

Blocked on microsoft/vscode#3948.

[Khady]

The colors are not working properly when I'm using the lwt ppx. The let and match colors seems to break.

let test () =
    let _a = 3 in
    match Some 1 with
    | Some _ -> Lwt.return_unit
    | None ->
    let%lwt _v = Lwt.return_none in
    let _placeholder = 3 in
    match Some 2 with
    | Some _ -> Lwt.return_unit
    | None -> Lwt.return_unit

@Khady thanks for the report. I don't think I added any parsing for ppx stuff yet but it shouldn't be hard to fix. I'll make sure it's working in the upcoming released.

@Khady I think your ppx example should be fixed on master now.

[Khady]

Indeed it works. Thanks!

[Khady]

One more error because of a % I think:

let test_lwt () =
  let%lwt _value = begin Lwt.return_unit end [%finally Lwt.return_unit] in
  let val2 = 3 in
  Lwt.return val2

let test f = f (1, 2)

gives me

This issue appears to be due to the [%finally Lwt.return_unit] part of the syntax. If it is removed it looks like it's highlighting correctly:

@Khady I think your second example should be fixed now.

[Khady]

Indeed it works. Thanks!

[maelvls]

Hi,

I have the same issue as @Enzojz: the syntax coloring stops somewhere in the document and all strings are not detected.

I also encountered another issue with the following example foo.ml:

let foo ast :ast =
  let bar = ""
  in
  match ast_without_loc ast with
  | Touist_code (formulas) ->
    eval_ast_formula env (process_formulas (affect_vars formulas))
  | e -> raise_with_loc ast ("this " ^ string_of_ast e ^"\n")

With no space between : and ast, the highlighting breaks:

But adding a space after ::

In the dev tools console, I get the error

 [1] - Grammar is in an endless loop - Grammar pushed & popped a rule without advancing

---

Note: to debug tmlanguages, I hacked grammar-debug; to use it

npm install -g https://github.com/maelvalais/grammar-debug#cli-in-path
grammar-debug ~/.vscode/extensions/vscode-reasonml/syntaxes/ocaml.json foo.ml

Here is the result:

TOKENIZING LINE 1
let foo ast :ast =
[1] - Grammar is in an endless loop - Grammar pushed & popped a rule without advancing

  LINE CONTAINS 8 TOKENS:
    => TOKEN (0-3):
      let
      * source.ocaml
      * keyword.control markup.underline
    => TOKEN (3-4):

      * source.ocaml
    => TOKEN (4-7):
      foo
      * source.ocaml
      * entity.name.function strong emphasis
    => TOKEN (7-8):

      * source.ocaml
    => TOKEN (8-11):
      ast
      * source.ocaml
      * string.other.link variable.language variable.parameter emphasis
    => TOKEN (11-12):

      * source.ocaml
    => TOKEN (12-13):
      :
      * source.ocaml
      * variable.other.class.js message.error variable.interpolation string.regexp strong
    => TOKEN (13-19):
      ast =
      * source.ocaml

  LINE END RULE STACK CONTAINS 4 RULES:
      * [306,1] BeginEndRule "null", entered @13
      * [304,2] BeginEndRule "null", entered @13
      * [263,3] BeginEndRule "null", entered @0
      * [1,4] IncludeOnlyRule "source.ocaml", entered @-1

[maelvls]

By the way, how do you do to add/edit matches in ocaml.json in this repo? After un-minifying it, I feel a bit lost with the ~2800 lines ☺️

Also, when using Developer: inspect TM scopes, I saw that the "names" fields always contain multiple names (à la CSS classes), e.g. in commentBlock:

"name": "comment constant.regexp",
"contentName": "emphasis"

How do you manage the names (CSS classes-like?) Maybe it is customary in VSCode tmLanguages? (I was used to the textmate naming conventions).

cc @freebroccolo

I have strings systematically un-highlighted, any explain to this?

It is just a scoping issue. Should be fixed in the upcoming release.

In the dev tools console, I get the error
[1] - Grammar is in an endless loop - Grammar pushed & popped a rule without advancing

I'll take a look at this. Not sure what the issue is yet.

By the way, how do you do to add/edit matches in ocaml.json in this repo? After un-minifying it, I feel a bit lost with the ~2800 lines ☺️

The textmate grammar is generated from this.

How do you manage the names (CSS classes-like?) Maybe it is customary in VSCode tmLanguages? (I was used to the textmate naming conventions).

Most of the scoping stuff is here. The problem with the textmate "naming conventions" is that they aren't really standard and were simply never adopted very consistently across the different themes and editors. Because of that, I don't really bother even trying to use scope names according to their "semantic" notion. Instead, I try to tune the scopes to look reasonably good across the editors and themes that I test with regularly.

[maelvls]

Thank you for your answers!!

True, the textmate "semantic" conventions are not really well followed (and also kinf of hard to follow). That said, I noticed that the comments are colored differently than for other syntaxes (in Dark+, OCaml comments use the coloring of constant.regexp but in Reason comments they are colored using comment.block:

Maybe we could unify that?

Anyway, thank you for your great work! 😃

The plan is to eventually unify the coloring. What happened is that I rewrote the OCaml highlighter using this more advanced approach but haven't done the same thing for the Reason highlighter yet (which was written by hand and is harder to maintain). I will probably wait until the next round of syntax changes upstream before I finish that.

With no space between : and ast, the highlighting breaks:

But adding a space after ::

In the dev tools console, I get the error
[1] - Grammar is in an endless loop - Grammar pushed & popped a rule without advancing

This should be fixed now in master.

[maelvls]

Wow, that was fast! Thank you! I cloned the repo (and also had to clone ocaml-language-server to make it work) and it is fixed. 😃