syntax highlighting : standard strings in Python about nextvi

[kyx0r]

Hi,

The syntax highlighting pattern for standard Python strings fails with embedded escaped quotes.

Yeah, that's probably because nobody ever bothered to make them better :D
For instance c quotes pattern is exceptionally robust. I rarely deal with python in my work
so it had less of my attention.

P. S. I am trying to understand how the highlight structure's end and blkend work. Please give me some hints.

I am in the processes of making changes with some ideas gained over the year, so readme is a bit out of date. It will be updated once I feel like I've exhausted my ideas.

The way end works, its a 16 byte array of chars, each char can take 3 states: -1, 0, 1. It determines where the result of a pattern match continues. 16 corresponds to 16 groups that could be defined, more than enough.

0 means nothing special happens (default) advance sp by the length of the matched group.
-1 means the pattern group that matched will be highlighted but treated like it didn't match. thus the string pointer is set to sp+1. Notice the +1 is necessary otherwise we would be stuck in an infinite loop matching the same pattern over and over. So its not exactly no match operation, more like a way to let other patterns passthrough.
1 same concept as -1 except we advance by the starting location of the group match. i.e if match was (5,10) add sp+5

blkend tells which group is going to terminate the multiline block. if its negative the same group that started the block terminates the block. There are a few gochas, for example when screen is positioned going up instead of down, it is always assumed that group 1 will do the termination in that case.

Hope this helps, I know its arcane, but the purpose is to have a minimal and lean implementation.

Hi, As I continue to experiment, I am often surprised by the results. The order changes the behaviour: putting the triple-quoted string patterns before the single-quoted string patterns produces a better result. And I have returned to the original patterns for the single-quoted strings. :-) Here are the patterns I am using for Python. /* python */ {"py", NULL, {14 | SYN_BD}, {1}, 0, 2}, {"py", NULL, {9}, {0}, 0, 1}, {"py", "#.*", {2}}, {"py", "\\<(?:and|break|class|continue|def|del|elif|else|except|finally|\ for|from|global|if|import|in|is|lambda|not|or|pass|print|raise|return|try|while)\\>", {5}}, {"py", "([a-zA-Z0-9_]+)\\(", {0, SYN_BD}}, /* triple-quoted strings */ {"py", "\"{3}.*?\"{3}", {4 | SYN_IT}, {0}}, /* works on one line */ /* multi-line experimentation */ {"py", "((?:[!\"\"\"]*\"{3}\n$)|(?:^\"{3}.*)|\"{3})", {4 | SYN_IT}, {0}, -1}, /* single-quoted strings */ {"py", "[\"](\\\\\"|[^\"])*?[\"]", {4}}, {"py", "['](\\\\'|[^'])*?[']", {4}}, I have attached the old Python program which I have been using for testing; I added extra strings near the top. The highlighting is quite good, but the result is sometimes bizarre: on line 591, the *fd.write* call uses a triple-quoted string; the string ends on line 616, but the highlighting continues until line 626. The highlighting changes as I move around in the file. Is that expected? Is this all because multi-line is difficult? I have not worked this much with C and regular expressions in a long time. Have a good day. P. S. I am using yesterday's master and building with the *build.sh* script.. My computer runs Debian 11 on an AMD64. P. P. S. I am going to use the following since the result is the least surprising. The triple quotes of multi-line strings are highlighted and the contents are not; single-line strings are fully highlighted. /* triple-quoted strings */ {"py", "\"{3}.*?\"{3}", {4 | SYN_IT}, {0}}, /* works on one line */ /* multi-line experimentation */ {"py", "(\"{3})", {4 | SYN_RV}},

…

-- Claude Marinier

[kyx0r]

I have attached the old Python program which I have been using for testing;

Use null pointer, I don't think github reply attachments work (or you forgot to attach it).
https://0x0.st/

I added extra strings near the top. The highlighting is quite good, but the result is sometimes bizarre: on line 591, the fd.write call uses a triple-quoted string; the string ends on line 616, but the highlighting continues until line 626. The highlighting changes as I move around in the file. Is that expected?

Moving up screen up won't do it right if the pattern is not fully enclosed (screen wise)

Btw, my multiline pattern now looks like this:
{"py", "((?:[!\"\"\"]*\"{3}\n$)|(?:\"{3}[!\"\"\"]*)|\"{3})", {6}, {0}, -1},

Did you happen to test it? I think its better because it handles this:

fsjfjsdj """ jdfsjfj

"""

jdfsjfj will be highlighted.

I use 6 for color, a special string a separate color I guess ha ha.

-Kyryl

File uploaded: https://0x0.st/HLkh.py

…

On Mon, Aug 21, 2023 at 9:04 AM Kyryl ***@***.***> wrote: I have attached the old Python program which I have been using for testing; Use null pointer, I don't think github reply attachments work (or you forgot to attach it). https://0x0.st/ I added extra strings near the top. The highlighting is quite good, but the result is sometimes bizarre: on line 591, the *fd.write* call uses a triple-quoted string; the string ends on line 616, but the highlighting continues until line 626. The highlighting changes as I move around in the file. Is that expected? Moving up screen up won't do it right if the pattern is not fully enclosed (screen wise) Btw, my multiline pattern now looks like this: {"py", "((?:[!\"\"\"]*\"{3}\n$)|(?:\"{3}[!\"\"\"]*)|\"{3})", {6}, {0}, -1}, Did you happen to test it? I think its better because it handles this: fsjfjsdj """ jdfsjfj """ jdfsjfj will be highlighted. I use 6 for color, a special string a separate color I guess ha ha. -Kyryl — Reply to this email directly, view it on GitHub <#50 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH67Y7Y5VOARTUK6LN2PIDXWNMG7ANCNFSM6AAAAAA3XOV37I> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Claude Marinier