UTF-8 encoded sources (with BOM) support about ofrontplus

[Oleg-N-Cher]

• b0dd134

[Oleg-N-Cher]

From: Josef Templ
Thu, 22 Jun 2017 16:36:17

Hi Oleg,

first of all I have added another fix in OpenReader:

IF pos > 0 THEN ... ELSIF pos < 0 THEN pos := 0 END;

Regarding the BOM, I am not sure if it gives sense in ofront because there is no UNICODE support at all.

In general, I am also not sure that the BOM support is best placed in OpenReader.
I guess (without having implemented it) that it is better placed in Open, where there is also the check for ascii.

• Josef

[Oleg-N-Cher]

Dear Josef,

I fully agree with your proposal to check pos < 0 in OpenReader.

About checking BOM presence in OpenReader - of course, it could better placed in Open. That's what happens, if I do not know completely how these modules do work. I'm most often working with the code in parts.

I even do not use the Texts module in graphical mode (under ETH Oberon-like environment), where the UTF-8 encoding without additional support, could disrupt the overall logic of the view.

On the contrary, in the command-line mode, UTF-8 encoded strings can be entered by a user from the outside, for example, a file name from Args.Get. So the Args and Console modules already have support UTF-8 encoded strings, without any additional modifications. So we can say that we already have basic support of UTF-8, like:

Console.String("Текст на русском языке - Text in Russian language");

will be compiled and runned OK under Linux - if the line is UTF-8 encoded and the current locale is set to use the UTF-8 encoding.

UTF-8 characters are encoded by 1-4 bytes with value > 127 (any byte of a UTF-8 character is out of ASCII table). So Ofront can process such strings with multi-byte characters without any modifications (Ofront allows characters with > 127, and it does not matter that one character will occupy several bytes).

I actively use comments in Russian in my sources. And I do not want to use the encodings like Windows 1251 or KOI8-R (they are non-portable between Windows and Linux). And the encoding UTF-8 looks very tempting to use for comments in a national language. And thus there is an opportunity to edit such sources in Windows Notepad (with BOM), pico, vim or gedit (with BOM too, they even support 0DX+0AX as end-of-line!).

It is for that I propose to support checking the BOM by the code:

• 81c06ec

And I'll happily listen to your comments. I implemented only reading. At the moment, this is enough for me. In future, we could save the BOM flag to correctly overwrite existing files that already have it.

[Oleg-N-Cher]

• 0703e02