Non ASCII characters in string constants are distorted about saltarellecompiler

[mattleibow]

It does the same on my en-ZA system. It comes out as 'Erik K�ll�n'

[erik-kallen]

Can you try with the version currently in the develop branch? I can't reproduce the issue myself.

[dested]

en-US outputs 'Erik Källén'; properly (without updating to the latest branch)

[ashrafsabrym]

In my windows regional settings, I changed my "standards and formats" to en-US, and the "Language for non unicode programs" setting to en-US too, and the compiler now works fine (the current version, not the development branch). It correctly transforms both Arabic texts and the accented letters in your name "Erik Källén".

I can't figure the reason of this. If a part of the compiler uses code pages instead of unicode, why does en-US fixes the problem of both Arabic and accented characters although they are from different code pages?

I can't keep my PC running in en-US, as some programs will misbehave and may distort my data, so I'll revert back to Arabic and may try the development branch later.

Thanks for your replies.

[ashrafsabrym]

I made a mistake in my previous post.
When I restarted my machine in en-US, I found the JavaScript file, generated from a previous compilation, displayed properly. Meaning that just changing the locale and language made the file display properly without a new compilation.

More weirdly, now after I reverted back to my previous language settings, everything works properly and the problem is resolved!

I don't know what to say, but it seems to be a bug in Visual Studio or in Windows itself.

[drysart]

What editor are you using to view the generated Javascript files? It looks like the compiler driver (always?) outputs UTF-8, and "Erik Kأ¤llأ©n" looks an awful lot like an editor opening a file that contains UTF-8 but instead parsing it as some SBCS encoding (probably auto-detected, with your Windows locale providing a starting point/fallback default if the editor can't suss out the encoding on its own). If your editor has an option to load the file using an explicit character encoding, try that.

For instance, in Visual Studio when you open a .js file by default it'll try to auto-detect the encoding. You can specify one explicitly by using the Open dialog, clicking the dropdown arrow on the Open button, choosing "Open With...", then select "Script Editor with Encoding". When you click OK, a separate dialog will appear to specify which encoding to use.

The important thing is to change the encoding while opening the file. Changing it after it's already been opened won't help, because the mangled characters themselves that were parsed out in the wrong encoding will just be translated to the new encoding; it doesn't actually return to the original bytes and reparse. That only happens during initial load.

When using the script on a webpage, force the content-type of the script as loaded by the browser to be UTF-8 to eliminate the problem there too, by adding charset="UTF-8" on the <script> tag.

[drysart]

This seems like it might be a common enough problem that it might be a better idea to just have the compiler always output \u000000-style Unicode escape sequences into the Javascript when dealing with strings that contain any non-ASCII characters. Most developers don't handle encoding issues correctly; and the web browser scenario will almost never get it right out of the box, almost always falling back on auto-detection, because servers don't/can't send charset data with a response containing a JS file because the server also doesn't know the actual charset of what's basically a plain text file.

[ashrafsabrym]

I'm currently totally unable to reproduce this issue.
I try creating a blank project, add the nuget packages, and write some non ASCII text, but it's translated correctly to JavaScript.
I can't figure why it happened the first time I tried Saltarelle.
I was using Visual Web Developer Express 2010 when the problem occurred, and I think it always defaults to UTF-8. I've never got such problems before at least with my native language.

Your last suggestion is good, but I suggest that it's kept optional (a compiler option for example), because it's not always desirable to use the \u0000 notation, specially while debugging.

[erik-kallen]

Always generating unicode escape sequences for non-ASCII characters will make scripts in languages other than enqlish close to unreadable. Saltarelle assumes both input and output files to be UTF-8.