Add encoding="UTF-8" argument to source() function about radiant

[vnijs]

@suimong I'm certainly open to that. Are you talking about source statements such as in server.R? Or in data/csv files? Perhaps you could give a concrete example where this causes problems?

FYI I have in the past enforced the locale to avoid some problems. Would this help in your situation?

if (.Platform$OS.type == 'windows') {
  Sys.setlocale(category = 'LC_ALL','English_United States.1252')
} else {
  Sys.setlocale(category = 'LC_ALL','en_US.UTF-8')
}

[suimong]

@vnijs I'm talking about source() statements in server.R etc.
For example, I'd like to add extra features or additional UI elements to analysis tools in the quant app. I'd like the UI texts to be Chinese and I'm running it on Windows. The settings that works for me is:

1. set system locale: Sys.setlocale(category = 'LC_ALL', 'Chinese') in .Rprofile
2. source code file like this: source("init.R", encoding = "UTF-8", local = TRUE)

In my experience, the above two steps are necessary to ensure the file to be correctly sourced. I barely used R on Mac but as far as I know, R users on Mac face far less encoding problems than those on Windows.

[vnijs]

@suimong I wouldn't use settings in .Rprofile. At least not if someone else should also be able to use the code. I put some new code at the top of global.R where if you (un)comment a few lines you should have what you need (see also below). For example, options will ensure all source commands use UTF-8 encoding. Let me know if this work for you

## encoding
options(encoding = "UTF-8")      ## for chines
## use getOption("encoding") to see if things were changed

loc <- function(os, language = "english") {
  switch(language,
         english = ifelse(os == "Windows", "English_United States.1252", "en_US.UTF-8"),
         chinese = ifelse(os == "Windows", "Chinese", "zh_CN.utf-8"))
}

## set locale
Sys.setlocale(category = "LC_ALL", loc(Sys.info()[["sysname"]], "chinese"))

[suimong]

@vnijs You are right about not using .Rprofile. It's just my old habit doing baby projects locally. I used to set options(encoding = "UTF-8") to deal with mutibyte characters, but shiny has this weird bug that when the above option is set, it throws an error for files that contain multibyte characters like this:

Error in checkEncoding(file) : 
  The file "D:\Documents\My Projects\R - Radiant Dashboard\inst\quant/ui.R" is not encoded in UTF-8. Please convert its encoding to UTF-8 (e.g. use the menu `File -> Save with Encoding` in RStudio). See http://shiny.rstudio.com/articles/unicode.html for more info.

Here I added a few Chinese characters to ui.R, which is indeed UTF-8 encoded. I discussed this with Yihui Xie @rstudio, and he discouraged me to set this option. He wrote an article in Shiny's official website that discussed this matter.

So I think perhaps setting encoding explicitly in source() and other I/O functions (such as read.table()) would be a better solution?

[shrektan]

@suimong I will agree. I think it's a good practice to set encoding for all I/O functions explicitly and do not touch the option(encoding). It's because it seems like many R function only allows native.enc encoding or assume the native.enc encoding by default.

I've tried to option(encoding = "UTF-8") in my windows computer in the past, and all I got is pain...

[suimong]

@shrektan :) and it won't affect anything on Linux or Mac anyway.

[vnijs]

@suimong @shrektan I can add a encoding = ... to each source(...). However, how should the user set the encoding value? One way might be a add a parameter to the win_launcher function (e.g., encoding =...). If the user creates a launcher with an encoding option that would be used in the source() calls. For example, options("r_encoding = ...") and then Radiant would use that value as appropriate. A similar approach might be used to set alternative paths to look for help-files, ui.R components etc. How does that sound?

https://github.com/vnijs/radiant/blob/master/R/radiant.R#L297-L346

[vnijs]

@suimong @shrektan Somewhat related question ... do you currently have any trouble running Radiant (as-is) when the locale is not "English_United States.1252"?

[suimong]

@vnijs there's no problem running radiant when the locale is "Chinese (Simplified)_China.936".
Your suggestion sounds good. However IMHO it is good practice to save R source file in UTF-8 and I would boldly guess that most R users who have to deal with multibyte characters will have done so already. Therefore, would you consider to force Windows radiant users to save source files in UTF-8? Explicitly setting encoding = "UTF-8" won't affect anything on Linux or Mac and it will enhance radiant's cross-platform capability. For instance it will allow users to develop in Windows but later deploy it in a Linux web server.
On the other hand, I think it might be necessary to add an encoding option in the "Load Data" section in UI. Since data file comes from all kinds of places, there is a much bigger chance it's not encoded in UTF-8 or even the system's locale.

[vnijs]

@suimong I added "UTF-8" encoding to all source calls. If you can install from github using devtools please try it out.

Can you post a small file (rda and csv) with multibyte characters so i can see what would need to change in the data load (and save?) functions?

[suimong]

@vnijs Thank you for making the changes.
I created a sample csv file that contains some chinese characters and latin characters with accent(such as é). The zip file in the link comprises of both ANSI & UTF-8 encoded version. Here is some test results on my Windows 8.1 machine:

1. When the locale of R is set to "English_United States.1252" & load the ANSI encoded csv:
   
2. When the locale of R is set to "English_United States.1252" & load the UTF-8 encoded csv:
   
3. When the locale of R is set to "Chinese (Simplified)_China.936" & load the ANSI encoded csv:
   
4. When the locale of R is set to "Chinese (Simplified)_China.936" & load the UTF-8 encoded csv:
   

As you can see, scenario no.3 is the only correct one.
P.S. Sorry for accidentally closing this issue.

[vnijs]

Thanks for the data and examples @suimong. Everything loads fine on Mac and if the user has the locale set correctly and the data is encoded in ansi everything works as you want (tested on Windows 8). I tried saving a csv from Radiant on Windows and loading it back in and that seemed to work fine as well. Close the issue or is there still something missing?

[suimong]

@vnijs Thank you for testing and I'm sorry that all these windows-specific problems took up your time when you could develop new features.
So currently, on a windows machine, one has to convert all data files(including those UTF-8 files) into ansi encoding before loading them into radiant. Personally I still prefer to convert any text file that contains multibyte characters into UTF-8, but I guess this is what I'll have to do now. Perhaps you'd like to add some remarks in the documentation so that windows users wouldn't get confused?
Anyway this is not related to the original issue so I'll close it now. Thank you again for fixing the source() issue.

[vnijs]

@suimong The need for ANSI is an R requirement correct? FYI I use hadley/readr in Radiant to load csv files.

I don't have Excel on Windows. I assume there is an option to save xls files to csv ANSI in Excel? When I try opening either of the csv files into Excel on mac I get a warning that "... a SYLK file ..."

Could you try copy-and-pasting from Excel to Radiant (Data > Load data > clipboard) when the xls file has multi-byte characters? I'm curious if that works.

[vnijs]

@suimong FYI I added a note about ANSI formatting on the http://vnijs.github.io/radiant/base/manage.html page

[suimong]

@vnijs I just realised your switch to hadley/readr since recent commits.
Actually this is a good point. Saving xls file to csv in Excel will result in an ANSI csv file. I encountered the same warning on Windows, and I think this might be the reason. In my case if you ignored the warning then the ansi file opened up in excel just fine.
The copy and pasting from Excel works fine.

[vnijs]

That is such a strange error in Excel ... 'just add and apostrophe' ... really :) Weird. OK @suimong, I'll close this issue for now but let me know if you hit anymore roadblocks.

Just to be sure ... Radiant has an open source license which, more or less, ensures that changes to the code should also be open-sourced. Are you planning to put your changes on Github?

[suimong]

I guess it's some kind of legacy compatibility issue that Microsoft doesn't bother to take care of anymore:)
@vnijs yes I will put my project on github shortly. I intend to wrap radiant components in a shinydashboard framework for internal use at workplace. I hope it's not a violation of AGPL license.

[vnijs]

If you put the project on github that all sounds fine @suimong.