Issue building RCall on Windows 8 about rcall.jl

[simonbyrne]

@tkelman Do you have any suggestions for what we could try here?

[tkelman]

Not an R user, so not really. Try looking at the R dll in Dependency Walker to see if there's anything obviously missing. And try dlopen on it manually, see if the error message is any different.

[simonbyrne]

Thanks @tkelman

@skrisna Rather than move the dlll, my suggestion would be to modify deps/deps.jl so that const libR points to the original location of R.dll. If that works, we can modify the build step so that it points to the right file.

[skrisna]

@tkelman @simonbyrne

Based on your suggestions and after some debugging, I figured it would be best if I installed R in the C:\ folder instead of in "Program Files". I also changed the corresponding environment variables. After making the changes, I was able to Pkg.build("RCall") properly. The deps/deps.jl file after being written out looked like this-

# This is an auto-generated file; do not edit

const libR="C:\\R-2.15.3\bin\x64\R.dll"
ENV["R_HOME"]="C:\\R-2.15.3"
ENV["R_DOC_DIR"]="C:\\R-2.15.3\\doc"
ENV["R_INCLUDE_DIR"]="C:\\R-2.15.3\\include"
ENV["R_SHARE_DIR"]="C:\\R-2.15.3\\share"
ENV["LD_LIBRARY_PATH"]="C:\\R-2.15.3"

Notice that libR is not set properly. I changed it to

const libR="C:\\R-2.15.3\\bin\\x64\\R.dll"

I closed the julia session, opened it again and did-

julia> using RCall
Error in inDL(x, as.logical(local), as.logical(now), ...) :
  unable to load shared object 'C:/R-2.15.3/library/stats/libs/x64/stats.dll':
  LoadLibrary failure:  The specified module could not be found.

During startup - Warning message:
package 'stats' in options("defaultPackages") was not found

I tried dlopen'ing it manually-

julia> cd("C:/R-2.15.3/library/stats/libs/x64")

shell> ls
stats.dll

julia> dlopen("stats.dll")
ERROR: could not load module stats.dll: The specified module could not be found.


 in dlopen at c.jl:19

So this is where I am stuck. The stats.dll file exists but I am not able to load it. I tried both the i386 and x64 versions of stats.dll and could not load either. Am I supposed to be able to load stats.dll file with dlopen just like I could for R.dll?

[simonbyrne]

I don't think you should have to load stats.dll yourself, that should be handled by the library.

Searching for stats.dll R, I came across this similar error:
https://rdotnet.codeplex.com/discussions/444463
which suggests adding extra strings to your path. You could try something like that?

[simonbyrne]

What happens if you simply do:

const libR="C:\\R-2.15.3\bin\x64\R.dll"
argv = ["REmbeddedJulia","--silent","--no-save"]
ccall((:Rf_initEmbeddedR,libR),Cint,(Cint,Ptr{Ptr{Uint8}}),length(argv),argv)

?

[skrisna]

Ok. We might have made some progress. I was able to get RCall working under cygwin with some modifications. But I was not able to get it work in the default windows shell, i.e. directly running julia.exe by double clicking. So it appears that the core problem is probably to do with Windows/Unix paths. Here is my setup from scratch to get it to work in cygwin so others can replicate it-

• Installed R-3.2.0 (64-bit) in C:. Deselected the 32-bit option to avoid confusion.
• Added the following environment variables-

R_HOME = "C:\\R-3.2.0"
R_DOC_DIR = "C:\\R-3.2.0\\doc"
R_INCLUDE_DIR = "C:\\R-3.2.0\\include"
R_SHARE_DIR = "C:\\R-3.2.0\\share"
LD_LIBRARY_PATH = "C:\\R-3.2.0"

• Added "C:\R-3.2.0\bin" to PATH environment variable
• Ensured that my cygwin did not have R installed so there are not conflicts
• Edited RCall/deps/build.jl

# Original
const libR = joinpath(ENV["R_HOME"],"lib",string("libR.",BinDeps.shlib_ext))

# Modified
const libR = joinpath(ENV["R_HOME"],"bin","x64",string("R.",BinDeps.shlib_ext))

• Started julia from cygwin

julia> Pkg.build("RCall")
INFO: Building RCall
julia> using RCall
Warning: error initializing module RCall:

ErrorException("error compiling __init__: could not load module C:\R-3.2.indR.dll: The specified module could not be found.")

• Opened RCall/deps/deps.jl and edited it

# Original
const libR="C:\\R-3.2.0\bin\x64\R.dll"

# Modified
const libR="C:\\R-3.2.0\\bin\\x64\\R.dll"

• Restarted julia from within cygwin

julia> using RCall
julia> reval("pdf('aa.pdf')")
NilSxp(16777344,Ptr{Void} @0x00000000029d0788)

julia> reval("plot(1:10)")
NilSxp(16777344,Ptr{Void} @0x00000000029d0788)

julia> reval("dev.off()")
IntSxp(536870989,Ptr{Void} @0x0000000005e43a78,Ptr{Void} @0x0000000012b95538,Ptr{Int32} @0x0000000012b95560,1,0)

julia> isfile("aa.pdf")
true

• Hurray!!

But I was not able to get the same setup running julia outside of cygwin-

• Double click julia.exe

julia> using RCall
Error in inDL(x, as.logical(local), as.logical(now), ...) :
  unable to load shared object 'C:/R-3.2.0/library/stats/libs/x64/stats.dll':
  LoadLibrary failure:  The specified module could not be found.

During startup - Warning message:
package 'stats' in options("defaultPackages") was not found

• Same problem as before

If anyone has suggestions on getting it working in the plain windows terminal, I will be happy to test it out.

[skrisna]

More progress. Looking at the article that @simonbyrne sent [https://rdotnet.codeplex.com/discussions/444463], I made the following changes to the Windows Path variable-

# Original
PATH=<blah>;C:\R-3.2.0\bin

# Modified
PATH=<blah>;C:\R-3.2.0\bin\x64;C:\R-3.2.0\bin

This fixed the issue. I am now able to run in the default windows terminal without issues.

[simonbyrne]

Thats's interesting. It would be nice if we could do this without the user needing to modify their PATH variable.

Does it work if you instead modify the PATH from within julia? say by

ENV["PATH"] *= ";C:\\R-3.2.0\\bin\\x64;C:\\R-3.2.0\\bin"

before using RCall?

[simonbyrne]

You could also try appending to DL_LOAD_PATH?

[skrisna]

Actually, when I removed the earlier added entry, it worked!

# Did not work
PATH=<blah>;C:\R-3.2.0\bin

# Worked
PATH=<blah>;C:\R-3.2.0\bin\x64;C:\R-3.2.0\bin

# Worked!
PATH=<blah>;

So when I do not have R in my path, it worked. No need to add anything to path. I think the R_* environment variables take care of things.

Final suggestions for helping easy installing for Windows-

# Current RCall/deps/build.jl
const libR = joinpath(ENV["R_HOME"],"lib",string("libR.",BinDeps.shlib_ext))

# Needs to be conditionally (for Windows) modified to 
const libR = joinpath(ENV["R_HOME"],"bin","x64",string("R.",BinDeps.shlib_ext))

# Modification to RCall/deps/deps.jl
# Currently written as
const libR="C:\\R-3.2.0\bin\x64\R.dll"

# Needs to be conditionally modified to write
const libR="C:\\R-3.2.0\\bin\\x64\\R.dll"

Thanks.

[simonbyrne]

I've pushed a patch to master, which I think should fix it: would you be able to try it out? If it's all okay, I can then tag a new release.

[skrisna]

Hi Simon,

I started from a fresh install of Julia (renamed the .julia directory) and tried to install RCall keeping the R_* environment variables intact. I am getting the following error-

================================[ ERROR: RCall ]================================

Unable to load C:\\R-3.2.0\lib\libR.dll

Please re-run Pkg.build(package), and restart Julia.
while loading C:\Users\Krishna\.julia\v0.3\RCall\deps\build.jl, in expression st
arting on line 14

================================================================================

================================[ BUILD ERRORS ]================================

WARNING: RCall had build errors.

 - packages with build errors remain installed in C:\Users\Krishna\.julia\v0.3
 - build the package(s) and all dependencies with `Pkg.build("RCall")`
 - build a single package by running its `deps/build.jl` script

================================================================================

I think this has to do with trying to call C:\R-3.2.0\lib\libR.dll. For Windows, the deps.jl needs to be modified to something like-

const libR = joinpath(ENV["R_HOME"],"bin","x64",string("R.",BinDeps.shlib_ext))

for 64 bit installation of R.

Thanks.

[simonbyrne]

@skrisna Sorry I should have said that you need to check out the master branch. Try:

Pkg.checkout("RCall","master")
Pkg.build("RCall")

using RCall
...

Once you're done, you can set it back with Pkg.free("RCall")

[skrisna]

I had to do two things to get it to work (probably should be documented)

1. In addition to earlier environment variables, needed to add R_ARCH and set it to x64
2. Needed to add both the bin and the bin\arch directories to my windows path (without this, I was getting the following error-

Error in inDL(x, as.logical(local), as.logical(now), ...) :
  unable to load shared object 'C:/R-3.2.0/library/stats/libs/x64/stats.dll':
  LoadLibrary failure:  The specified module could not be found.

During startup - Warning message:
package 'stats' in options("defaultPackages") was not found

My current settings for the two environment variables are -

C:\>echo %R_ARCH%
x64

C:\>echo %PATH%
<blah>;<blah>;C:\R-3.2.0\bin;C:\R-3.2.0\bin\x64

The problem has been successfully resolved. Please close the ticket (or should I do it?) when ever you thick fit.

[simonbyrne]

Hmm. It would be nice if we could do without users having to manually change their environmental variables.

Does this:

readall(`Rscript -e 'for (nm in c("R_HOME","R_DOC_DIR","R_INCLUDE_DIR","R_SHARE_DIR","R_ARCH","LD_LIBRARY_PATH")) print(Sys.getenv(nm),quote=FALSE)'`)

work on Windows?

[skrisna]

With my current setup as described in the previous emails, windows reported that it could not find Rscript (both in command line and within julia). It turns out that Rscript.exe is present in both the <R_HOME>\bin and <R_HOME>\bin\x64 locations and the one in <R_HOME>\bin does not work. Since my PATH was setup as ";;C:\R-3.2.0\bin;C:\R-3.2.0\bin\x64", the wrong Rscript.exe was getting picked up. So I had to remove "<R_HOME>\bin" from my path and retain "<R_HOME>\bin\x64".

With this change, I was able to get the command to execute--

julia> readall(`Rscript -e 'for (nm in c("R_HOME","R_DOC_DIR","R_INCLUDE_DIR","R_SHARE_DIR","R_ARCH","LD_LIBRARY_PATH)) print(Sys.getenv(nm),quote=FALSE)'`)
"[1] C:/R-3.2.0\r\n[1] C:\\\\\\\\R-3.2.0\\\\\\\\doc\r\n[1] C:\\\\\\\\R-3.2.0\\\\\\\\include\r\n[1] C:\\\\\\\\R-3.2.0\\\\\\\\share\r\n[1] /x64\r\n[1] C:\\\\\\\\R-3.2.0\r\n"

But this, I figured was probably because I had already setup these environment variables. When I remove dthe R_* environment variables as well as LD_LIBRARY_PATH, I got this from both within R and julia-

julia> readall(`Rscript -e 'for (nm in c("R_HOME","R_DOC_DIR","R_INCLUDE_DIR","R_SHARE_DIR","R_ARCH","LD_LIBRARY_PATH)) print(Sys.getenv(nm),quote=FALSE)'`)
"[1] C:/R-3.2.0\r\n[1] \r\n[1] \r\n[1] \r\n[1] /x64\r\n[1] \r\n"

It it was only able to automatically infer R_HOME and R_ARCH.

[simonbyrne]

I think I've come up with a reasonable solution: R stores its installation path in the Windows Registry, which we can query using REG QUERY. Unfortunately, I came across another bug in doing this (JuliaLang/julia#11170).

I'll have another look at it over the weekend.

[simonbyrne]

@skrisna Having managed to get my hands on a Windows VM, and learning more than I cared about Windows environmental variables, I think I have a fairly reasonable solution that doesn't require the user to modify their PATH. Could you please try

Pkg.checkout("RCall","master")
Pkg.build("RCall")

using RCall

again, and let me know how it goes?

[skrisna]

@simonbyrne Sorry for the delayed response. I finally got a chance to test out your changes. But ran into a problem-

julia> Pkg.checkout("RCall","master")
julia> Pkg.build("RCall")
julia> using RCall
ERROR: `methods` has no method matching methods(::Module)
 in anonymous at no file:135
 in include at boot.jl:245
 in include_from_node1 at loading.jl:128
 in include at boot.jl:245
 in include_from_node1 at loading.jl:128
 in reload_path at loading.jl:152
 in _require at loading.jl:67
 in require at loading.jl:51
while loading C:\Users\ksubrama\.julia\v0.3\RCall\src\types.jl, in expression starting on line 247
while loading C:\Users\ksubrama\.julia\v0.3\RCall\src\RCall.jl, in expression starting on line 71

I have not seen this problem earlier. For this test, I removed all my R_* environment variables as well as removed R from my Windows path. Let me know if I should have some of the environment variables set. I am using a slightly dated version of Julia (0.3.7).

[simonbyrne]

Hmm, I have no idea what is happening there. Any chance you could try 0.3.8? (or does anyone know where I can download 0.3.7?)

[skrisna]

Let me install 0.3.8.

[skrisna]

I am getting the same problem on 0.3.8. Also, when I tried the 0.4.0-dev+4825 (2015-05-14), starting from a completely fresh install, I get--

julia> Pkg.add("RCall")
INFO: Initializing package repository C:\Users\ksubrama\.julia\v0.4
INFO: Cloning METADATA from git://github.com/JuliaLang/METADATA.jl
INFO: Updating cache of ArrayViews...
INFO: Updating cache of BinDeps...
INFO: Updating cache of Compat...
INFO: Updating cache of DataArrays...
INFO: Updating cache of DataFrames...
INFO: Updating cache of Docile...
INFO: Updating cache of GZip...
INFO: Updating cache of SHA...
INFO: Updating cache of SortingAlgorithms...
INFO: Updating cache of StatsBase...
INFO: Updating cache of URIParser...
INFO: Installing ArrayViews v0.6.2
INFO: Installing BinDeps v0.3.12
INFO: Installing Compat v0.4.4
INFO: Installing DataArrays v0.2.14
INFO: Installing DataFrames v0.6.5
INFO: Installing Docile v0.5.0
INFO: Installing GZip v0.2.15
INFO: Installing RCall v0.2.0
INFO: Installing Reexport v0.0.2
INFO: Installing SHA v0.0.4
INFO: Installing SortingAlgorithms v0.0.5
INFO: Installing StatsBase v0.6.15
INFO: Installing URIParser v0.0.5
INFO: Building RCall
================================[ ERROR: RCall ]================================


LoadError: could not spawn `Rscript -e 'for (nm in c("R_HOME","R_DOC_DIR","R_INC
LUDE_DIR","R_SHARE_DIR","LD_LIBRARY_PATH")) print(Sys.getenv(nm),quote=FALSE)'`: no such file or directory (ENOENT)
while loading C:\Users\ksubrama\.julia\v0.4\RCall\deps\build.jl, in expression starting on line 3

================================================================================


================================[ BUILD ERRORS ================================


WARNING: RCall had build errors.

 - packages with build errors remain installed in C:\Users\ksubrama\.julia\v0.4
 - build the package(s) and all dependencies with `Pkg.build("RCall")`
 - build a single package by running its `deps/build.jl` script

================================================================================

INFO: Package database updated

julia> Pkg.checkout("RCall","master")
INFO: Checking out RCall master...
INFO: Pulling RCall latest master...
INFO: No packages to install, update or remove

julia> Pkg.build("RCall")
INFO: Building RCall

julia> using RCall

julia>

So it works without issues in 0.4.0-dev but not in 0.3.8. My setup is clean as far as environment variables and path go (registry ofcourse has R entries).

[skrisna]

I looked at the RCall\src\types.jl file @ line 247 and see this-

if VERSION < v"v0.4-"
    @doc """
    Extract the original SEXP (pointer to an R SEXPREC)

    Written as a `convert` method for convenience in `ccall`
    """->
    Base.convert(::Type{Ptr{Void}},s::SEXPREC) = s.p
else
    @doc """
    Extract the original SEXP (pointer to an R SEXPREC)

    Written as a `unsafe_convert` method for convenience in `ccall`
    """->
    Base.unsafe_convert(::Type{Ptr{Void}},s::SEXPREC) = s.p
end

Not sure if that gives you some ideas.

Thanks.

[simonbyrne]

This appears to be a problem with Docile.jl. If you Pkg.pin("Docile",v"0.4.11"), that should temporarily fix it. I'll file an issue.

[simonbyrne]

Thanks again for all your patience with this, it's really great to have this working.

[simonbyrne]

Okay, I've tagged a new release. You can get back to releases from the master branch via Pkg.free("RCall")

[skrisna]

@simonbyrne I tested RCall on 0.3.8 after pinning Docile and it worked. How do I unpin Docile?

No problem. I learnt a lot about julia during this time.

[simonbyrne]

Same: Pkg.free("Docile").

[skrisna]

@simonbyrne This is what I am getting-

julia> Pkg.pin("Docile",v"0.4.11")
INFO: Creating Docile branch pinned.15b90bae.tmp
INFO: No packages to install, update or remove

julia> using RCall

julia> Pkg.free("Docile")
INFO: Initializing package repository C:\Users\ksubrama\Documents\.julia\v0.3
INFO: Cloning METADATA from git://github.com/JuliaLang/METADATA.jl
ERROR: Docile is not a git repo
 in free at pkg/entry.jl:198
 in anonymous at pkg/dir.jl:28
 in cd at file.jl:30
 in cd at pkg/dir.jl:28
 in free at pkg.jl:36

Also, this-

julia> Pkg.free("RCall")
ERROR: RCall is not a git repo
 in free at pkg/entry.jl:198
 in anonymous at pkg/dir.jl:28
 in cd at file.jl:30
 in cd at pkg/dir.jl:28
 in free at pkg.jl:36

[skrisna]

@simonbyrne Please ignore my last comment. I am all set.

[simonbyrne]

Great. Docile has now been updated, so a Pkg.update() should have this working now.