Installation failed; can't access URL on isolated system about x13binary

[tedmoorman]

The configure file has the following code:

## On Linux, download binary
if [ ${platform} = "linux" ]; then
    cd inst/bin
    download https://github.com/x13org/x13prebuilt/raw/master/v1.1.57/linux/${flavour}/x13ashtml
    chmod 0775 x13ashtml
    echo "*** downloaded Linux binary"
    cd ${cwd}
elif [ ${platform} = "mac" ]; then
    cd inst
    download https://github.com/x13org/x13prebuilt/raw/master/v1.1.57/mac/${flavour}/x13ashtml.tar.gz
    tar -zxvf x13ashtml.tar.gz
    cp -r x13ashtml/bin .
    cp -r x13ashtml/lib .
    rm -r x13ashtml
    rm x13ashtml.tar.gz
    cd ${cwd}
fi

I just replaced with the following:

## On Mac, download binary
if [ ${platform} = "mac" ]; then
    cd inst
    download https://github.com/x13org/x13prebuilt/raw/master/v1.1.57/mac/${flavour}/x13ashtml.tar.gz
    tar -zxvf x13ashtml.tar.gz
    cp -r x13ashtml/bin .
    cp -r x13ashtml/lib .
    rm -r x13ashtml
    rm x13ashtml.tar.gz
    cd ${cwd}
fi

The installation worked perfectly!
Thank you so much, Dirk!

[christophsax]

Nice!

seasonal:::get_x13_path() looks for Sys.getenv("X13_PATH"), so X13_PATH may the obvious name.

Alternatively, and this was your first idea, a true/flase variable X13BINARY_NO_DOWNLOAD_FROM_GITHUB (or similar) may be even more transparent and would make it obivious what you try to achive. If you turn this on, you should be aware of the consequences.

[eddelbuettel]

Took a moment to find some time, sorry -- somewhat crazy week.

I added a simple test that if X13_PATH is found and has executable, we believe it and move. Demo:

edd@rob:~/git/x13binary(master)$ install.r x13binary_1.1.57-2.1.tar.gz 
* installing *source* package ‘x13binary’ ...
** using staged installation
*** downloaded Linux binary
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (x13binary)
edd@rob:~/git/x13binary(master)$ cp -vax /usr/local/lib/R/site-library/x13binary/bin/x13ashtml /tmp
'/usr/local/lib/R/site-library/x13binary/bin/x13ashtml' -> '/tmp/x13ashtml'
edd@rob:~/git/x13binary(master)$ X13_PATH=/tmp/x13ashtml install.r x13binary_1.1.57-2.1.tar.gz 
* installing *source* package ‘x13binary’ ...
** using staged installation
*** existing x13 binary found, not downloading
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (x13binary)
edd@rob:~/git/x13binary(master)$ 

Note how the first installation shows the usual *** downloaded Linux binary whereas the second one (where X13_PATH is set to a (temp, I don't need it) copy) shows *** existing x13 binary found, not downloading.

I am going to put this into a fresh branch now. When you have a moment, can you play and see if it suits?

[eddelbuettel]

New change made, tested both ways, and commited and pushed.

[eddelbuettel]

New versions of x13binary is now up on CRAN:

[christophsax]

The client could now get it from CRAN and use it successfully on an isolated system. Thanks a lot!

[eddelbuettel]

In the "everything is possible" we (theoretically) could but (our volunteer) time is also finite, and we need to trade off costs vs benefits -- as well as opportunity costs of time we could have spent on more widely used packages / more applicable solutions. The package clearly states it is accessing a remote resource for local use -- if you cannot access a remote resource it is likely not for you.

More generally, this seems like something more specific to your setup so maybe I can suggest to just comment out the failing download step (or maybe remove all of configure) to let the installation proceed and then "somehow, and only you know how" get the required external binary onto the system and into the right place on that airgapped machine.

[christophsax]

I have a client with a very similar problem (so I am adding it here, but we can also open a new issue):

• They use Linux (it wouldn't be a problem on Win or Mac, because they get the binaries)
• They can neither access nor whitelist https://github.com/x13org/x13prebuilt
• But they can download the binaries manually and put them somewhere.

In principle, seasonal allows you to set a system variable X13_PATH. If set, seasonal will use the path in the system variable instead of system.file("bin", package="x13binary") (returned by x13binary::x13path()).

The problem is that x13binary throws an error on installation, which propagates to seasonal.

I see two potential 'solutions':

1. seasonal could suggest x13binary and ask users to install x13binary. This would make the life of 99.9% of seasonal users more complicated. So, not really a solution.

2. x13binary tries to download from GitHub. If it fails, the installation of x13binary still succeeds but x13binary::x13path() subsequently points to "". seasonal then asks users to set X13_PATH on start-up (the current version already does that), and everyone is happy.

What do you think? I could do a PR for 2 (probably still requires a technical discussion on how to 'try' a command on bash). Perhaps we limit the permissive download to Linux and still insist on a successful download for Windows and Mac? That way, we can ensure the CRAN binaries for these systems always ship with X-13 binaries.

[eddelbuettel]

I have to think about it. I don't think 2) without additional layers is "The Right Thing" to do because tapering over install failures is not really the correct way to go about it. Could we, and that is just a first suggestion, maybe do something like 2. if and when a user has opted into that behaviour via, say, another environment variable?

Otherwise maybe we can be smarter and avoid the failure. For example if X13_PATH is set and a binary is at that spot (and maybe we even verify it is the right one?) then the download is skipped?

Also otherwise, and something I did think about in another context, is that we do have sources so we could include them in the tarball and then decide policy -- say if download is possible do that but if not, or if it fails, or ... then compile. (We could also always compile but I find that wasteful.)

[christophsax]

I like both otherwises. The first one seems easy to do and would solve the problem of the OP and the client.

The second one sounds interesting, too. After all, we had compilation problems on Win and Mac but not so much on Linux. Still, probably quite a bit more complicated than the first otherwise.

[eddelbuettel]

I will try to find some time to generalize the configure script to use a local file pointed to by X13_BINARY (or alike) and use that, avoiding a download. Air-gapped systems can then prepare a binary and point X13_BINARY at it, and the rest continues as before. Should be applicable for Linux and macOS. It's a "buyer beware" thing: if you opt in, but point to a bad or non-working binary, you get the pieces. For others it should continue to work.

[christophsax]

Thanks a lot! That looks great to me. Should I be able to run it somehow? I don't find a branch with the code.

[eddelbuettel]

Yes we should probably test it / have one of your contacts or clients test it.

My bad. Had committed here but not pushed a new branch. It is there now as feature/skip_download_if_x13_path and I set the version to 1.1.57-2.1 as above (which I had changed locally but not committed) so it should really be as the above. Have done any use of x13 with it; but it will cleanly/quietly proceed with the installation now.

[christophsax]

It works as promised, and I can skip the download both on Mac and Linux. Nice!

There is now an inconsistency in the definition of X13_PATH between here and seasonal. seasonal expects it to point to the folder where x13ashtml is located, while your code assumes it is the file itself. That way, I have to set a different X13_PATH during installation and usage. The seasonal definition is also consistent with what x13binary::x13path() returns - the folder, not the file.

It can be adjusted by changing:

x13binary/configure

Line 85 in 960f55c

if test -x ${X13_PATH}; then

into:

    if test -x ${X13_PATH}x13ashtml; then

Does that make sense?

[eddelbuettel]

Or test -d ${X13_PATH} to test for the outer directory and/or test -s ${X13_PATH}/x13ashtml.

That's the type of feedback I needed here I guess. I would like to suggest though that you, if possible, "eventually" migrate to a scheme where the environment variable does point to binary file as the combination of a "given directory plus an unspoken / unknown / maybe changing" suffix for the actual executable seems ... a little loose, no?

[christophsax]

"eventually" migrate to a scheme where the environment variable does point to binary file

"eventually" yes: There is a logic in seasonal that depends on whether the binaries are called x13ashtml or x13as. It determines whether seasonal can use the HTML goodies.

So allowing for different or arbitrary filenames would require another mechanism to find out if we have the HTML version. It would be easier if x13as were dropped altogether, which may happen at some point. Until then, perhaps easier to leave it as it is.

[eddelbuettel]

Just thought about that over dinner. Maybe introducing a new env var X13_BINARY may help. You can then test if the value ends in x13as or x13ashtml; either way would be finer control than just 'trusting' there be a binary in a pointed-at directory. Or at least that is what it seems like to me. But you are closer to the rubber and the issue so whatever works for you...