CRAN release about x13binary

[jeroen]

We could also do it in install.libs.R like so: #77 although maybe Dirks' solution is cleaner.

[eddelbuettel]

Winning so far thanks to @jeroen and #82 :

Dear maintainer,

package x13binary_1.1.60.tar.gz has been auto-processed and is pending an automated reverse dependency check. This service will typically respond to you within the next day. For technical reasons you may receive
+a second copy of this message when a team member triggers a new check.

Log dir: <https://win-builder.r-project.org/incoming_pretest/x13binary_1.1.60_20240122_155725/>
The files will be removed after roughly 7 days.
Installation time in seconds: 139
Check time in seconds: 52
R Under development (unstable) (2024-01-20 r85814 ucrt)

Pretests results:
Windows: <https://win-builder.r-project.org/incoming_pretest/x13binary_1.1.60_20240122_155725/Windows/00check.log>
Status: OK
Debian: <https://win-builder.r-project.org/incoming_pretest/x13binary_1.1.60_20240122_155725/Debian/00check.log>
Status: OK

Last released version's CRAN status: OK: 8, NOTE: 4
See: <https://CRAN.R-project.org/web/checks/check_results_x13binary.html>

CRAN Web: <https://cran.r-project.org/package=x13binary>

*** Strong rev. depends ***: gunsales seasonal x12

Best regards,
CRAN teams' auto-check service

[eddelbuettel]

Also: Jeroen made some very recent improvements in mac builds and now has separate arm64 and x86_64 binaries. We could commit something simple to the main branch to trigger a new build. Maybe just increase the version to 1.1.59.2 as we will release as 1.1.60 anyway?

[Antonov548]

I can try the helloworld idea during the weekend. Will see if there are other ideas until then.

I was trying hello world test before, when this issues appeared. Meantime I will try to find results.

[jeroen]

You don't have to set NeedsCompilation yourself; R automatically injects NeedsCompilation: yes in the DESCRIPTION when you build the source package. But because you don't have a source dir, it is setting NeedsCompilation: no. I think this may affect how the macbuilder treats it.

[jeroen]

Good catch. I tweaked the r-universe build setup a bit such that the libs at link time are in the standard location. I think this has fixed the problem. Actually no this hasn't fixed it.

When we build shared libraries in R packages, R takes care of relocating libfortran. But for a standalone executable it is probably different.

The R admin manual suggests you would need something like this:

install_name_tool -change
/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/12.2.0/libgfortran.5.dylib \
/Library/Frameworks/R.framework/Resources/lib/libgfortran.5.dylib \
x13ashtml

install_name_tool -change
/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/12.2.0/libquadmath.0.dylib \
/Library/Frameworks/R.framework/Resources/lib/libquadmath.0.dylib \
x13ashtml

[eddelbuettel]

@christophsax Can you try running the build steps locally and then run the install_name_tool invocation, and/or try the install_name_tool invocation on the binary you get from r-universe?

[christophsax]

If I run @jeroen's lines on x13ashtml, it works perfectly:

install_name_tool -change /opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/12.2.0/libgfortran.5.dylib \
/Library/Frameworks/R.framework/Resources/lib/libgfortran.5.dylib \
x13ashtml

install_name_tool -change /opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/12.2.0/libquadmath.0.dylib \
/Library/Frameworks/R.framework/Resources/lib/libquadmath.0.dylib \
x13ashtml

x13binary::checkX13binary()
#> x13binary is working properly

[eddelbuettel]

Either way works but the above could do with a generalization of finding the two dylib files.

[christophsax]

I installed the r-universe binaries on my M2 and on an older Intel Mac. Both work as advertised! 🎉

[eddelbuettel]

Department of TaaDaa calling us on line one:

Dear maintainer,

thanks, package x13binary_1.1.60.tar.gz is on its way to CRAN.

Best regards,
CRAN teams' auto-check service

[christophsax]

According to @Antonov548, mac builder does not work for some mysterious reason. It works, however, on GHA mac and on all (several) macs I tried so far, so I would suggest risking it and trying it on CRAN?

@eddelbuettel Please let me know if I can help with any of the tasks above.

[eddelbuettel]

There are two mac architecture, and then of course different releases. That said, mac-builder is basically what CRAN uses for the newer arm64 so we need to build there.

r-universe now builds for both architectures. We could try another simple and see what happens.

[eddelbuettel]

What is the supposed issue at macbuilder? Would it be possible to have everybody working on this actually chiming into the same issue? @Antonov548 can you share what worked or didn't?

For what it is worth, on the master branch as it is right now I built the source tar.gz the usual way, tested it with my r-devel built here (no issues) and submitted it to 'mac-builder' and all is good on arm64:

• build log https://mac.r-project.org/macbuilder/results/1705328044-588bfa27645413be/
• 00install.out https://mac.r-project.org/macbuilder/results/1705328044-588bfa27645413be/
• check log https://mac.r-project.org/macbuilder/results/1705328044-588bfa27645413be/
• artifacts https://mac.r-project.org/macbuilder/results/1705328044-588bfa27645413be/results.tar.bz2

Given that we also have ✔️ on the CI platforms, it seems we are ready. Am I missing something here?

(Looks like Simon's webpage tricked me. I reported three times the same URL for the three tabs therein. Doh. Filed under 'today I learned ...'.)

[christophsax]

Michael's older note, regarding issue on macbuilder (perhaps this is no longer the case?):

it's really strange, but executable which is created in macbuilder is not for mac actually.
I think it's related to how macbuilder builds the packages. I think it's not really mac host machines.

The executable is for Linux. I tried to run executable from macbuilder artifacts on linux and it works.

[eddelbuettel]

Can you please re-upload to macbuilder (if needed) or use the links I posted yesterday and validate that this is true? It strikes me as doubtful as I happen to know that the macbuilder is in fact an M1 box. Similarly Jeroen's work does not flake so we should know.

Shall we add (at least in CI) a test that invokes the new binary and, say, runs at least --version or equivalent?

[Antonov548]

What is the supposed issue at macbuilder? Would it be possible to have everybody working on this actually chiming into the same issue? @Antonov548 can you share what worked or didn't?

For what it is worth, on the master branch as it is right now I built the source tar.gz the usual way, tested it with my r-devel built here (no issues) and submitted it to 'mac-builder' and all is good on arm64:

• build log https://mac.r-project.org/macbuilder/results/1705328044-588bfa27645413be/
• 00install.out https://mac.r-project.org/macbuilder/results/1705328044-588bfa27645413be/
• check log https://mac.r-project.org/macbuilder/results/1705328044-588bfa27645413be/
• artifacts https://mac.r-project.org/macbuilder/results/1705328044-588bfa27645413be/results.tar.bz2

Given that we also have ✔️ on the CI platforms, it seems we are ready. Am I missing something here?

(Looks like Simon's webpage tricked me. I reported three times the same URL for the three tabs therein. Doh. Filed under 'today I learned ...'.)

I'm not very sure, but as far as I remember mac builder using aarch64 M1. And seems binary is not compatible with this platform. In the test there is check that platform is not aarch64: https://github.com/x13org/x13binary/blob/master/tests/simpleTest.R#L1

So, yes, it's compiling, but I think it doesn't work actually.

[eddelbuettel]

Hm.

One the one hand that is what the tests are supposed to uncover.

On the other that is a bit of puzzle as we are invoking fairly standard compilations with standard tools. So maybe this needs escalation to macOS specialists? Can you three take charge there as I understand that at least some of you have access to a M1 machine and possibly a subscription to the r-sig-mac mailing to discuss why/how Fortran would produce the wrong binary? Maybe we only lack a proper flag on that platform.

[eddelbuettel]

So this made no change:

1.59.2 just like 1.59.1 just has 'any' for apple. Witness for example another package (by me, from my universe)

It has proper arm and x86 binaries. So maybe this happens to us because we have no src/ and maybe we should add a simple helloworld function that requires compilation? It's just a guess on my part. Any other ideas?

[christophsax]

I can try the helloworld idea during the weekend. Will see if there are other ideas until then.

[eddelbuettel]

I can also reach out to Jeroen. Maybe he has a moment. He spent a lot of time recently looking at build and port issues and may know what's up.

[eddelbuettel]

Reached out to @jeroen who may be taking a look. His immediate response points to one oversight: We left are getting NeedsCompilation: no in DESCRIPTION. Whoops.

[jeroen]

R is not very smart, it simply checks for the presence of a src dir if the package need compilation, and hence, if it is specific to the architecture: https://github.com/r-devel/r-svn/blob/8a3fc6542f6c24d2f4ea8d440653e58c032b182e/src/library/tools/R/build.R#L168-L173

And agian when building the binary package, it checks if there is a src dir, to identify if the package is portable or not: https://github.com/r-devel/r-svn/blob/8a3fc6542f6c24d2f4ea8d440653e58c032b182e/src/library/tools/R/admin.R#L73

So the easiest solution is simply creating a src dir with some dummy text file.

[eddelbuettel]

Some more offline chatting with @jeroen: The way we build now ignores R's own Makeconf so we likely do not do the right things on ports and crossbuilds. Easiest fix may be bring the build of the binary into src/Makevars as a build target which also copies into the desired place. That way we get right compiler invocations and options. I can take a stab, time permitting.

[jeroen]

There is also another bug that your makefile is generating an executable called x13as_html (with underscore) but your R code checkX13binary is looking for x13ashtml.

[eddelbuettel]

Two more additions, maybe:

• Add Jeroen to AUTHORS
• Add a test that the binary loads, maybe via capture.output checking the expected text (or parts of it) is there

[eddelbuettel]

Looking much better now thanks to all the work done by everybody and especially @jeroen pushing us over the (macOS-coloured) finish line:

[christophsax]

Amazing work, thank you so much @jeroen!

I still have a problem on my Mac: When I install from r-universe:

install.packages('x13binary', repos = c('https://x13org.r-universe.dev', 'https://cloud.r-project.org'))

It does not find some libraries:

x13binary::checkX13binary()

dyld[99255]: Library not loaded: /opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/12.2.0/libgfortran.5.dylib
  Referenced from: <4BD7F637-67CD-3F1E-ACFC-5779CF824785> /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/x13binary/bin/x13ashtml
  Reason: tried: '/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/12.2.0/libgfortran.5.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/12.2.0/libgfortran.5.dylib' (no such file), '/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/12.2.0/libgfortran.5.dylib' (no such file), '/usr/local/lib/libgfortran.5.dylib' (no such file), '/usr/lib/libgfortran.5.dylib' (no such file, not in dyld cache)

libgfortran.5.dylib can be found on my machine but it is in:

/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib

Any idea? I will play around with this later on.

[eddelbuettel]

Now we have to find a clever to 'find' / 'compute' the respective first part (in /opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/12.2.0) to set the second.

[christophsax]

There is otool, perhaps something like this?

otool -L x13ashtml
x13ashtml:
	/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/12.2.0/libgfortran.5.dylib (compatibility version 6.0.0, current version 6.0.0)
	/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/12.2.0/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

[eddelbuettel]

Nice one! My mental model was on find. We can run otool into grep to get the line and then use it as the first arg.

[christophsax]

I guess something like this could work?

libgfortran_path=$(otool -L x13ashtml | grep libgfortran.5.dylib | awk '{print $1}')
libquadmath_path=$(otool -L x13ashtml | grep libquadmath.0.dylib | awk '{print $1}')

install_name_tool -change "$libgfortran_path" \
/Library/Frameworks/R.framework/Resources/lib/libgfortran.5.dylib \
x13ashtml

install_name_tool -change "$libquadmath_path" \
/Library/Frameworks/R.framework/Resources/lib/libquadmath.0.dylib \
x13ashtml

In which file should we put it? And I guess we also need to make sure this only happens on macs.

[eddelbuettel]

Very good. Wait one sec and I'll have something related, using a 'conditional' I use elsewhere.

Please replace src/Makefile with this:

all:
	FC="$(FC)" LINKER="$(DYLIB_LD)" LDFLAGS="$(FLIBS)" $(MAKE) --directory=../tools/x13as_html -f makefile.gf x13ashtml
	if [ `uname -s` = 'Darwin' ]; then
		install_name_tool -change `otool -L x13ashtml | awk '/libgfortran.5.dylib/ '{print $1}'` /Library/Frameworks/R.framework/Resources/lib/libgfortran.5.dylib ../tools/x13as_html/x13ashtml
		install_name_tool -change `otool -L x13ashtml | awk '/libquadmath.0.dylib/ '{print $1}'` /Library/Frameworks/R.framework/Resources/lib/libquadmath.0.dylib ../tools/x13as_html/x13ashtml
	fi
	mkdir -p ../inst/bin
	cp -f ../tools/x13as_html/x13ashtml ../inst/bin/

clean:
	rm -f ../tools/x13as_html/x13ashtml

which adds the two new (and long, no indentation) lines wrapped by an 'if' for macOS so in diff notation:

modified   src/Makefile
@@ -1,5 +1,9 @@
 all:
 	FC="$(FC)" LINKER="$(DYLIB_LD)" LDFLAGS="$(FLIBS)" $(MAKE) --directory=../tools/x13as_html -f makefile.gf x13ashtml
+	if [ `uname -s` = 'Darwin' ]; then
+		install_name_tool -change `otool -L x13ashtml | awk '/libgfortran.5.dylib/ '{print $1}'` /Library/Frameworks/R.framework/Resources/lib/libgfortran.5.dylib ../tools/x13as_html/x13ashtml
+		install_name_tool -change `otool -L x13ashtml | awk '/libquadmath.0.dylib/ '{print $1}'` /Library/Frameworks/R.framework/Resources/lib/libquadmath.0.dylib ../tools/x13as_html/x13ashtml
+	fi
 	mkdir -p ../inst/bin
 	cp -f ../tools/x13as_html/x13ashtml ../inst/bin/

[eddelbuettel]

I think we are ready. Finalizing a release branch for review.

[christophsax]

Amazing, thank you so much!