Environment for cross compiling packages about esy-issues

[wokalski]

I realise that this write up is missing detail but the high level problems made it hard to make it super concrete.

[jordwalke]

Thank you for the research. I will comment with some thoughts:

@jordwalke suggested emulation as a possible solution. Technically it could be possible. However, the cost of emulator, which is (afaik) very slow, eliminates the merits of native. We might as well compile it down to js and run in V8.

I would not suggest emulation for the actual resulting artifacts, but rather just for running the build which generates the final artifacts. We would then run those artifacts natively. This would be like running esy build *on your iPhone` to produce an ARM/iOS executable. Except we don't actually want to do that, so perhaps instead we can run it inside of an emulator on our powerful desktop computers, just to get that resulting ARM binary which can run on device at full speed. What it effectively does, is simplifies all the problems that occur when runtime architecture/platform !== build time architecture/platform. I don't know if this is viable, do you?

[jordwalke]

Populating env vars

One approach I had suggested in the past, is to create a wrapper of the ocaml package, with a dummy ocamlopt, which looks for env vars $arch, and uses it to switch between multiple implementations. For this we need a different kind of environment variable propagation (downward). @wokalski pointed out that this might be handy, but not enough since packages need to be forked sometimes for platforms, and we need to say "when building for archX, use this package instead". So one solution for overriding packages per architecture is the proposed "sandbox package overrides". My original idea of env var based switching is too simplistic.

[wokalski]

@jordwalke re emulation. Yes, it does simplify a bit. I can see three potential issues:

1. Emulator needs to be able to emulate platform X.
2. All dependencies of the whole build system need to be compatible with emulated host
3. Forking/patching might still be needed when a package makes assumptions about host. Like shared libs and iOS in the example.

This is a very interesting solution in general, but it's not interesting in my context. By the time I make Apple ARM + iOS emulate on qemu (or something else) I would've forked all opam packages there are 😛.

[wokalski]

https://github.com/cgreenhalgh/ocaml-crosscompiling
ocaml/opam#1536
ocaml/opam#2476

[jordwalke]

I think #2 is the most challenging:

All dependencies of the whole build system need to be compatible with emulated host

Some depenencies are really just build time dependencies (like Make for example), and might not work on the emulated architecture - yet with the full emulation approach, you instantly make it so you cannot use them.

I think we're on the same page, and we both see that having the build process allow platform/target specific package overrides is the only way to go if we really want this to be seamless - esy build and you're done.

[wokalski]

Yes, I think so. If a package works without changes (from what I've read above it's possible for some packages), great, if not override a platform specific package.