NetBeans 12.5/12.6 missing ARM based binaries for MacOSX terminal support in NetBeans about netbeans

[oyarzun]

Remove needs:triage - think we're sure this is a bug - @vieiro @oyarzun please remove that label on responding to an issue.

I would say this is a feature rather than a bug as it is a new unsupported architecture. While we are adding this, we probably should add Linux arm as there are no binaries either.

For NB13 I wonder whether we should make the quick fix change at least, if we can't get this in?

IMHO, these changes can wait until after NB13.

Swift launcher also going to be related to this?

I looked at this as well, for the launcher we can create a universal binary that contains both the x86-64 and arm-64. Currently the NetBeans launcher is built when the macOS installer is created. @mcdonnell-john has been building these on his Mac. It can be created with the following command with the latest Swift 5.4 tools, and maybe 5.3.

swift build --configuration release --arch arm64 --arch x86_64

Did you see the issue about possible resource leak above comment? in #3467 (comment)

@tmulle I just pushed a commit which should fix building the killall macOS arm binary.

Also added needs:discussion as something to follow up on dev@ here - we should move the location of the binaries to Maven if possible, similar to the Windows launcher, and wind down usage of OSUOSL https://lists.apache.org/thread/9tf28w5r3wxx9fwbl12m6x36sb3pfdry

There are a few platform specific binaries that would benefit from moving the compilation and packaging to a better process. I will start a discussion thread on the dev mailing list.

[tmulle]

@oyarzun Hi, just wondering if this will make it into the official release process for NB18+ so I don't have to keep copying my files on each upgrade?

Thanks!

[ctipper]

Also an unnecessary change, and it stopped the terminal from working.

[oyarzun]

@tmulle can you build and test the binaries on a M1 Mac? oyarzun@5523c7f

@vieiro FYI, we can cross compile the binaries on an Intel mac.

[tmulle]

@oyarzun I cloned your git REPO and ran "ant -Dcluster.config=basic" like I did during my testing and I don't see any arm binaries in the directory. I ran the netbeans build and the terminal was blank.

netbeans git:(master) git config --get remote.origin.url
https://github.com/oyarzun/netbeans.git

After the build finished:

netbeans git:(master) ls nbbuild/netbeans/ide/bin/nativeexecution/
Linux-sparc_64/  Linux-x86_64/    MacOSX-x86_64/   SunOS-x86/       Windows-x86/     dorun.sh*
Linux-x86/       MacOSX-x86/      SunOS-sparc_64/  SunOS-x86_64/    Windows-x86_64/  hostinfo.sh

[tmulle]

@oyarzun Sorry, my bad. I didn't checkout the macOS_arm branch after pulling the code.

But even after switching to the macOS_arm branch and building, the binaries are still not being generated.

Unless, I'm doing something wrong..

[oyarzun]

@tmulle the binaries are not built using ant. I believe ant will download precompiled binaries from http://netbeans.osuosl.org/binaries/06D8BC67B2F104F2DE773CB0F77AACE6B3DF553D-exechlp-1.0.zip
Assuming my changes works, a PMC member would need to upload a new zip that includes the macOS arm binaries.

To test my branch you would need to run sh ./buildall.sh in the tools dir and manually copy the exes to nbbuild/netbeans/ide/bin/nativeexecution/MacOS-arm

FYI, I just pushed another commit to that branch since there was a typo in one of the makefiles.

[vieiro]

Hi @tmulle, as @oyarzun says it's just the sh ./buildall.sh that is required, i.e., you don't need to build the whole NetBeans.

The steps to verify that the library can be built in M1 would be:

cd a-working-directory-you-want
git clone https://github.com/oyarzun/netbeans.git
cd netbeans
git checkout macos_arm
git pull # make sure we're in the last commit in this branch
cd ide/dlight.nativeexecution/tools/
sh ./buildall.sh
ls -ali ../release/bin/nativeexecution

[tmulle]

@oyarzun @vieiro ok, that might have worked!

Just a small change:

1. I had to copy the files built to MacOSX-arm_64 and not MacOSX-arm.

The org.netbeans.modules.nativeexecution.api.util.HelperUtility.java class builds the file it's looking for as 'MacOSX-cpufamily_bitness'

Here is where the path is used in that class..path is set to "bin/nativeexecution/MacOSX-arm_64/pty"
Using the original MacOS-arm as you mentioned causes a "MissingResourceException"

protected File getLocalFile(final ExecutionEnvironment env)
           throws ParseException, MissingResourceException {

       InstalledFileLocator fl = InstalledFileLocatorProvider.getDefault();
       MacroExpander expander = MacroExpanderFactory.getExpander(env);
       String path = expander.expandPredefinedMacros(pattern);

       File file = fl.locate(path, codeNameBase, false);

       if (file == null || !file.exists()) {
           throw new MissingResourceException(path, null, null); //NOI18N
       }

       return file;
   }

Once I had the correct directory, I then could open the terminal and according to Activity Monitor the PTY process was Apple instead of Intel which means it worked.

I'm not sure why 2 PTY processes open when there is only 1 terminal visible in NetBeans, however.

Here is the file info from the Inspector in Activity Monitor:

One thing also, is if I close the terminal in NetBeans.. those processes don't seem to close.
If I open more terminal windows, more PTY processes show up and never quit, even if I shutdown NetBeans..might be a resource leak somewhere??

Here is a ps -ef | grep pty after I shutdown NetBeans and you can see the PTY processes still hanging around:

nativeexecution git:(macos_arm) ps -ef | grep pty
  501 26430 24726   0 11:23PM ttys002    0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox pty
  501 25620     1   0 11:07PM ttys006    0:00.00 /private/var/folders/hh/zxltvdg95x57jm4ln9b7wl380000gn/T/dlight_tmulle/4ab37fae/0983141439/pty --dir /Users/tmulle/Development/Projects/glazedlists-tutorial --set-erase-key --env DYLD_LIBRARY_PATH= --env LD_LIBRARY_PATH= --env PROMPT_COMMAND=printf "^[]3;${PWD}^G"; ideopen() { printf "^[]10;ext[::] ideopen $*;^G"; printf "Opening $# file(s) ...\012";} --env TERM=xterm-256color /bin/zsh
  501 25890     1   0 11:22PM ttys006    0:00.00 /private/var/folders/hh/zxltvdg95x57jm4ln9b7wl380000gn/T/dlight_tmulle/4ab37fae/0983141439/pty --dir /Users/tmulle/Development/Projects/glazedlists-tutorial --set-erase-key --env DYLD_LIBRARY_PATH= --env LD_LIBRARY_PATH= --env PROMPT_COMMAND=printf "^[]3;${PWD}^G"; ideopen() { printf "^[]10;ext[::] ideopen $*;^G"; printf "Opening $# file(s) ...\012";} --env TERM=xterm-256color /bin/zsh
  501 26024     1   0 11:22PM ttys006    0:00.00 /private/var/folders/hh/zxltvdg95x57jm4ln9b7wl380000gn/T/dlight_tmulle/4ab37fae/0983141439/pty --dir /Users/tmulle/Development/Projects/glazedlists-tutorial --set-erase-key --env DYLD_LIBRARY_PATH= --env LD_LIBRARY_PATH= --env PROMPT_COMMAND=printf "^[]3;${PWD}^G"; ideopen() { printf "^[]10;ext[::] ideopen $*;^G"; printf "Opening $# file(s) ...\012";} --env TERM=xterm-256color /bin/zsh
  501 26152     1   0 11:22PM ttys006    0:00.00 /private/var/folders/hh/zxltvdg95x57jm4ln9b7wl380000gn/T/dlight_tmulle/4ab37fae/0983141439/pty --dir /Users/tmulle/Development/Projects/glazedlists-tutorial --set-erase-key --env DYLD_LIBRARY_PATH= --env LD_LIBRARY_PATH= --env PROMPT_COMMAND=printf "^[]3;${PWD}^G"; ideopen() { printf "^[]10;ext[::] ideopen $*;^G"; printf "Opening $# file(s) ...\012";} --env TERM=xterm-256color /bin/zsh

[vieiro]

Hi @tmulle , thanks!

Would you please run this as well so we understand what exact files are generated?

cd a-working-directory-you-want # same places as above
cd netbeans
cd ide/dlight.nativeexecution/tools/
find ../release/bin/nativeexecution

i.e., we want to know the exact names (i.e., if we're getting MacOSX_arm-64 or MacOSX_arm_64

You may want to zip that directory and attach here.

@oyarzun , we'll need at least another independent compilation (to ensure @tmulle is not adding an extra bitcoin miner to the binaries :-D ) and then upload the zip and new hash with these.

[tmulle]

@vieiro haha..no bitminer code here :)

Here is the output of the find command.. I ran it inside the current macos_arm branch I previously built.

The NetBeans code wants the directory to be called 'MacOSX-arm_64'

You can see it's building to the wrong directory name that the code is looking for as mentioned in my previous comment a few mins ago.

➜  netbeans git:(macos_arm) cd ide/dlight.nativeexecution/tools
➜  tools git:(macos_arm) find ../release/bin/nativeexecution
../release/bin/nativeexecution
../release/bin/nativeexecution/MacOSX-arm
../release/bin/nativeexecution/MacOSX-arm/pty_open
../release/bin/nativeexecution/MacOSX-arm/process_start
../release/bin/nativeexecution/MacOSX-arm/privp
../release/bin/nativeexecution/hostinfo.sh
../release/bin/nativeexecution/dorun.sh

[vieiro]

Thanks @tmulle ! Well, it's a pity we don't have a bitcoin miner in NetBeans :-D.

@oyarzun even though the binaries are added to the zip file, we still will need to modify the nbproject/project.properties file [1] so NetBeans adds them to the module (you can do that in Properties/Libraries/Jar files as well).

[1]
https://github.com/oyarzun/netbeans/blob/b693623ebc0cf40f16fc45015cc8eef73ec86e7e/ide/dlight.nativeexecution/nbproject/project.properties#L76

[neilcsmith-net]

Remove needs:triage - think we're sure this is a bug - @vieiro @oyarzun please remove that label on responding to an issue.

Also added needs:discussion as something to follow up on dev@ here - we should move the location of the binaries to Maven if possible, similar to the Windows launcher, and wind down usage of OSUOSL - cc @ebarboni and https://lists.apache.org/thread/9tf28w5r3wxx9fwbl12m6x36sb3pfdry

@oyarzun Swift launcher also going to be related to this?

For NB13 I wonder whether we should make the quick fix change at least, if we can't get this in?

[tmulle]

@vieiro @oyarzun Did you see the issue about possible resource leak above comment? in #3467 (comment)

Just want to make sure it's not missed in case it is a real issue..

[vieiro]

@neilcsmith-net , let's move this to the mailing list then.
@tmulle , it seems we have a "killall" utility in-there, maybe it's not working on M1. Needs some research :-).

[neilcsmith-net]

Just to add, I noticed this also fails when running with an x86_64 JDK on Apple Silicon.

[oyarzun]

I think the JDK is irrelevant since it uses the hostinfo.sh shell script to determine the underlying system info. Which once fixed to support arm would still return arm when running a x86_64 JDK.

https://github.com/apache/netbeans/blob/c084119009d2e0f736f225d706bc1827af283501/ide/dlight.nativeexecution/release/bin/nativeexecution/hostinfo.sh

netbeans/ide/dlight.nativeexecution/src/org/netbeans/modules/nativeexecution/support/hostinfo/impl/UnixHostInfoProvider.java

Line 69 in c084119

hostinfoScript = fl.locate("bin/nativeexecution/hostinfo.sh", "org.netbeans.modules.dlight.nativeexecution", false); // NOI18N

[tmulle]

@oyarzun Hi again... just wanted to let you know that this is still missing in NetBeans 14. I had to copy my modified hostinfo.sh and the ARM folder for Mac to the installation that was created by the above steps.

Wasn't sure if this was included in the official releases yet.

[oyarzun]

@tmulle correct no changes have been incorporated into an official release or master.

[tmulle]

I just installed NetBeans 16 and looks like the changes still aren't in for the terminal.
I have to copy my modified hostinfo.sh and ARM folder I locally built.

Will this be in a future release? I know not everyone has M1 Macs..

I think there are companies you can rent M1 Macs in the cloud for build processes for those that don't or can't have the proper machines.

[tmulle]

Hi, any status on this? I just installed Netbeans 17 (working great so far!) - Although not a fan of the new MacOS icons. I don't like the outline folders...:(

Anyway, in order to get my terminal to open in Netbeans I had to copy the files once again.

Will this be going into master anytime soon?

[ctipper]

I am unable to tell what the blocker is on this thread, Jan 2022 this PR was raised, any status update? Netbeans Terminal not functional on Apple Silicon is not a minority problem anymore.

[tmulle]

@ctipper @neilcsmith-net
If it helps I can submit the binaries that I built a while back that I keep copying from new release of NB..it consists of the binaries for the libs and the modify hostinfo.sh file.

I see #6012 was just posted so other people are now hitting this issue so it's now more important I would think.

It would be nice to not have to keep copying my binaries to get the terminal working in NB.

I just had to do it for NB 18.

[ctipper]

I now have an M2 Mac and I'm curious why I was able to install Netbeans 18 with aarch64 JDK 17 and have a working Terminal, I think it's running in Rosetta, I had a new installation.

I used the package installer, #6012 reporter was using the zip archive, this may be significant.

[fixbugbroken]

I can confirm similar behavior using the Netbeans 18 zip archive with M2 Mac. The terminal is blank and does not show any prompts. I'm using the zip archive since it runs as a native app (kind=apple under activity monitor with no prompts for Rosetta) where as the package installer does not.

However, I am getting the following error message when while launching the zip archive version of Netbeans:

This appears twice before the Netbeans window opens. Note this has been reported here #6014

[ctipper]

So if you use package installer and have aarch64 JDK in /Library/Java/VirtualMachines Netbeans does execute natively. It is only the Terminal that runs in Rosetta. All my experience is that trying to execute netbeans without the launcher runtime i.e. from the bundled script, fools all the logic that triggers tccutil and may mess up the logic Apple uses to trigger Rosetta.

[fixbugbroken]

@ctipper thanks for info.

I'm trying to keep Rosetta of my dev machine so that I get immediate warnings from macOS if our universal app bundle for macOS fails to be recognized as a universal app. We have had issues with Info.plist settings and launcher scripts that trigger Rosetta (even though the project is built to be universal). This allows us to catch these kind of issues early and easily.

[neilcsmith-net]

@fixbugbroken I assume that you tried the suggested workaround for the error message on libjnidispatch? I guess this might be because loading an unsigned jnilib will fail except in Rosetta? That might rule out the zip package for macOS M1/M2 for good?!

I'm happy to look at building a variant of the Codelerity installer with an aarch64 JDK and/or no JDK, if you'd be interested in testing? Contact me via email on GitHub profile. I'm not sure if there's any universal JDK yet?

[fixbugbroken]

@neilcsmith-net Yes clearing the extended attribute removes the warning from macOS. However, as mentioned by the OP the zip file does not contain a directory for MacOSX-aarch64 binaries under bin/nativeexecution directory.

I could certainly test a Codelerity installer if that will help Netbeans build an installer that does not prompt for Rosetta. I will reach out to you via e-mail.

There are arm64 builds for all JDK versions as far back as JDK 8 (see OpenJDK, Adoptium and Azul). I'm currently running an arm64 distribution of JDK 20 with the Netbeans zip package. Of course the arm64 JDK distributions do not trigger prompts for Rosetta.

[neilcsmith-net]

@fixbugbroken take a look at the installer linked from #6052 (comment)

The nativeexecution directory that initiated this issue is a slightly separate concern that will still need addressing.

There are arm64 builds for all JDK versions as far back as JDK 8 (see OpenJDK, Adoptium and Azul). I'm currently running an arm64 distribution of JDK 20 with the Netbeans zip package. Of course the arm64 JDK distributions do not trigger prompts for Rosetta.

That wasn't what I meant by universal JDK though. I'm aware of the arm64 JDKs. We (as in Codelerity) might ship a separate installer with an arm64 JDK bundled, although I'd rather see this issue resolved first. The Apache release will never bundle a JDK.

The other thing is considering using NBPackage to build the Apache released macOS installer as well, with a universal build of the Swift launcher, as well as fixing the signing issues. That's being discussed in the other issue.

[ctipper]

I tried Netbeans 19 the work around has no effect now and this PR is more urgent than ever. Not a particular fan of the new pkg installer, it's only meant for distributing libraries and it prevents the work around I mentioned above.

[neilcsmith-net]

Not a particular fan of the new pkg installer, it's only meant for distributing libraries and it prevents the work around I mentioned above.

I'm not sure what you mean by the "only meant for distributing libraries" comment? Using pkg? The old installer was the same, just broken in multiple other ways.

We might look at shipping the app bundle directly in a dmg or zip (easier and preferred in my opinion). I doubt this will make a difference for you - everything is now signed with the hardened runtime and the launcher is universal architecture when no JDK is specified. So, altering files in the bundle is not possible, and the fact that the launcher architecture might have fooled macOS in some circumstances was a happy accident.

You could use nbpackage to build from the zip yourself, forcing the launcher to x86_64 rather than universal, or just look at one of various macOS options to force use of Rosetta for the launcher? That might still work.

Simply put, we need to fix this properly by finding a way to work with the correct architecture, ideally by NB20.

[ctipper]

So I think not triggering Rosetta is governed by how app is installed. DMG method before and I think it should be brought back, helps installing side by side.

[ctipper]

I think I'll stick to Netbeans 18, I really have no clue why this issue has been open for three years.

[neilcsmith-net]

The previous installer was still a pkg, just inside a dmg! It has nothing to do with the change in the triggering of Rosetta. That is because the Swift launcher in NB18 was built with x86_64 and the one in NB19 is built with universal architecture.

[ctipper]

I didn't see that that would explain it.

[neilcsmith-net]

Also an unnecessary change, and it stopped the terminal from working.

The changes were a necessary part of fixing #6052 and other related issues. The changes were also tested and discussed at https://lists.apache.org/thread/gvp7s1l1s0lkltmm5q7q2q1gxc38wnll

NetBeans does not yet fully support running on macOS Arm unfortunately. You can run on an x86_64 JDK and use arm64 JDKs for projects.

Personally, I'd prefer we stopped shipping a macOS installer without bundled JDK runtime anyway - using the zip in those circumstances is preferable IMO for a multitude of reasons. Either directly, or by running NBPackage on the zip with a JDK of your choice.

[tmulle]

I agree, we need better support for macOS Arm.

Each release I manually copy the binaries I built on my M1 mac to the new Netbeans folder to get the terminal to work.

I run straight ARM jdk using SDKMAN and things work fine.

If it would help I can make a GitHub repo with my folder that I built for the ARM support to get the terminal working.

It's not perfect, but it will get your terminal working again.

[neilcsmith-net]

@tmulle what we probably need to do is extricate the existing source to a separate repo (similar to https://github.com/apache/netbeans-native-launchers ) then look at using GitHub actions to build them, and run a release vote on them. The NetBeans build can then use the released native binaries.

I think your input into that would be very welcomed! I also have a local M1 mac, so happy to do some testing and helping out with this.

cc/ @ebarboni also for input

[sroeper]

The problem still exists on Apple Silicon with NB19.

Looks like the binaries are a "closed shop" and not open source. Where is the source code for these? Do we have to wait until a PMC member wants to use Netbeans on Apple Silicon?
Come on, I am willing to do something but I am a bit disappointed (or frustrated). This is marked as feature but it is clearly a bug. There is no way for simple contributors to help out and this is open for more than 1 and half year now.

[neilcsmith-net]

@sroeper of course it's open source - the source is at https://github.com/apache/netbeans/tree/master/ide/dlight.nativeexecution/tools

If we can get a PR in that includes the relevant changes (eg. @oyarzun or @tmulle changes linked above??) then we can look at a similar workflow as for the profiler to release the binaries for consumption in the build. I for one would like to see this fixed in NB20. There is a lot that "simple contributors" can help out with here.

[tmulle]

@neilcsmith-net

I can submit a PR for the changes I did, but it is just a few lines added to hostinfo.sh.

As far as any other changes I did, I didn't do anything else but following the instructions at #3467 (comment)

To test building on my M1.

If you follow the comments from that link down you can see what I ran into.

TL;DR - running buildall.sh builds the code on my M1 mac to netbeans/ide/dlight.nativeexecution/tools/MacOSX-arm when the final directory needs to be MacOSX-arm_64 as mentioned in the comments above.

[vieiro]

@neilcsmith-net @tmulle

To clarify the situation.

1. ide/dlight.nativeexecution uses native binaries exechlp that are currently hosted in OSUOSL. These are compiled (see the contents of the zip file) for the following platforms and oses:

• Linux (ARM 64, Sparc 64, x86, x86_64)
• MacOSX (x86, x86_64)
• SunOS (sparc, x86, x86_64)
• Windows (x86, x86_64)

2. The code for these binaries is actually APLv2, and is hosted in the ide/dlight.nativeexecution/tools subdirectory (wich in turn is a NetBeans CND project).

3. This APLv2 source code was compiled ages ago (2012, 2015, 2017, 2022, if the dates in the zip file are correct) for those different platforms. Of course at the time of compilation there was not such a thing as "Apple M1" or "Apple M2" and their variants.

4. NetBeans tries to read this "external" dependency at runtime from this binary dependency. For "Apple M1" and "Apple M2" boxes it cannot find the proper binary, and that's why this fails.

To solve the issue at once:

a) Compile the binary once in the required platform using the "ide/dlight.nativeexecution/tools/buildall.sh" script (I assume that if you build this in an Apple M1 box it will work well for Apple M2 boxes)
b) Add the compiled binaries files to the directory MacOSX-arm_64/ (that @tmulle recommends, so NetBeans can find them) to the downloaded external/exechlp-1.2.zip file. Rename this to, say, external/exechlp-1.3.zip.
c) Upload the binary file to OSUOSL with the updated SHA1 sum.
d) Modify external/binaries-list to reflect the proper SHA1 sum.
e) Wait for apple to release Apple M3, and goto a).

To solve the issue forever:

a) Ensure that Apple becomes an sponsor of the Apache Software Foundation. D'oh! Apple is a platinum sponsor!
b) Ensure that Apple is a very good sponsor (it already is) and makes some Apple M1 boxes for us (ASF) to build stuff for their customers.
c) Install a cross platform compiler (clang/gcc) in the Apple M1 donated by Apple.
d) Create a build job that uses this Apple M1 computer to build binaries for Apple M1 boxes. If you've succeeded at d) ensure that the compiler cross compiles for other platforms/boxes.
e) Integrate this build-bot in the NetBeans release cycle (github actions, etc.).
f) Enjoy a NetBeans release bot that compiles binaries for all Apple architectures.

Cheers,
Antonio

[neilcsmith-net]

Thanks @vieiro but I wasn't intending to use OSUOSL or user compiled binaries (which would both be -1 anyway). We already have access to macOS cross-compilation and/or Apple Silicon via GitHub Actions. As long as the build process in the main repo works, I intend to fork the workflow I've just merged for the profiler binaries and release via dist.a.o - which is close to your second suggestion.

See #6502 I'd like to see if we can fix both these for NB20.

As well as merging required fixes, we need to clarify exactly which folders are required (ie. anything outside of dlight.nativeexecution/tools). We might need the same hack as currently in the profiler source bundle to replicate the destination folders?

Both Apple and GitHub are already ASF sponsors btw! 😄

[neilcsmith-net]

... actually, I may have jumped the gun on availability of the architecture on GitHub actions. Had seen a reference to being in public beta.

However, as @oyarzun points out above (and his commit seems to include) we can cross-compile on macOS. If those changes make it in, we might be OK just using an x86_64 runner as we are for profiler binaries, for now?

[sroeper]

@neilcsmith-net Yes, it's enough to have a x86_64 runner and cross compile the binaries. There is no need for a M1/M2/Mwhatever.

The only change you have to do is to call clang with the -arch option.
For x86_64 call with -arch x86_64
For arm64 call with -arch arm64

To build something which includes both (i.e. universal library/executable) call with -arch x86_64 -arch arm64

Works the same on a M1/M2

[neilcsmith-net]

I've adapted the workflows of @tmulle and @oyarzun for #6513 - thanks both! Artefacts from the workflow can be found at eg. https://github.com/apache/netbeans/actions/runs/6382908341 ... or look for a later run.

If anyone is willing to check those, that would be great! There's still some work to do on that, but it might be enough to get us some terminal support for Apple Silicon in NB20?!

[neilcsmith-net]

Should be working in NB20. A release candidate should be available next week. Please help testing that when it's ready.