[Bug]: Correlation with many items causes engine restart, no user output about jasp-issues

[tomtomme]

Interesting.
Tested with 0.18.3 flatpak

I could compute this for 128 variables even... BUT only if I added them step by step. I started with 100 variables and then added 10 more step by step.
If I start with roughly >110 then I can reproduce the bug (only grey area in output window)

Data
test.jasp.zip

[github-actions]

@richlv, thanks for taking the time to create this issue. If possible (and applicable), please upload to the issue website (#2746) a screenshot showcasing the problem, and/or a compressed (zipped) .jasp file or the data file that causes the issue. If you would prefer not to make your data publicly available, you can send your file(s) directly to us, [email protected]

[richlv]

Link to logs sent in email.

Edit: email delivery failed. Attempting a resend to the expanded address.

[tomtomme]

How many variables did you try to correlate?
How many rows did the data contain?
How much RAM (memory in GByte) does your machine have?

[richlv]

126 variables, 55 rows, 16GB (available memory might be half or less at times).
As mentioned, this was just a test - and if something is actually unfeasible (likely), providing some feedback to the user would be useful :)

[tomtomme]

Test with 0.19 beta (23. may)

Here only 75 variables work. At least >84 vars also reveals the bug.

[JohnnyDoorn]

I think this issue is better suited for @RensDofferhoff, @JorisGoosen, or @boutinb - I can forward the logs to whoever can help here.

[boutinb]

On my Mac (32GB, 10 Cores M1 Pro), the test.jasp takes a bit long to refresh (~5 minutes), but it works...

[tomtomme]

Also if you add all 128 vars in one go?
I think I saved the state that worked with 100 or 110 vars. Did I?

[boutinb]

It works also if I add all 128 vars in one go.

What strikes me is that the Desktop log file is huge (207 MB), and not the Engine log file (just 17 KB).
JASP 2024-05-29 09_58_07 Desktop.log.zip

This is also the case in the log files of @richlv

When I have a look at the log file, I see that the Desktop receives many temporary answers from the Engine. Just at the end, the Engine sends a JSON string of 3.1 MB.
If I export the result in HTML, it is 26,7 MB, for a table of 128 * 2 rows and 128 columns
Maybe this transformation of such a JSON string to a HTML string takes so long... But if I have a look at my CPU, I see that that nearly most of the time is taken by the Engine: when the Engine stops, the result is really quickly displayed. That means that the JSON transformation was really quick.

I see also in the log files of @richlv, that it takes 11 minutes before the Engine is ready, at the end it generates a JSON string of 6.4MB, the double of what I get, so I can't imagine that it is the problem.

[JorisGoosen]

string of 6.4MB, the double of what I get, so I can't imagine that it is the problem.

Did you check ram usage while this was being calculated?

[tomtomme]

@boutinb
Opening the file, then refreshing does work for me too!
But closing the analysis and starting it from scratch with 128 vars does not!

@JorisGoosen
RAM usage of JASP peaked at about 4GB and I had plenty free memory space left (of 32GB)

[JorisGoosen]

The RAM usage of 4GB was when it worked or when it did not work?
Or perhaps both?

On the other hand, whenever a new vriable gets dragged into the list it asks the engine to restart, maybe something in that process is breaking this.

[tomtomme]

@JorisGoosen
both.

[tomtomme]

still valid for current flatpak beta. But now I get a new error message from terminal:

[86:12:0100/000000.551719:ERROR:platform_shared_memory_region_posix.cc(47)] Descriptor access mode (2) differs from expected (0)

does this help debugging it?