Investigate join-by performance about json-stat

[jfsiii]

I am glad to look into this, but am not familiar with the various valid forms of values that should be supported.

If you publish tests showing some examples, I can take a crack at tuning the function.

[jfsiii]

Here are two screenshots from the trace.

The code I'm profiling is

const merged = JSONstatUtils.join(datasets, { by: 'date', label: meta.label });

where datasets is an array of 30 entries like the one in the issue description.

Note that there are 50 calls to toTable and many of the calls are 100-200ms

[badosa]

@jfsiii Thank you very much for looking into this. For generalization's sake, join relies on toTable and fromTable. So yes, in a dataset join by dimension (like your example) you need at least as many toTable as datasets (in your example, 30 datasets of around 800 values each: total 24000 values).

That might not be the most efficient solution but it's probably one of the simplest ones because normalization is taken care by toTable.

The issue can be tried to be solved using a different approach (avoiding toTable altogether) or improving toTable. The first solution requires modifying JJUS, the second, JJT. Something that has to be taken into account is that JJT was born many years ago and was originally designed to be compatible with very old browsers with no suport for ES5, which explains why the use of the "new" array methods (Array.indexOf, Array.forEach, Array.filter, Array.map, etc.) was included very recently (and very sparingly). Which means there might be room for improvement in JJT too.

I am glad to look into this, but am not familiar with the various valid forms of values that should be supported.

I understand the second solution does not require this familiarity. And, in the first one, the various valid forms can be avoided if instead of dealing with the original objects you deal with the jsonstat instance produced by JSONstat() using the JJT methods. This last approach seems preferable to make sure the solution is forward compatible with future additions.

The official samples are available here:

https://json-stat.org/samples/

I hope this helps. Please let me know if you need more information.

[badosa]

Issue closed because repository has been frozen (new version is at https://github.com/jsonstat/toolkit).