Will there be optimizations? about sts

[lcn2]

The bit runs count test could be optimized (runs.c). Uploading bits to a byte array (BitSequence **epsilon), IMHO, is also not a good idea...

Do you have a proposed patch that could be made into a pull request @zenbooster ?

[lcn2]

The bit runs count test could be optimized (runs.c). Uploading bits to a byte array (BitSequence **epsilon), IMHO, is also not a good idea...

The models that run are extremely data intensive. Early on we did model doing disk I/O for accessing data. As disk I/O, even in SSDs is slow compared to random access memory, code ran orders of magnitude slower.

An alternative approach could use memory mapped files. We looked at this and concluded that when the memory map the size of the data file, you might as well just read the entire dataset into ram at one time and skip the mapping overhead and complexity. Using a sliding window approach, one where the map region is a sub-set of the data size and move around did run well, introduced memory map complexity, and was eventually limited to I/O speed as blocks of disk were paged in.

The sts tool is designed to operate on very large data sets (in order to produce statically meaningful results).

For systems that do NOT have enough ram to hold the data set, the tool may be run in parallel over multiple machines, or in sequential batches (see the -j jobnum and -m mode flags).

[lcn2]

The bit runs count test could be optimized (runs.c). Uploading bits to a byte array (BitSequence **epsilon), IMHO, is also not a good idea...

The decision to put 1 bit into a byte was a performance-based decision. The code generated that needed to pull out bits in as byte slowed down the computation.

We found that the 1 to 8 expansion in ram was worth the performance improvement on the data access side.

[lcn2]

We would be open to a demonstration, @zenbooster, of a code patch that you might suggest in a pull request, assuming that such a pull request improved overall performance.

[zenbooster]

Something like that.

I thought that maybe other algorithms can be optimized somehow ...

[lcn2]

We would certainly consider a replacement proposal using Goncharov's algorithm, @zenbooster, in the form of a pull request.