[resolved] v3.0.2 regression, much slower gathering entropy vs. v3.0.1 about jitterentropy-library

[joshuaehill]

There were a few changes from 3.0.1 to 3.0.2 that would have the effect of slowing down the output; the default osr was increased from 1 to 3 in the default mode (specifically when the JENT_CONF_DISABLE_LOOP_SHUFFLE macro is defined) and the optimizer is turned off (current setting is -O0; it was set to -O2). There is also a change that fixes the number of loops performed to the previous minimum number of loops that should make things a bit faster; I'd expect a slowdown on the order of 2-3x, which is consistent with your results with one core.

These changes were made after a few folks noticed that the entropy was overreported on some tested platforms (See Issue #21). If you wanted to figure out if either of these contribute to the slowdown you are seeing, it's easy enough to test; you could remove the #define for JENT_CONF_DISABLE_LOOP_SHUFFLE and see how the timing changes, and then recompile with the optimizer enabled by changing the optimizer compiler flag in the Makefile (and removing the #error line that tries to prevent you from enabling the optimizer). I wouldn't suggest actually using the library in an optimized form, but it would help you determine where you are seeing the slowdown.

I don't know how to explain what is going on across multiple cores in rngd; perhaps Stephan has a better insight into that.

[smuellerDD]

Am Sonntag, 25. April 2021, 17:23:39 CEST schrieb Lonnie Abelbeck: Hi Lonnie,

Our project uses `rngd` and `jitterentropy-library` when needed. For example on a PC Engines APU2 board (AMD GX-412TC, 1 GHz 4x core). Our rngd startup script monitors `/proc/sys/kernel/random/entropy_avail` and waits up to 10 seconds while `entropy_avail` is less then 256. Throws an error if it takes over 10 seconds. Using `jitterentropy-library` v3.0.1, it takes 4-5 seconds for `entropy_avail` to reach over 256. The total `rngd` CPU time (across 4 cores) is 17.39 seconds. ``` # /usr/sbin/rngd -t -x hwrng -x rdrand ... Entropy rate: 110.8 Kbits/sec averaged over 35 iterations for 55.01 seconds ``` Using `jitterentropy-library` v3.0.2, it takes over 10 seconds for `entropy_avail` to reach over 256. The total `rngd` CPU time (across 4 cores) is 72.71 seconds. ``` # /usr/sbin/rngd -t -x hwrng -x rdrand ... Entropy rate: 27.15 Kbits/sec averaged over 13 iterations for 41 seconds ``` I find it interesting that it is 4x slower, with a 4 core CPU ... possibly just a coincidence ... or a clue. Any guidance which commit this regression may have occurred in ?

As Josh just mentioned, this reduction in performance is expected (thank you Josh!). Yet, I find your numbers intriguing. With a given performance of a couple of kBit per second, I find it hard to believe that the entropy_avail does not increase above 256 within a fraction of a second. Note, you can easily do a performance measurement of the Jitter RNG (and thus the behavior of the jitterentropy-rngd): 1. clone https://github.com/smuellerDD/jitterentropy-library 2. cd tests/raw-entropy/recording_userspace 3. make -f Makefile.lfsroutput 4. time ./jitterentropy-lfsroutput 10000 > out The time command will tell you the amount of time required for the operation. Now, calculate 320000 / <time returned by `time`> to get the kbytes / s rate. Could you return that value? I expect that this value should indicate that the 256 bit entropy_avail should be reached very fast. Thus, something else is wrong when your system takes more than 10 seconds with the 3.0.2 to get that value. Thank you. Ciao Stephan

[abelbeck]

@smuellerDD : Keep in mind my measurements are "all things equal" except for the jitterentropy-library version, so I would prefer to focus on the jitterentropy-library change. As @joshuaehill mentioned, I'll look info JENT_CONF_DISABLE_LOOP_SHUFFLE.

[smuellerDD]

Am Sonntag, 25. April 2021, 19:51:29 CEST schrieb Lonnie Abelbeck: Hi Lonnie,

@smuellerDD : Keep in mind my measurements are "all things equal" except for the jitterentropy-library version, so I would prefer to focus on the jitterentropy-library change. As @joshuaehill mentioned, I'll look info `JENT_CONF_DISABLE_LOOP_SHUFFLE`.

I totally agree - but my point is that if you need more than 10 seconds to get the entropy estimator above 256, something is really strange. And thus I pointed out how you can measure the performance of the Jitter RNG. Ciao Stephan

[abelbeck]

Compiling with:

--- rng-tools-6.5/jitterentropy-library/jitterentropy.h.orig	2021-04-25 12:58:10.615690346 -0500
+++ rng-tools-6.5/jitterentropy-library/jitterentropy.h	2021-04-25 12:59:37.025192013 -0500
@@ -78,7 +78,7 @@
  *
  * By enabling this flag, the time of collecting entropy may be enlarged.
  */
-#define JENT_CONF_DISABLE_LOOP_SHUFFLE
+/* #define JENT_CONF_DISABLE_LOOP_SHUFFLE */
 
 /***************************************************************************
  * Jitter RNG State Definition Section

made things worse. 8.6x slower than v3.0.1

The total rngd CPU time (across 4 cores) is now 92.96 seconds.

# /usr/sbin/rngd -t -x hwrng -x rdrand
...
Entropy rate:  12.83 Kbits/sec averaged over 5 iterations for  45.66 seconds

[joshuaehill]

That is unexpected!

That suggests that SHA3 is somewhat slower on your architecture than on my test machine. When JENT_CONF_DISABLE_LOOP_SHUFFLE is not defined, you have to do on average 1,156.5 hashes per jent_random_data call (which outputs a 256-bit block of conditioned entropy; that is 4.5 SHA3 hashes on average per jent_meansure_jitter and 257 jent_measure_jitter per jent_random_data). When JENT_CONF_DISABLE_LOOP_SHUFFLE is defined you instead need to perform a fixed 1 SHA3 per jent_measure_jitter, and 769 jent_measure_jitter per jent_random_data.

That being said, the rest of the processing overwhelms that difference on my machine, but clearly not on your test machine.

I guess try defining JENT_CONF_DISABLE_LOOP_SHUFFLE and turning optimization back on and see what happens.

[abelbeck]

Compiling with: (previous patch removed)

--- rng-tools-6.5/jitterentropy-library/Makefile.orig	2021-04-25 13:30:54.336767045 -0500
+++ rng-tools-6.5/jitterentropy-library/Makefile	2021-04-25 13:31:29.240760050 -0500
@@ -3,7 +3,7 @@
 CC ?= gcc
 #Hardening
 CFLAGS ?= -fwrapv --param ssp-buffer-size=4 -fvisibility=hidden -fPIE -Wcast-align -Wmissing-field-initializers -Wshadow -Wswitch-enum
-CFLAGS +=-Wextra -Wall -pedantic -fPIC -O0 -fwrapv -Wconversion
+CFLAGS +=-Wextra -Wall -pedantic -fPIC -O2 -fwrapv -Wconversion
 LDFLAGS +=-Wl,-z,relro,-z,now
 
 GCCVERSIONFORMAT := $(shell echo `$(CC) -dumpversion | sed 's/\./\n/g' | wc -l`)
--- rng-tools-6.5/jitterentropy-library/jitterentropy-base.c.orig	2021-04-25 13:35:59.946249588 -0500
+++ rng-tools-6.5/jitterentropy-library/jitterentropy-base.c	2021-04-25 13:36:34.522711644 -0500
@@ -67,10 +67,6 @@
  * None of the following should be altered
  ***************************************************************************/
 
-#ifdef __OPTIMIZE__
- #error "The CPU Jitter random number generator must not be compiled with optimizations. See documentation. Use the compiler switch -O0 for compiling jitterentropy.c."
-#endif
-
 /*
  * JENT_POWERUP_TESTLOOPCOUNT needs some loops to identify edge
  * systems. 100 is definitely too little.

We are almost back to v3.0.1 performance (BTW, this -O2 difference surprised me)

The total rngd CPU time (across 4 cores) is now 19.5 seconds.

# /usr/sbin/rngd -t -x hwrng -x rdrand
...
Entropy rate:  106.7 Kbits/sec averaged over 39 iterations for  56.01 seconds

But sadly, there is now an intermittent startup issue I have never seen before, from rngd

JITTER rng fails with code 10

Failed to init entropy source 5: JITTER Entropy generator

As such rngd typically does not start at system boot, though occasionally will. I never saw this failure with v3.0.1.

[abelbeck]

@joshuaehill :

That is unexpected!
That suggests that SHA3 is somewhat slower on your architecture than on my test machine.

I'm testing on a 1 GHz, 4-core x86_64 box, acting as a Router/Firewall/PBX.

Flags:  fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good acc_power nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt topoext perfctr_nb bpext ptsc perfctr_llc cpb hw_pstate ssbd vmmcall bmi1 xsaveopt arat npt lbrv svm_lock nrip_save tsc_scale flushbyasid decodeassists pausefilter pfthreshold overflow_recov

The newer faster boxes typically have rdrand, so the older, slower boxes is where we need a jitterentropy source.

[joshuaehill]

The optimizer makes a huge difference in run time performance (i.e., samples per second), but unfortunately recent testing shows that the optimizer also really decreases the variability of the process and tends to encourage repeating patterns (and thus yields a substantial decrease in entropy per sample).

So, your process completes much more quickly with the optimizer enabled, but you get much less entropy per request than was expected, so that's not a great trade-off. :-)

On the processor, that's again surprising, as that's architecturally fairly close to my test platform.

[joshuaehill]

On your startup issue, the only change between 3.0.1 and 3.0.2 that I can think of that might be related was commit f37124e8ca2005d9feb6eb5ed4bee68c3a85657e. If the calls were very regular (and the 2nd order delta was 1 or less on average) then this would trigger an initialization error. That said, such an error is desirable, as that suggests that the timer is too consistent to be a good source of entropy. This isn't the sort of error that I'd expect to see on a x86_64 system, as the TSC counter is very very fine grained.

[abelbeck]

@smuellerDD :

4. time ./jitterentropy-lfsroutput 10000 > out

The time command will tell you the amount of time required for the operation.
Now, calculate 320000 / <time returned by time> to get the kbytes / s rate.
Could you return that value?

Our project cross-compiles with a toolchain, but I cheated and natively used Debian 10 to build jitterentropy-lfsroutput and copied the binary to my test box. It ran.

The jitterentropy-lfsroutput test took 432.9 seconds -> 320000 / 432.9 = 739 Kbits/sec

Note this runs in a single core.

[abelbeck]

@joshuaehill :

On your startup issue, the only change between 3.0.1 and 3.0.2 that I can think of that might be related was commit f37124e8ca2005d9feb6eb5ed4bee68c3a85657e.

Compiling with:

-- rng-tools-6.5/jitterentropy-library/jitterentropy-base.c.orig	2021-04-25 16:15:53.553476853 -0500
+++ rng-tools-6.5/jitterentropy-library/jitterentropy-base.c	2021-04-25 16:17:21.521700758 -0500
@@ -1402,7 +1402,7 @@
 	 * than 1 to ensure the entropy estimation
 	 * implied with 1 is preserved
 	 */
-	if ((delta_sum) <= JENT_POWERUP_TESTLOOPCOUNT) {
+	if ((delta_sum) <= 1) {
 		ret = EMINVARVAR;
 		goto out;
 	}

The intermittent startup issue is pretty much solved. Under normal boot conditions it now works.

But, while testing in the foreground, 1/10 times it still fails.

[abelbeck]

The intermittent startup issue is pretty much solved. Under normal boot conditions it now works.

I spoke too soon, it failed on boot.

I would say it is more intermittent with the patch. :-)

[joshuaehill]

@abelbeck : Would it be possible to output the return value from jent_entropy_init when it fails? That should identify the issue that you are running into.

[joshuaehill]

As a general matter, it's good that we've identified the likely source some of the problems that you are running into, but the prior value of 1 for the cutoff was clearly wrong (though it may be possible to argue some value less than JENT_POWERUP_TESTLOOPCOUNT is more appropriate). More importantly, disabling the optimizer appears to have been a very important change, and if a choice needs to be made between returning quickly and returning the expected amount of entropy, then it seems to me that the only reasonable choice is to return the expected amount of entropy (even if that might take longer than it previously did).

[abelbeck]

@abelbeck : Would it be possible to output the return value from jent_entropy_init when it fails? That should identify the issue that you are running into.

jent_entropy_init() returns 10

#define ERCT		10 /* RCT failed during initialization */

which is a jent_rct_failure(), which is more complex to track.

So it seems the EMINVARVAR patch was a red herring.

[abelbeck]

Is ec->rct_count initialized somewhere ?

[joshuaehill]

Yes, but not explicitly (I'm a bit surprised that static analysis doesn't flag this!)

In the jent_entropy_collector_alloc call, the space for the rand_data structure is allocated using jent_zalloc, which (in the last line prior to returning) memsets all the memory used by this structure to '0' bytes. The variable rct_count is an integer, so this will be the value 0.

For this failure, it looks like you are seeing a RCT failure on startup (meaning a run of "stuck" values. In this context "stuck" means that the delta, delta2 (difference of adjacent delta values) or delta3 (difference of adjacent delta2 values) values are 0. Based on the other failure you reported (EMINVARVAR) , I'd guess that you are seeing a bunch of outputs that have delta2 of 0. I've somewhat lost the thread of your testing: are these failures occurring with the optimizer enabled (-O2) or disabled (-O0)?

[abelbeck]

The ERCT failures are with -O2, but I have not tested much with -O0.

I tested again v3.0.1 (which has -O2) and there is never a startup error.

BTW, I never saw a EMINVARVAR error, your suspicions convinced me I did :-)

[joshuaehill]

Got it. So we still don't know if we are seeing a bunch of delta == 0 (basically not possible on x86-64 using an invariant TSC in a bare metal environment), delta2 == 0 or delta3 == 0.

To be clear, several people noted that with the -O2 flag, the library could fall into consistent patterns of behavior, and the sort of behavior that you are reporting is consistent with that failure mode.

The reason that this wasn't obvious in version 3.0.1 and earlier is that those versions had substantial pseudorandom variation from delta to delta, so historically this style of problem would have been masked unless the underlying hardware counter was so slow that the pseudorandom variation didn't tend to perturb the results. Pseudorandom variation doesn't contribute to entropy, but it does mask failures. You should turn off the optimizer when compiling the library. :-)

[abelbeck]

@joshuaehill : I wonder if a few carefully chosen volatile data types in the inner loop which would allow -O2 for the rest ?

[smuellerDD]

Am Sonntag, dem 25.04.2021 um 14:11 -0700 schrieb Lonnie Abelbeck:

@smuellerDD : > 4. time ./jitterentropy-lfsroutput 10000 > out > > The time command will tell you the amount of time required for the > operation. > Now, calculate 320000 / <time returned by `time`> to get the kbytes / s > rate. > Could you return that value? Our project cross-compiles with a toolchain, but I cheated and natively used Debian 10 to build `jitterentropy-lfsroutput` and copied the binary to my test box.  It ran. The `jitterentropy-lfsroutput` test took 432.9 seconds -> 320000 / 432.9 = 739 Kbits/sec

So, with 740 kBytes / sec, it should take no time to fill the Linux entropy pool. Thus, I feel that there is something else that causes the issue (even with the old code where you said it takes 5 seconds). Let me dig into it. Ciao Stephan

…

Note this runs in a single core.

[smuellerDD]

Am Sonntag, dem 25.04.2021 um 20:00 -0700 schrieb Lonnie Abelbeck:

@joshuaehill : I wonder if a few carefully chosen `volatile` data types in the inner loop which would allow `-O2` for the rest ?

The issue that Lonnie is seeing here seems to be quite clear to me: I understand you compiled with: - O2 instead of O0 - you define JENT_CONF_DISABLE_LOOP_SHUFFLE which implies that only one hash loop is performed - you use an OSR of 3 That implies that the distribution of the execution time of SHA-3 is very narrow and causes a lot of stuck results. Note to get back to 3.0.1 state you would need: - O2 - disable JENT_CONF_DISABLE_LOOP_SHUFFLE - use an OSR of 1 Especially the first two imply that the stuck tests are limited in your environment. In 3.0.2 this is offset with O0. Again, Lonnie, I would not change a bit in the Jitter RNG code, but rather search why the rngd startup time is 5 seconds (old) or more than 10 seconds (new). As you reported, sufficient entropy is available. Let me look into the rngd operation. Ciao Stephan

[abelbeck]

Using rng-tools 6.5 (last version to support static linking of jitterentropy-library)

The startup time is in this loop:
https://github.com/nhorman/rng-tools/blob/0f525ec3a0260a149cf783ece1b104efb69fa752/rngd_jitter.c#L458

v3.0.1 startup: 4 seconds
v3.0.2 startup: 16 seconds

No entropy is added to the kernel until the /* Make sure all our threads are doing their jobs */ loop is completed.

BTW, the latest rng-tools has a similar startup loop:
https://github.com/nhorman/rng-tools/blob/54c9139b328203c4b8c014039e581fc461628b4e/rngd_jitter.c#L436

[abelbeck]

Followup, the #36 (comment) startup times are with no patches, using default compile options.

[abelbeck]

More testing of v3.0.2 with defaults (ie. -O0) I have never had a failed startup.

The slower entropy gathering of v3.0.2 is not a problem for our use, the 16 second startup (vs. 4) is an issue we would like to solve, if possible.

[abelbeck]

More testing of rngd, the jitter buffer size defaults to 16535 which is filled at startup. If I reduce the buffer by a factor of 4 ...

# /usr/sbin/rngd -t -x hwrng -x rdrand -O jitter:buffer_size:4133 -O jitter:refill_thresh:4133
...
Entropy rate:  28.05 Kbits/sec averaged over 43 iterations for  55.01 seconds

I'm back to a 4 second startup.

Sound like a valid solution ? Was there something special with the 16535 buffer_size ?

[smuellerDD]

Am Montag, dem 26.04.2021 um 07:33 -0700 schrieb Lonnie Abelbeck:

More testing of `rngd`, the jitter buffer size defaults to `16535` which is filled at startup.  If I reduce the buffer by a factor of 4 ... ``` # /usr/sbin/rngd -t -x hwrng -x rdrand -O jitter:buffer_size:4133 -O jitter:refill_thresh:4133 ... Entropy rate:  28.05 Kbits/sec averaged over 43 iterations for  55.01 seconds ``` I'm back to a 4 second startup. Sound like a valid solution ?  Was there something special with the `16535` buffer_size ?  

Sorry, I am currently distracted with another task - I will check the rngd tonight. But I see no reason to fill any buffer here (this statement is made without looking at the rngd implementation and considering possible reasons). Entropy should be generated when it is needed. The business of accumulating entropy is a challenge (I am currently sitting in front of SP800-90C draft which specifies this, and it is not an easy task).

[abelbeck]

@smuellerDD : rngd captures a buffer of entropy and spits it into /dev/random, captures a new buffer, etc. .

[smuellerDD]

Am Montag, dem 26.04.2021 um 07:33 -0700 schrieb Lonnie Abelbeck:

More testing of `rngd`, the jitter buffer size defaults to `16535` which is filled at startup.  If I reduce the buffer by a factor of 4 ... ``` # /usr/sbin/rngd -t -x hwrng -x rdrand -O jitter:buffer_size:4133 -O jitter:refill_thresh:4133 ... Entropy rate:  28.05 Kbits/sec averaged over 43 iterations for  55.01 seconds ``` I'm back to a 4 second startup. Sound like a valid solution ?  Was there something special with the `16535` buffer_size ?  

Ok, I checked the code in rngd. Indeed, I find no reason why this buffer is as it is. Let me ckeck with Neil Hormann, the maintainer. Barring any explanation, I see no reason to keep the buffer that high (apart from handling bursts later on during runtime - but entropy is badly needed during startup and much less during runtime). So, I would even say, a buffer size of, say, 64, is good enough. I will report back after speaking with Neil. Ciao Stephan

[smuellerDD]

Am Montag, dem 26.04.2021 um 07:33 -0700 schrieb Lonnie Abelbeck nhorman/rng-tools#124

[abelbeck]

Thanks @smuellerDD for confirming a lower rngd jitter buffer size is a valid solution.

Keeping the same "character" of 16535, I chose ((16535 + 1) / 4) - 1 = 4133

For the v3.0.2 tests below, I patched rngd.c with new 4133 defaults:
(Alternatively add to the rngd command line -O jitter:buffer_size:4133 -O jitter:refill_thresh:4133)

 	[JITTER_OPT_BUF_SZ] = {
 		.key = "buffer_size",
-		.int_val = 16535,
+		.int_val = 4133,
 	},
 	[JITTER_OPT_REFILL] = {
 		.key = "refill_thresh",
-		.int_val = 16535,
+		.int_val = 4133,
 	},

These new rngd defaults with v3.0.2 keeps things working much as before.

Here are some tests:

Proxmox VM 2-core @ 2096 MHz

# timeout 60 /usr/sbin/rngd -t -x hwrng -x rdrand
v3.0.1: Entropy rate:  253.9 Kbits/sec averaged over 58 iterations for  58.01 seconds
v3.0.2: Entropy rate:  58.58 Kbits/sec averaged over 58 iterations for  58.01 seconds
4.3x slower
v3.0.1: startup time: 0.67 seconds
v3.0.2: startup time: 0.70 seconds

AMD GX-412TC SOC 4-core @ 966 MHz

# timeout 60 /usr/sbin/rngd -t -x hwrng -x rdrand
v3.0.1: Entropy rate:  114.7 Kbits/sec averaged over 41 iterations for  55.01 seconds
v3.0.2: Entropy rate:  27.55 Kbits/sec averaged over 46 iterations for  56.01 seconds
4.2x slower
v3.0.1: startup time: 4.2 seconds
v3.0.2: startup time: 4.4 seconds

Intel Atom D525 2-core 4-thread @ 1800 MHz

# timeout 60 /usr/sbin/rngd -t -x hwrng -x rdrand
v3.0.1: Entropy rate:  112.9 Kbits/sec averaged over 50 iterations for  55.01 seconds
v3.0.2: Entropy rate:  22.42 Kbits/sec averaged over 42 iterations for     54 seconds
5.0x slower
v3.0.1: startup time: 4.4 seconds
v3.0.2: startup time: 5.5 seconds

Additionally, since we had some -O2 startup failures with a patched v3.0.2, would the default -O0 solve that problem ? I ran timeout 15 /usr/sbin/rngd -t -x hwrng -x rdrand in a loop for almost an hour (on two boxes) and searched the terminal screens for "fail" and there were no matches. Looks good.

[joshuaehill]

In my testing, the raw data variability from the library when compiled with -O2 was dramatically lower than it is when compiled with -O0. The raw data distribution is difficult to predict a priori, so I can't say "Failures will never happen when compiled with -O0", but I can say that I expect that failures will happen much much less frequently when compiled with -O0 than when compiled with -O2.

[smuellerDD]

Am Montag, dem 26.04.2021 um 16:44 -0700 schrieb Lonnie Abelbeck:

Thanks @smuellerDD for confirming a lower rngd jitter buffer size is a valid solution. Keeping the same "character" of 16535, I chose `((16535 + 1) / 4) - 1` = `4133` For the v3.0.2 tests below, I patched `rngd.c` with new `4133` defaults: (Alternatively add to the `rngd` command line `-O jitter:buffer_size:4133 -O jitter:refill_thresh:4133`) ```         [JITTER_OPT_BUF_SZ] = {                 .key = "buffer_size", -               .int_val = 16535, +               .int_val = 4133,         },         [JITTER_OPT_REFILL] = {                 .key = "refill_thresh", -               .int_val = 16535, +               .int_val = 4133,

Note, in the next release of the RNG, there will be another performance reduction of about 15% (maybe I limit that to the FIPS mode only, I am not fully sure). Thus, if you apply a change now, I would reduce it even further. Besides, if you look into my considerations I posted at the rng-tools project, I personally think this buffer should be as small as possible. Ciao Stephan

[smuellerDD]

Am Montag, dem 26.04.2021 um 21:52 -0700 schrieb Joshua E. Hill:

In my testing, the raw data variability from the library when compiled with  `-O2` was dramatically lower than it is when compiled with `-O0`. The raw data distribution is difficult to predict a priori, so I can't say "Failures will never happen when compiled with `-O0`", but I can say that I expect that failures will happen much much less frequently when compiled with `-O0` than when compiled with `-O2`.

Thank you for reporting.

[joshuaehill]

@smuellerDD If you are referring to the introduction of ENTROPY_SAFETY_FACTOR, then I think that it seems completely reasonable to limit this behavior to FIPS mode. I support the use of this "safety factor" within 90C, but practical difference is small. I would think that the health tests have a more significant security impact.

[smuellerDD]

Am Montag, dem 26.04.2021 um 23:32 -0700 schrieb Joshua E. Hill:

@smuellerDD If you are referring to the introduction of `ENTROPY_SAFETY_FACTOR`, then I think that it seems completely reasonable to limit this behavior to FIPS mode. I support the use of this "safety factor" within 90C, but practical difference is small. I would think that the health tests have a more significant security impact.

Yes, I refer to this factor. Thank you for chiming in and confirming my hunch to limit it to FIPS mdoe. Thanks Stephan

[abelbeck]

@smuellerDD :

Besides, if you look into my considerations I posted at the rng-tools project, I personally think this buffer should be as small as possible.

Mathematically, I tend to agree, but let's test using this script:

#/bin/sh

for size in 16535 8267 4133 2501 2067 1033 515 257 127 63; do
  echo "$(date +'[%m/%d/%y %H:%M:%S]')  Buffer_Size=$size"

  rstr="$(timeout 600 /usr/sbin/rngd -t -x hwrng -x rdrand -O jitter:buffer_size:$size -O jitter:refill_thresh:$size 2>&1)"

  echo "$rstr" | grep '^Entropy rate:'
  echo ""
done

Output: (Otherwise idle AMD GX-412TC SOC 4-core @ 966 MHz)
Note: I added 2501 later after looking at the rngd.c code noticing a #define FIPS_RNG_BUFFER_SIZE 2500.

[04/27/21 09:43:21]  Buffer_Size=16535
Entropy rate:  28.05 Kbits/sec averaged over 367 iterations for  580.1 seconds

[04/27/21 09:53:33]  Buffer_Size=8267
Entropy rate:  27.87 Kbits/sec averaged over 464 iterations for  592.1 seconds

[04/27/21 10:03:41]  Buffer_Size=4133
Entropy rate:  27.58 Kbits/sec averaged over 551 iterations for  594.1 seconds

[04/27/21 11:32:19]  Buffer_Size=2501
Entropy rate:  25.95 Kbits/sec averaged over 319 iterations for  596.1 seconds

[04/27/21 10:13:44]  Buffer_Size=2067
Entropy rate:  20.22 Kbits/sec averaged over 585 iterations for  597.1 seconds

[04/27/21 10:23:46]  Buffer_Size=1033
Entropy rate:  16.26 Kbits/sec averaged over 299 iterations for  598.1 seconds

[04/27/21 10:33:47]  Buffer_Size=515
Entropy rate:  15.64 Kbits/sec averaged over 478 iterations for  597.1 seconds

[04/27/21 10:43:48]  Buffer_Size=257
Entropy rate:  7.812 Kbits/sec averaged over 238 iterations for  595.1 seconds

[04/27/21 10:53:48]  Buffer_Size=127
Entropy rate:  3.906 Kbits/sec averaged over 118 iterations for  590.1 seconds

[04/27/21 11:03:48]  Buffer_Size=63
Entropy rate:  1.953 Kbits/sec averaged over 58 iterations for  580.1 seconds

So, rngd is more efficient with larger buffers (as Neil seemed to imply with your question).

It seems 2501 is optimal to my eye, I'll try this script on my faster Proxmox VM to make certain.

[abelbeck]

Full set of tests using the shell script above #36 (comment)

My conclusion is to patch rngd.c with:
(Alternatively add to the rngd command line -O jitter:buffer_size:2501 -O jitter:refill_thresh:2501)

 	[JITTER_OPT_BUF_SZ] = {
 		.key = "buffer_size",
-		.int_val = 16535,
+		.int_val = 2501,
 	},
 	[JITTER_OPT_REFILL] = {
 		.key = "refill_thresh",
-		.int_val = 16535,
+		.int_val = 2501,
 	},

Using 2501 is the smallest buffer size (quickest startup) while maintaining near to the top rngd entropy rate.

Here is the test output:

Proxmox VM 2-core @ 2096 MHz

[04/27/21 11:55:53]  Buffer_Size=16535
Entropy rate:  86.51 Kbits/sec averaged over 336 iterations for  596.1 seconds

[04/27/21 12:05:55]  Buffer_Size=8267
Entropy rate:  65.18 Kbits/sec averaged over 401 iterations for  598.1 seconds

[04/27/21 12:15:57]  Buffer_Size=4133
Entropy rate:  58.58 Kbits/sec averaged over 598 iterations for  598.1 seconds

[04/27/21 12:25:57]  Buffer_Size=2501
Entropy rate:  39.05 Kbits/sec averaged over 597 iterations for  597.1 seconds

[04/27/21 12:35:57]  Buffer_Size=2067
Entropy rate:  19.53 Kbits/sec averaged over 597 iterations for  597.1 seconds

[04/27/21 12:45:57]  Buffer_Size=1033
Entropy rate:  14.65 Kbits/sec averaged over 448 iterations for  597.1 seconds

[04/27/21 12:55:58]  Buffer_Size=515
Entropy rate:  7.811 Kbits/sec averaged over 238 iterations for  595.1 seconds

[04/27/21 13:05:58]  Buffer_Size=257
Entropy rate:  3.905 Kbits/sec averaged over 118 iterations for  590.1 seconds

[04/27/21 13:15:58]  Buffer_Size=127
Entropy rate:  1.953 Kbits/sec averaged over 58 iterations for  580.1 seconds

[04/27/21 13:25:58]  Buffer_Size=63
Entropy rate: 0.9763 Kbits/sec averaged over 28 iterations for  560.1 seconds

AMD GX-412TC SOC 4-core @ 966 MHz

[04/27/21 11:59:38]  Buffer_Size=16535
Entropy rate:  27.91 Kbits/sec averaged over 412 iterations for  580.1 seconds

[04/27/21 12:09:51]  Buffer_Size=8267
Entropy rate:  27.69 Kbits/sec averaged over 445 iterations for  591.2 seconds

[04/27/21 12:19:59]  Buffer_Size=4133
Entropy rate:  27.38 Kbits/sec averaged over 533 iterations for  594.1 seconds

[04/27/21 12:30:02]  Buffer_Size=2501
Entropy rate:  24.08 Kbits/sec averaged over 345 iterations for  596.1 seconds

[04/27/21 12:40:05]  Buffer_Size=2067
Entropy rate:  19.59 Kbits/sec averaged over 596 iterations for  597.1 seconds

[04/27/21 12:50:07]  Buffer_Size=1033
Entropy rate:  16.26 Kbits/sec averaged over 299 iterations for  598.1 seconds

[04/27/21 13:00:08]  Buffer_Size=515
Entropy rate:  15.64 Kbits/sec averaged over 478 iterations for  597.1 seconds

[04/27/21 13:10:09]  Buffer_Size=257
Entropy rate:  7.812 Kbits/sec averaged over 238 iterations for  595.1 seconds

[04/27/21 13:20:09]  Buffer_Size=127
Entropy rate:  3.906 Kbits/sec averaged over 118 iterations for  590.1 seconds

[04/27/21 13:30:09]  Buffer_Size=63
Entropy rate:  1.953 Kbits/sec averaged over 58 iterations for  580.1 seconds

Intel Atom D525 2-core 4-thread @ 1800 MHz

[04/27/21 12:00:46]  Buffer_Size=16535
Entropy rate:  23.32 Kbits/sec averaged over 305 iterations for    577 seconds

[04/27/21 12:11:01]  Buffer_Size=8267
Entropy rate:  23.45 Kbits/sec averaged over 398 iterations for    588 seconds

[04/27/21 12:21:08]  Buffer_Size=4133
Entropy rate:  22.94 Kbits/sec averaged over 484 iterations for    595 seconds

[04/27/21 12:31:12]  Buffer_Size=2501
Entropy rate:  19.46 Kbits/sec averaged over 151 iterations for    600 seconds

[04/27/21 12:41:18]  Buffer_Size=2067
Entropy rate:  22.34 Kbits/sec averaged over 516 iterations for  597.1 seconds

[04/27/21 12:51:20]  Buffer_Size=1033
Entropy rate:  16.72 Kbits/sec averaged over 507 iterations for  598.1 seconds

[04/27/21 13:01:22]  Buffer_Size=515
Entropy rate:  15.64 Kbits/sec averaged over 478 iterations for  597.1 seconds

[04/27/21 13:11:23]  Buffer_Size=257
Entropy rate:  7.812 Kbits/sec averaged over 238 iterations for  595.1 seconds

[04/27/21 13:21:23]  Buffer_Size=127
Entropy rate:  3.906 Kbits/sec averaged over 118 iterations for  590.1 seconds

[04/27/21 13:31:23]  Buffer_Size=63
Entropy rate:  1.953 Kbits/sec averaged over 58 iterations for  580.1 seconds

[smuellerDD]

Am Dienstag, dem 27.04.2021 um 11:54 -0700 schrieb Lonnie Abelbeck:

Full set of tests using the shell script above #36 (comment) My conclusion is to patch `rngd.c` with: (Alternatively add to the `rngd` command line `-O jitter:buffer_size:2501 -O jitter:refill_thresh:2501`) ```         [JITTER_OPT_BUF_SZ] = {                 .key = "buffer_size", -               .int_val = 16535, +               .int_val = 2501,         },         [JITTER_OPT_REFILL] = {                 .key = "refill_thresh", -               .int_val = 16535, +               .int_val = 2501,         }, ``` Using `2501` is the smallest buffer size (quickest startup) while maintaining near to the top `rngd` entropy rate.

Fine with me, if the runtime performance is your criteria. Ciao Stephan

[abelbeck]

@joshuaehill :

In my testing, the raw data variability from the library when compiled with -O2 was dramatically lower than it is when compiled with -O0. The raw data distribution is difficult to predict a priori, so I can't say "Failures will never happen when compiled with -O0", but I can say that I expect that failures will happen much much less frequently when compiled with -O0 than when compiled with -O2.

Let's test using this script:

#/bin/sh

N=0
good=0
failed=0

while true; do
  N=$((N+1))
  rstr="$(timeout 10 /usr/sbin/rngd -t -x hwrng -x rdrand 2>&1)"

  if echo "$rstr" | grep -q '^JITTER rng fails'; then
    failed=$((failed+1))
  elif echo "$rstr" | grep -q '^Enabling JITTER rng'; then
    good=$((good+1))
  fi
  echo "$(date +'[%m/%d/%y %H:%M:%S]')  N=$N  Success=$good  Failed=$failed"
  sleep 5
done

Using unmodified v3.0.2 and rngd with 2501 buffer_size, run continually for 20 hours, yielding a total rngd startups of 12,279 times across 3 test boxes.

No failures, test data below.

Proxmox VM 2-core @ 2096 MHz

[04/27/21 14:17:20]  N=1  Success=1  Failed=0
...
[04/28/21 10:16:15]  N=4792  Success=4792  Failed=0

AMD GX-412TC SOC 4-core @ 966 MHz

[04/27/21 14:16:49]  N=1  Success=1  Failed=0
...
[04/28/21 10:16:37]  N=3656  Success=3656  Failed=0

Intel Atom D525 2-core 4-thread @ 1800 MHz

[04/27/21 14:16:45]  N=1  Success=1  Failed=0
...
[04/28/21 10:17:04]  N=3831  Success=3831  Failed=0

I'm satisfied to put v3.0.2 into production, together with rngd using 2501 buffer_size.

Much thanks to @smuellerDD and @joshuaehill for all your insights here and in jitterentropy-library.

[abelbeck]

@smuellerDD : Reporting 3.4.0 is ~20% slower vs. 3.3.1 (per the test hardware herein)
I suspect the new SHA-3 state integration is the reason.
No issue or problem, just reporting what I see.

[smuellerDD]

Am Montag, 21. März 2022, 17:28:52 CET schrieb Lonnie Abelbeck: Hi Lonnie,

@smuellerDD : Reporting 3.4.0 is ~20% slower vs. 3.3.1 (per the test hardware herein) I suspect the new SHA-3 state integration is the reason. No issue or problem, just reporting what I see.

Thank you very much for the report. I noticed it is a bit slower (albeit not by 20%), but I first wanted to err on the safe side. I am currently performing my analysis whether the performance can be increased by using my "worst" system, a RISC-V platform. Once that completes, I hope I can increase the performance. The changes were due to reviews for NIST compliance who had an issue with a self-feeding hash loop. Hence the update to the state maintenance. Ciao Stephan

[abelbeck]

Thank you very much for the report. I noticed it is a bit slower (albeit not by 20%), but I first wanted to err on the safe side. @smuellerDD

I found the startup (filling rngd buffer of size 2501 and waiting for /proc/sys/kernel/random/entropy_avail greater than 256) to be ~20% slower across all systems.

Interestingly, the ongoing entropy production rate rngd -t ... was reduced much less than 20% for most systems, except the Intel Atom D525 2-core 4-thread @ 1800 MHz system which slowed by 20%.