Inference.rs uses the new power curve, but the default parameters are from v4 about fsrs-rs

[Expertium]

Also, please rename "FSRS-4.5-dry-run" to "FSRS-4.5 (default parameters)". I know I use the phrase "dry run", but for other people it may not be very clear.

[L-M-Sherlock]

But the benchmark also uses that parameters. It's a loop. We couldn't know the new parameters until we run the benchmark with previous parameters.

[Expertium]

I know, I just find it weird that you decided to implement FSRS-4.5 before finishing the benchmark to obtain the new default parameters. Feel free to close this issue once you finish benchmarking and obtain the new default parameters, just don't forget to tell Dae to update Anki 23.12.

[L-M-Sherlock]

Other questions:

1. Should we re-run the benchmark with the new default parameters?
2. Which value should be chosen for the new default parameters in this case?

[Expertium]

1. Ideally, if you have the time before 23.12 is released, yes.
2. The mode, I think. If this is with what you call "strong filter".

[L-M-Sherlock]

The mode parameters are [0.1, 0.58, 1.64, 5.38, 5.22, 1.14, 0.66, 0.0, 1.69, 0.1, 1.06, 2.12, 0.01, 0.28, 1.46, 0.0, 2.81]

It seems too aggressive (desired retention is 90%):

1:again, 2:hard, 3:good, 4:easy

first rating: 1
rating history: 1,3,3,3,3,3,3,3,3,3,3
interval history: 0d,1d,2d,6d,16d,1.4m,3.4m,8.0m,1.5y,3.1y,6.4y
factor history: 0.0,0.0,2.00,3.00,2.67,2.62,2.45,2.34,2.22,2.13,2.05
difficulty history: 0,7.5,7.5,7.5,7.5,7.5,7.5,7.5,7.5,7.5,7.5

first rating: 2
rating history: 2,3,3,3,3,3,3,3,3,3,3
interval history: 0d,1d,3d,11d,1.2m,3.5m,9.6m,2.0y,5.0y,11.7y,25.9y
factor history: 0.0,0.0,3.00,3.67,3.18,2.97,2.77,2.60,2.45,2.33,2.22
difficulty history: 0,6.4,6.4,6.4,6.4,6.4,6.4,6.4,6.4,6.4,6.4

first rating: 3
rating history: 3,3,3,3,3,3,3,3,3,3,3
interval history: 0d,2d,8d,1.0m,3.6m,11.5m,2.8y,7.7y,19.8y,48.4y,100.0y
factor history: 0.0,0.0,4.00,3.88,3.48,3.19,2.95,2.75,2.58,2.44,2.07
difficulty history: 0,5.2,5.2,5.2,5.2,5.2,5.2,5.2,5.2,5.2,5.2

first rating: 4
rating history: 4,3,3,3,3,3,3,3,3,3,3
interval history: 0d,5d,23d,3.1m,11.5m,3.1y,9.7y,27.6y,73.6y,100.0y,100.0y
factor history: 0.0,0.0,4.60,4.09,3.66,3.34,3.07,2.85,2.67,1.36,1.00
difficulty history: 0,4.1,4.1,4.1,4.1,4.1,4.1,4.1,4.1,4.1,4.1

The preview of current parameters:

1:again, 2:hard, 3:good, 4:easy

first rating: 1
rating history: 1,3,3,3,3,3,3,3,3,3,3
interval history: 0d,1d,2d,5d,13d,1.0m,2.3m,5.0m,10.3m,1.7y,3.2y
factor history: 0.0,0.0,2.00,2.50,2.60,2.38,2.26,2.16,2.05,1.97,1.90
difficulty history: 0,7.1,7.0,6.9,6.8,6.7,6.6,6.5,6.4,6.3,6.3

first rating: 2
rating history: 2,3,3,3,3,3,3,3,3,3,3
interval history: 0d,1d,3d,9d,25d,2.1m,4.9m,10.9m,1.9y,3.8y,7.2y
factor history: 0.0,0.0,3.00,3.00,2.78,2.52,2.35,2.22,2.09,2.00,1.91
difficulty history: 0,6.1,6.1,6.0,5.9,5.9,5.8,5.8,5.8,5.7,5.7

first rating: 3
rating history: 3,3,3,3,3,3,3,3,3,3,3
interval history: 0d,1d,4d,13d,1.3m,3.5m,8.5m,1.6y,3.4y,7.0y,13.5y
factor history: 0.0,0.0,4.00,3.25,3.00,2.67,2.46,2.29,2.15,2.03,1.93
difficulty history: 0,5.1,5.1,5.1,5.1,5.1,5.1,5.1,5.1,5.1,5.1

first rating: 4
rating history: 4,3,3,3,3,3,3,3,3,3,3
interval history: 0d,6d,21d,2.2m,6.1m,1.3y,3.0y,6.5y,13.3y,26.0y,48.2y
factor history: 0.0,0.0,3.50,3.10,2.80,2.54,2.34,2.18,2.05,1.95,1.86
difficulty history: 0,4.1,4.1,4.2,4.3,4.3,4.4,4.4,4.4,4.5,4.5

[Expertium]

You could do two dry runs (btw, the benchmark repo should include a dry run both for v4 and for 4.5) with two different sets of parameters, mode and median, to see which one results in lower RMSE.

[Expertium]

Also, if it turns out the mode parameters are better (lower RMSE), then the benchmark entries for unoptimized v4 and 4.5 should use mode parameters; if it turns out the median parameters are better, then the benchmark entries should use median parameters.

[L-M-Sherlock]

OK. I will do it tomorrow.

[dae]

@L-M-Sherlock is this critical for 23.12, or can it wait until 23.12.1?

[L-M-Sherlock]

It's not critical.

[L-M-Sherlock]

The dry run result of current default parameters:

Algorithm	Log Loss	RMSE	RMSE(bins)
FSRS-4.5-dry-run	0.3901	0.3310	0.1044

The result of the mode parameters:

Algorithm	Log Loss	RMSE	RMSE(bins)
FSRS-4.5-dry-run	0.4190	0.3489	0.1401

The result shows that the current default parameters are better.

[Expertium]

But the current default parameters are median parameters from v4. We need median parameters from 4.5.

[L-M-Sherlock]

The median parameters are: [0.5614, 1.2546, 3.5878, 7.9731, 5.1043, 1.1303, 0.823, 0.0465, 1.629, 0.135, 1.0045, 2.132, 0.0839, 0.3204, 1.3547, 0.219, 2.7849]

The result of the median parameters:

Algorithm	Log Loss	RMSE	RMSE(bins)
FSRS-4.5-dry-run	0.3856	0.3257	0.0947

1:again, 2:hard, 3:good, 4:easy

first rating: 1
rating history: 1,3,3,3,3,3,3,3,3,3,3
interval history: 0d,1d,3d,8d,20d,1.6m,3.7m,8.1m,1.4y,2.7y,5.3y
factor history: 0.0,0.0,3.00,2.67,2.50,2.45,2.27,2.18,2.08,1.99,1.92
difficulty history: 0,7.4,7.3,7.2,7.1,7.0,6.9,6.8,6.7,6.6,6.6

first rating: 2
rating history: 2,3,3,3,3,3,3,3,3,3,3
interval history: 0d,1d,4d,12d,1.2m,3.1m,7.5m,1.4y,3.1y,6.4y,12.5y
factor history: 0.0,0.0,4.00,3.00,2.92,2.63,2.46,2.30,2.17,2.06,1.97
difficulty history: 0,6.2,6.2,6.1,6.1,6.0,6.0,6.0,5.9,5.9,5.8

first rating: 3
rating history: 3,3,3,3,3,3,3,3,3,3,3
interval history: 0d,4d,14d,1.5m,4.4m,11.5m,2.3y,5.3y,11.3y,23.0y,44.6y
factor history: 0.0,0.0,3.50,3.21,2.91,2.64,2.44,2.28,2.14,2.03,1.94
difficulty history: 0,5.1,5.1,5.1,5.1,5.1,5.1,5.1,5.1,5.1,5.1

first rating: 4
rating history: 4,3,3,3,3,3,3,3,3,3,3
interval history: 0d,8d,1.0m,3.5m,10.4m,2.3y,5.7y,13.2y,28.4y,57.9y,100.0y
factor history: 0.0,0.0,3.88,3.35,2.99,2.71,2.48,2.30,2.16,2.04,1.73
difficulty history: 0,4.0,4.0,4.1,4.1,4.2,4.2,4.3,4.3,4.3,4.4

[L-M-Sherlock]

I will update the default parameters this week and re-run the benchmark.

[Expertium]

But then how did you get a lower number here?

[L-M-Sherlock]

They are weighted by different rules.