[BUG] Huge difference between Rust optimizer and Python optimizer about fsrs-rs

[L-M-Sherlock]

@asukaminato0721 and I will deal with it.

[L-M-Sherlock]

I forget to remove outliers in the Rust optimizer:

        def remove_outliers(group: pd.DataFrame) -> pd.DataFrame:
            grouped_group = (
                group.groupby(by=["r_history", "delta_t"], group_keys=False)
                .agg({"y": ["mean", "count"]})
                .reset_index()
            )
            sort_index = grouped_group.sort_values(
                by=[("y", "count"), "delta_t"], ascending=[True, False]
            ).index

            total = sum(grouped_group[("y", "count")])
            has_been_removed = 0
            for i in sort_index:
                count = grouped_group.loc[i, ("y", "count")]
                if has_been_removed + count >= total * 0.05:
                    break
                has_been_removed += count
            group = group[
                group["delta_t"].isin(
                    grouped_group[grouped_group[("y", "count")] >= count]["delta_t"]
                )
            ]
            return group

        df[df["i"] == 2] = (
            df[df["i"] == 2]
            .groupby(by=["r_history", "t_history"], as_index=False, group_keys=False)
            .apply(remove_outliers)
        )
        df.dropna(inplace=True)

        def remove_non_continuous_rows(group):
            discontinuity = group["i"].diff().fillna(1).ne(1)
            if not discontinuity.any():
                return group
            else:
                first_non_continuous_index = discontinuity.idxmax()
                return group.loc[: first_non_continuous_index - 1]

        df = df.groupby("card_id", as_index=False, group_keys=False).progress_apply(
            remove_non_continuous_rows
        )

[L-M-Sherlock]

fsrs-rs:

fsrs-py:

[L-M-Sherlock]

To increase num_epochs could reduce the errors. But it also will slow down the the optimization.

[L-M-Sherlock]

The left is generated by Anki. The right is generated by python optimizer.

[Expertium]

To increase num_epochs could reduce the errors. But it also will slow down the the optimization.

By the way, how many epochs does the optimizer in the beta version use? Also, does it use splits, with averaging of the parameters afterwards?
And I don't think that the optimization becoming 2 or even 3 times slower is that important. Currently, the optimizer is blazingly fast, and even on a large collection optimization takes a minute or so. I don't think users will be very upset if the optimization takes 2-3 minutes instead of 1 minute.

[L-M-Sherlock]

By the way, how many epochs does the optimizer in the beta version use? Also, does it use splits, with averaging of the parameters afterwards?

It uses 16 epochs and doesn't have splits because the framework doesn't support splits.

[dae]

Are we confident that the differences are due to the number of epochs, and not due to a difference in revlog filtering?

…

On Mon, 25 Sep 2023 at 9:51 pm, Jarrett Ye ***@***.***> wrote: By the way, how many epochs does the optimizer in the beta version use? Also, does it use splits, with averaging of the parameters afterwards? It uses 16 epochs and doesn't have splits because the framework doesn't support splits. — Reply to this email directly, view it on GitHub <#78 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABMCPSUAIGJF72L6N6THMTX4FV3HANCNFSM6AAAAAA5FWSZ3M> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[user1823]

Why are we talking about num_epochs here? Wouldn't this problem be solved just by adding the outlier filter to the rust optimizer?

[Expertium]

So the code related to removing outliers will be added in the next release?

[L-M-Sherlock]

I implement the outlier filter, but the first four weights are still very different from the Python optimizer. So it's not caused by outlier filter. By the way, I think the weights generated by the Python optimizer doesn't fit the forgetting curve well:

A smaller value of stability would be better:

Maybe RMSE is not a good loss function here. I plan to use log loss.

[Expertium]

Maybe RMSE is not a good loss function here. I plan to use log loss.

I would recommend running the benchmark with both RMSE and logloss to determine whether there is a difference in the final RMSE.

[user1823]

I implement the outlier filter, but the first four weights are still very different from the Python optimizer. So it's not caused by outlier filter. By the way, I think the weights generated by the Python optimizer doesn't fit the forgetting curve well:

The supposed poor fitting of weights produced by the Python optimizer is definitely worth investigating. But, for now, the main focus should be on finding out why the first four weights generated by the Rust and the Python optimizer very different.

[Expertium]

Maybe RMSE is not a good loss function here. I plan to use log loss.

I know this issue is closed, but I'm curious, did you end up testing RMSE vs logloss in pretrain? If so, which one is better?

[L-M-Sherlock]

I tested it. The log loss is more robust than RMSE in pretrain.