Why do I get better results with libfm? about fastfm

[merrellb]

I'm seeing some discrepancies between libfm and fastfm with movielens. Before diving into my observations can you confirm the following are equivalent:

/libFM -train ml1m_train.svml -test ml1m_test.svml -task r -dim '1,1,10' -iter 1000 -method mcmc

X_train, y_train = load_svmlight_file("ml1m_train.svml")
X_test, y_test = load_svmlight_file("ml1m_test.svml")

fm = mcmc.FMRegression(n_iter=0, rank=10)
fm.fit_predict(X_train, y_train, X_test)
for i in range(1000):
    y_pred = fm.fit_predict(X_train, y_train, X_test, n_more_iter=1)
    y_pred[y_pred > 5] = 5
    y_pred[y_pred < 1] = 1
    print(i, np.sqrt(mean_squared_error(y_pred, y_test)))

[ibayer]

I don't see a difference, but please check that the init_stdev parameter is the same.
Please have a look at my second comment above, this could explain your observation for small differences.

[arogozhnikov]

Hi, Immanuel!
I was comparing different LibFM implementations (I was testing MCMC for LibFM and FastFM in particular).

Unfortunately, the results of fastFM are not super optimistic
http://arogozhnikov.github.io/2016/02/15/TestingLibFM.html

Then I found this topic, so honestly I wasn't thinking about clipping values.
This trick may give some improvement in regression, but LibFM also easily wins in classification.

Maybe you know a reason? FastFM uses different priors or something else?

[chezou]

@arogozhnikov BTW, you need standardization especially for pyFM coreylynch/pyFM#3 (comment)

[arogozhnikov]

@chezou all the features are dummy (0-1) and table should be sparse. No, for the tests I am running this step neither needed nor possible.

[ibayer]

@arogozhnikov
Great comparison, I have a few suggestions that could make the evaluation even more useful for other people.

1. Provide the exact version of the software that you are testing.
2. You find that libFM is faster then fastFM; I fixed a runtime regression bug in ibayer/fastFM-core@d57a866 , is this still true for the most recent release?
3. Use clipping to make the performance comparison more meaningful (it make quite a difference in some cases).
4. You state for fastFM "supports linux, mac os (though some issues with mac os)" is this still true with the binaries that we now have?
5. Make multiple runs with different seeds to give the reader an idea of the randomness in the results.

As is, I'm not convinced that libFM is faster and performs better then fastFM for MCMC regression. I have done less comprehensive comparisons for MCMC classification but the algorithm / prior should be the same in both libraries. I would be interested to look into it if you can clearly show that libFM dominates fastFM systematically for MCMC classification.

[arogozhnikov]

@ibayer
Thanks for comments.

1. yup, you're right.
   2-3. ok, I'll give a try
2. I don't have clean Mac OS (this was not trivial to install unfortunately) - but I asked a friend to try and seems pip install works fine on Mac OS. (Also, I see now that travis tests MacOS - so I'll remove this remark).
3. This is hard part, will take forever and I don't see random seed in LibFM.
   For smaller tests I can just take different random subsets of data in training. Would this be enough convincing?

[ibayer]

@arogozhnikov It's possible to use a random seed with libFM.
"seed", "integer value, default=None"
https://github.com/srendle/libfm/blob/master/src/libfm/libfm.cpp#L93