Performance on the MVTec dataset about upd_study

[iolag]

Hey,

We did not evaluate on MVTec. As most methods did not work well out of the box, given that they were optimized for the data they were developed for, we tried to do some optimization ourselves especially with methods that completely failed on the datasets we used. I especially remember the case of CFLOW-AD, which performed terribly on MRIs even though it was one of the best performers on MVTec at the time. In that case, I had to change the Resnet layers that are used for feature extraction in order to get decent performance. This probably had a negative effect on its MVTec performance.

On the other hand, RD worked really great with the original hyperparameters and we kept it largely unchanged, which explains the results you get. I'm not sure why PADIM isn't performing great for you as we used the original implementation, if I remember correctly.

As for the rest of the methods, I would not expect their performance to translate to MVTec as they were optimized on very different types of data.

[cuicathy]

Hi iolag,

Thanks very much for your response. Do you remember what parameter did you tune? And how did you tune them (e.g., grid search)?

About the PADIM, I did not run it before last week since I only compared the models that required training for epochs last time. For the PADIM, it performed about AUC 0.96 (anomaly detection) on the hazelnut class of the MVTec.

[iolag]

Hi,

sorry for the late response.

For VAE, I remember using grid search on a lot of hyper-parameters. On others, I empirically changed certain parameters, mostly based on things I noticed during early experiments. I go into a bit more detail on which ones I changed compared to the official implementations in the end of my Masters thesis. There, only methods that I used before the journal revisions are mentioned, but the ones missing I mostly implemented unchanged.

[cuicathy]

Thanks for sharing. I will read it!