hfawaz / cd-diagram

Critical difference diagram with Wilcoxon-Holm post-hoc analysis.

Critical Difference Diagrams

This repository contains the necessary code to generate a critical difference diagram based on the Wilcoxon-Holm method to detect pairwise significance.

Result

By running the python3 main.py you will generate a critical difference diagram with Wilcoxon-Holm post-hoc analysis for the data present in the example.csv file. First the Friedman test is performed to reject the null hypothesis, we then proceed with a post-hoc analysis based on the Wilcoxon-Holm method. We can clearly see how on average clf3 and clf5 were the best algorithms over the 15 datasets. A thick horizontal line groups a set of classifiers that are not significantly different.

Relevant projects

Deep learning for time series classification: a review

In this paper we used the critical difference diagram to compare the recent deep learning models for time series classification where we evaluated 9 different architectures on 85 different datasets from the UCR/UEA archive.

Check out the code!

Deep Neural Network Ensembles for Time Series Classification

In this paper we used the critical difference diagram to show how ensembling hybrid architectures will allow deep learning models to reach even better accuracy when evaluated on 85 different datasets from the UCR/UEA archive.

Check out the code!

Requirements

To run this code you will need the following python packages:

• numpy
• pandas
• matplotlib
• scipy
• networkx

Reference

If you re-use this work, please cite:

@article{IsmailFawaz2018deep,
  Title                    = {Deep learning for time series classification: a review},
  Author                   = {Ismail Fawaz, Hassan and Forestier, Germain and Weber, Jonathan and Idoumghar, Lhassane and Muller, Pierre-Alain},
  journal                  = {Data Mining and Knowledge Discovery},
  Year                     = {2019},
  volume                   = {33},
  number                   = {4},
  pages                    = {917--963},
}

cd-diagram's Issues

for regression models

can we plot cd-diagram for regression models

Can one calculate the critical distance value based on your implementation of Holm's post hoc test?

Hello,
first, thanks for the open code!
I was wondering whether there's a way to calculate critical distance value as is calculated in the original implementation in Orange (by Demšar), where the Nemenyi or Bonferroni-Dunn test was used. Since your implementation only includes adjusted p-values, I assume it is not possible, as Holm's method doesn't have a table of critical values? (The only option is to visually inspect the diagram if that even makes sense?)
Thanks!

Question on output

Hi Hassan,

I've used your code for creating a critical difference diagram and got the following output. Am I misunderstanding something or should that extra line between M4 winner and M4 runner-up not be there?

Strange results obtained for a case study

Hi Hassan,
First of all, thank you for sharing the code.

I've run your code using several combinations of results, for all of them, I obtained interpretable diagrams except for one (I attach the results csv used - see the compressed file).

The thing is that it doesn't make sense, since the clf3 is a way better than the rest of the classifier.

Thank you again for your work.

Best,
David.

example.zip

Friedman Nemenyi Test

it will be more useful if Friedman Nemenyi Test is embedded instead of Wilcoxon_holm. Thank you

README.md Typo

In the example on your README.md example, don't you mean that cfl3 and cfl5 were the best overall (with average ranks around 1.5 and 2, respectively)?

Bonferroni-Dunn post-hoc test

Is this library can also be used for Bonferroni–Dunn post-hoc test?

Not getting cliques for two classifier which are not significantly different

Below is the result when I am trying to compare four different classifiers.

['CoEye' 'TSF' 'cBOSS' 'Mr-SEQL']
CoEye 5.0
Mr-SEQL 61.0
TSF 8.0
cBOSS 10.0
dtype: float64
CoEye 3.279570
TSF 2.559140
cBOSS 2.537634
Mr-SEQL 1.623656
dtype: float64
('CoEye', 'Mr-SEQL', 1.6071230220582003e-11, True)
('Mr-SEQL', 'TSF', 1.5905280411437273e-07, True)
('Mr-SEQL', 'cBOSS', 1.3264426790567074e-06, True)
('CoEye', 'TSF', 1.6515911035945563e-05, True)
('CoEye', 'cBOSS', 4.231130849948383e-05, True)
('TSF', 'cBOSS', 0.4035623277672452, False)
Index(['CoEye', 'TSF', 'cBOSS', 'Mr-SEQL'], dtype='object')
[1, 2]

From the above result, it is clear that classifiers TSF and cBOSS are not significantly different. But I am not getting any clique for them in the CD.
Below is the CD diagram obtained.