hfawaz / dl-4-tsc

Deep Learning for Time Series Classification

Deep Learning for Time Series Classification

This is the companion repository for our paper titled "Deep learning for time series classification: a review" published in Data Mining and Knowledge Discovery, also available on ArXiv.

Docker

Assuming you have docker installed. You can now use the docker image provided here.

Access the docker container via:

docker run --name somename --gpus all  -idt hassanfawaz/dl-4-tsc:0.3
docker exec -it somename bash

To run you will need to manually download the UCR archive into /dl-4-tsc/archives/:

cd /dl-4-tsc/archives
wget https://www.cs.ucr.edu/~eamonn/time_series_data_2018/UCRArchive_2018.zip
unzip -P $password UCRArchive_2018.zip

The password can be found here.

Now that you have the data and the code you can just run the code.

cd /dl-4-tsc
python -m main UCRArchive_2018 Coffee fcn _itr_0

You can also try and install with pip on your env.

Data

The data used in this project comes from two sources:

• The UCR/UEA archive, which contains the 85 univariate time series datasets.
• The MTS archive, which contains the 13 multivariate time series datasets.

Code

The code is divided as follows:

• The main.py python file contains the necessary code to run an experiement.
• The utils folder contains the necessary functions to read the datasets and visualize the plots.
• The classifiers folder contains nine python files one for each deep neural network tested in our paper.

To run a model on one dataset you should issue the following command:

python3 main.py TSC Coffee fcn _itr_8

which means we are launching the fcn model on the univariate UCR archive for the Coffee dataset (see constants.py for a list of possible options).

Prerequisites

All python packages needed are listed in pip-requirements.txt file and can be installed simply using the pip command. The code now uses Tensorflow 2.0. The results in the paper were generated using the Tensorflow 1.14 implementation which can be found here. Using Tensorflow 2.0 should give the same results.
Now InceptionTime is included in the mix, feel free to send a pull request to add another classifier.

• numpy
• pandas
• sklearn
• scipy
• matplotlib
• tensorflow-gpu
• keras
• h5py
• keras_contrib

Results

I added the results on the 128 datasets from the UCR archive 2018. Our results in the paper showed that a deep residual network architecture performs best for the time series classification task.

The following table contains the averaged accuracy over 10 runs of each implemented model on the UCR/UEA archive, with the standard deviation between parentheses.

Datasets	MLP	FCN	ResNet	Encoder	MCNN	t-LeNet	MCDCNN	Time-CNN	TWIESN
50words	68.4(7.1)	62.7(6.1)	74.0(1.5)	72.3(1.0)	22.0(24.3)	12.5(0.0)	58.9(5.3)	62.1(1.0)	49.6(2.6)
Adiac	39.7(1.9)	84.4(0.7)	82.9(0.6)	48.4(2.5)	2.2(0.6)	2.0(0.0)	61.0(8.7)	37.9(2.0)	41.6(4.5)
ArrowHead	77.8(1.2)	84.3(1.5)	84.5(1.2)	80.4(2.9)	33.9(4.7)	30.3(0.0)	68.5(6.7)	72.3(2.6)	65.9(9.4)
Beef	72.0(2.8)	69.7(4.0)	75.3(4.2)	64.3(5.0)	20.0(0.0)	20.0(0.0)	56.3(7.8)	76.3(1.1)	53.7(14.9)
BeetleFly	87.0(2.6)	86.0(9.7)	85.0(2.4)	74.5(7.6)	50.0(0.0)	50.0(0.0)	58.0(9.2)	89.0(3.2)	73.0(7.9)
BirdChicken	77.5(3.5)	95.5(3.7)	88.5(5.3)	66.5(5.8)	50.0(0.0)	50.0(0.0)	58.0(10.3)	60.5(9.0)	74.0(15.6)
CBF	87.2(0.7)	99.4(0.1)	99.5(0.3)	94.7(1.2)	33.2(0.1)	33.2(0.1)	82.0(20.5)	95.7(1.0)	89.0(4.9)
Car	76.7(2.6)	90.5(1.4)	92.5(1.4)	75.8(2.0)	24.0(2.7)	31.7(0.0)	73.0(3.0)	78.2(1.2)	78.3(4.0)
ChlorineConcentration	80.2(1.1)	81.4(0.9)	84.4(1.0)	57.3(1.1)	53.3(0.0)	53.3(0.0)	64.3(3.8)	60.0(0.8)	55.3(0.3)
CinC_ECG_torso	84.0(1.0)	82.4(1.2)	82.6(2.4)	91.1(2.7)	38.1(28.0)	25.0(0.1)	73.6(15.2)	74.5(4.9)	30.0(2.9)
Coffee	99.6(1.1)	100.0(0.0)	100.0(0.0)	97.9(1.8)	51.4(3.5)	53.6(0.0)	98.2(2.5)	99.6(1.1)	97.1(2.8)
Computers	56.3(1.6)	82.2(1.0)	81.5(1.2)	57.4(2.2)	52.2(4.8)	50.0(0.0)	55.9(3.3)	54.8(1.5)	62.9(4.1)
Cricket_X	59.1(1.1)	79.2(0.7)	79.1(0.6)	69.4(1.6)	18.9(23.8)	7.4(0.0)	49.5(5.3)	55.2(2.9)	62.2(2.1)
Cricket_Y	60.0(0.8)	78.7(1.2)	80.3(0.8)	67.5(1.0)	18.4(22.0)	8.5(0.0)	49.7(4.3)	57.0(2.4)	65.6(1.3)
Cricket_Z	61.7(0.8)	81.1(1.0)	81.2(1.4)	69.2(1.0)	18.3(24.4)	6.2(0.0)	49.8(3.6)	48.8(2.8)	62.2(2.3)
DiatomSizeReduction	91.0(1.4)	31.3(3.6)	30.1(0.2)	91.3(1.8)	30.1(0.7)	30.1(0.0)	70.3(28.9)	95.4(0.7)	88.0(6.6)
DistalPhalanxOutlineAgeGroup	65.7(1.1)	71.0(1.3)	71.7(1.3)	73.7(1.6)	46.8(0.0)	44.6(2.3)	74.4(2.2)	75.2(1.4)	71.0(2.1)
DistalPhalanxOutlineCorrect	72.6(1.3)	76.0(1.5)	77.1(1.0)	74.1(1.4)	58.3(0.0)	58.3(0.0)	75.3(1.8)	75.9(2.0)	71.3(1.0)
DistalPhalanxTW	61.7(1.3)	69.0(2.1)	66.5(1.6)	68.8(1.6)	30.2(0.0)	28.3(0.7)	67.7(1.8)	67.3(2.8)	60.9(3.0)
ECG200	91.6(0.7)	88.9(1.0)	87.4(1.9)	92.3(1.1)	64.0(0.0)	64.0(0.0)	83.3(3.9)	81.4(1.3)	84.2(5.1)
ECG5000	92.9(0.1)	94.0(0.1)	93.4(0.2)	94.0(0.2)	61.8(10.9)	58.4(0.0)	93.7(0.6)	92.8(0.2)	91.9(0.2)
ECGFiveDays	97.0(0.5)	98.7(0.3)	97.5(1.9)	98.2(0.7)	49.9(0.3)	49.7(0.0)	76.2(13.4)	88.2(1.8)	69.8(14.1)
Earthquakes	71.7(1.3)	72.7(1.7)	71.2(2.0)	74.8(0.7)	74.8(0.0)	74.8(0.0)	74.9(0.2)	70.0(1.9)	74.8(0.0)
ElectricDevices	59.2(1.1)	70.2(1.2)	72.9(0.9)	67.4(1.1)	33.6(19.8)	24.2(0.0)	64.4(1.2)	68.1(1.0)	60.7(0.7)
FISH	84.8(0.8)	95.8(0.6)	97.9(0.8)	86.6(0.9)	13.4(1.3)	12.6(0.0)	75.8(3.9)	84.9(0.5)	87.5(3.4)
FaceAll	79.3(1.1)	94.5(0.9)	83.9(2.0)	79.3(0.8)	17.0(19.5)	8.0(0.0)	71.7(2.3)	76.8(1.1)	65.7(2.5)
FaceFour	84.0(1.4)	92.8(0.9)	95.5(0.0)	81.5(2.6)	26.8(5.7)	29.5(0.0)	71.2(13.5)	90.6(1.1)	85.5(6.2)
FacesUCR	83.3(0.3)	94.6(0.2)	95.5(0.4)	87.4(0.4)	15.3(2.7)	14.3(0.0)	75.6(5.1)	86.9(0.7)	64.4(2.0)
FordA	73.0(0.4)	90.4(0.2)	92.0(0.4)	92.3(0.3)	51.3(0.0)	51.0(0.8)	79.5(2.6)	88.1(0.7)	52.8(2.1)
FordB	60.3(0.3)	87.8(0.6)	91.3(0.3)	89.0(0.5)	49.8(1.2)	51.2(0.0)	53.3(2.9)	80.6(1.5)	50.3(1.2)
Gun_Point	92.7(1.1)	100.0(0.0)	99.1(0.7)	93.6(3.2)	51.3(3.9)	49.3(0.0)	86.7(9.6)	93.2(1.9)	96.1(2.3)
Ham	69.1(1.4)	71.8(1.4)	75.7(2.7)	72.7(1.2)	50.6(1.4)	51.4(0.0)	73.3(4.2)	71.1(2.0)	72.3(6.3)
HandOutlines	91.8(0.5)	80.6(7.9)	91.1(1.4)	89.9(2.3)	64.1(0.0)	64.1(0.0)	90.9(0.6)	88.8(1.2)	66.0(0.7)
Haptics	43.3(1.4)	48.0(2.4)	51.9(1.2)	42.7(1.6)	20.9(3.5)	20.8(0.0)	40.4(3.3)	36.6(2.4)	40.4(4.5)
Herring	52.8(3.9)	60.8(7.7)	61.9(3.8)	58.6(4.8)	59.4(0.0)	59.4(0.0)	60.0(5.2)	53.9(1.7)	59.1(6.5)
InlineSkate	33.7(1.0)	33.9(0.8)	37.3(0.9)	29.2(0.9)	16.7(1.6)	16.5(1.1)	21.5(2.2)	28.7(1.2)	33.0(6.8)
InsectWingbeatSound	60.7(0.4)	39.3(0.6)	50.7(0.9)	63.3(0.6)	15.8(14.2)	9.1(0.0)	58.3(2.6)	58.3(0.6)	43.7(2.0)
ItalyPowerDemand	95.4(0.2)	96.1(0.3)	96.3(0.4)	96.5(0.5)	50.0(0.2)	49.9(0.0)	95.5(1.9)	95.5(0.4)	88.0(2.2)
LargeKitchenAppliances	47.3(0.6)	90.2(0.4)	90.0(0.5)	61.9(2.6)	41.0(16.5)	33.3(0.0)	43.4(2.8)	66.6(5.0)	77.9(1.8)
Lighting2	67.0(2.1)	73.9(1.4)	77.0(1.7)	69.2(4.6)	55.7(5.2)	54.1(0.0)	63.0(5.9)	63.6(2.5)	70.3(4.1)
Lighting7	63.0(1.7)	82.7(2.3)	84.5(2.0)	62.5(2.3)	31.0(11.3)	26.0(0.0)	53.4(5.9)	65.1(3.3)	66.4(6.6)
MALLAT	91.8(0.6)	96.7(0.9)	97.2(0.3)	87.6(2.0)	13.5(3.7)	12.3(0.1)	90.1(5.7)	92.0(0.7)	59.6(9.8)
Meat	89.7(1.7)	85.3(6.9)	96.8(2.5)	74.2(11.0)	33.3(0.0)	33.3(0.0)	70.5(8.8)	90.2(1.8)	96.8(2.0)
MedicalImages	72.1(0.7)	77.9(0.4)	77.0(0.7)	73.4(1.5)	51.4(0.0)	51.4(0.0)	64.0(1.4)	67.6(1.1)	64.9(2.7)
MiddlePhalanxOutlineAgeGroup	53.1(1.8)	55.3(1.8)	56.9(2.1)	57.9(2.9)	18.8(0.0)	57.1(0.0)	58.5(3.8)	56.6(1.5)	58.1(2.6)
MiddlePhalanxOutlineCorrect	77.0(1.1)	80.1(1.0)	80.9(1.2)	76.1(2.3)	57.0(0.0)	57.0(0.0)	81.1(1.6)	76.6(1.3)	74.4(2.3)
MiddlePhalanxTW	53.4(1.6)	51.2(1.8)	48.4(2.0)	59.2(1.0)	27.3(0.0)	28.6(0.0)	58.1(2.4)	54.9(1.7)	53.9(2.9)
MoteStrain	85.8(0.9)	93.7(0.5)	92.8(0.5)	84.0(1.0)	50.8(4.0)	53.9(0.0)	76.5(14.4)	88.2(0.9)	78.5(4.2)
NonInvasiveFatalECG_Thorax1	91.6(0.4)	95.6(0.3)	94.5(0.3)	91.6(0.4)	16.1(29.3)	2.9(0.0)	90.5(1.2)	86.5(0.5)	49.4(4.2)
NonInvasiveFatalECG_Thorax2	91.7(0.3)	95.3(0.3)	94.6(0.3)	93.2(0.9)	16.0(29.2)	2.9(0.0)	91.5(1.5)	89.8(0.3)	52.5(3.2)
OSULeaf	55.7(1.0)	97.7(0.9)	97.9(0.8)	57.6(2.0)	24.3(12.8)	18.2(0.0)	37.8(4.6)	46.2(2.7)	59.5(5.4)
OliveOil	66.7(3.8)	72.3(16.6)	83.0(8.5)	40.0(0.0)	38.0(4.2)	38.0(4.2)	40.0(0.0)	40.0(0.0)	79.0(6.1)
PhalangesOutlinesCorrect	73.5(2.1)	82.0(0.5)	83.9(1.2)	76.7(1.4)	61.3(0.0)	61.3(0.0)	80.3(1.1)	77.1(4.7)	65.4(0.4)
Phoneme	9.6(0.3)	32.5(0.5)	33.4(0.7)	17.2(0.8)	13.2(4.0)	11.3(0.0)	13.0(1.0)	9.5(0.3)	12.8(1.4)
Plane	97.8(0.5)	100.0(0.0)	100.0(0.0)	97.6(0.8)	13.0(4.5)	13.4(1.4)	96.5(3.2)	96.5(1.4)	100.0(0.0)
ProximalPhalanxOutlineAgeGroup	85.6(0.5)	83.1(1.3)	85.3(0.8)	84.4(1.3)	48.8(0.0)	48.8(0.0)	83.8(0.8)	82.8(1.6)	84.4(0.5)
ProximalPhalanxOutlineCorrect	73.3(1.8)	90.3(0.7)	92.1(0.6)	79.1(1.8)	68.4(0.0)	68.4(0.0)	87.3(1.8)	81.2(2.6)	82.1(0.9)
ProximalPhalanxTW	76.7(0.7)	76.7(0.9)	78.0(1.7)	81.2(1.1)	35.1(0.0)	34.6(1.0)	79.7(1.3)	78.3(1.2)	78.1(0.7)
RefrigerationDevices	37.9(2.1)	50.8(1.0)	52.5(2.5)	48.8(1.9)	33.3(0.0)	33.3(0.0)	36.9(3.8)	43.9(1.0)	50.1(1.5)
ScreenType	40.3(1.0)	62.5(1.6)	62.2(1.4)	38.3(2.2)	34.1(2.4)	33.3(0.0)	42.7(1.8)	38.9(0.9)	43.1(4.7)
ShapeletSim	50.3(3.1)	72.4(5.6)	77.9(15.0)	53.0(4.7)	50.0(0.0)	50.0(0.0)	50.7(4.1)	50.0(1.3)	61.7(10.2)
ShapesAll	77.1(0.5)	89.5(0.4)	92.1(0.4)	75.8(0.9)	13.2(24.3)	1.7(0.0)	61.3(5.3)	61.9(0.9)	62.9(2.6)
SmallKitchenAppliances	37.1(1.9)	78.3(1.3)	78.6(0.8)	59.6(1.8)	36.9(11.3)	33.3(0.0)	48.5(3.6)	61.5(2.7)	65.6(1.9)
SonyAIBORobotSurface	67.2(1.3)	96.0(0.7)	95.8(1.3)	74.3(1.9)	44.3(4.5)	42.9(0.0)	65.3(10.9)	68.7(2.3)	63.8(9.9)
SonyAIBORobotSurfaceII	83.4(0.7)	97.9(0.5)	97.8(0.5)	83.9(1.0)	59.4(7.4)	61.7(0.0)	77.4(6.7)	84.1(1.7)	69.7(4.3)
StarLightCurves	94.9(0.2)	96.1(0.9)	97.2(0.3)	95.7(0.5)	65.4(16.1)	57.7(0.0)	93.9(1.2)	92.6(0.2)	85.0(0.2)
Strawberry	96.1(0.5)	97.2(0.3)	98.1(0.4)	94.6(0.9)	64.3(0.0)	64.3(0.0)	95.6(0.6)	95.9(0.3)	89.5(2.0)
SwedishLeaf	85.1(0.5)	96.9(0.5)	95.6(0.4)	93.0(1.1)	11.8(13.2)	6.5(0.4)	84.6(3.6)	88.4(1.1)	82.5(1.4)
Symbols	83.2(1.0)	95.5(1.0)	90.6(2.3)	82.1(1.9)	22.6(16.9)	17.4(0.0)	75.6(11.5)	81.0(0.7)	75.0(8.8)
ToeSegmentation1	58.3(0.9)	96.1(0.5)	96.3(0.6)	65.9(2.6)	50.5(2.7)	52.6(0.0)	49.0(2.5)	59.5(2.2)	86.5(3.2)
ToeSegmentation2	74.5(1.9)	88.0(3.3)	90.6(1.7)	79.5(2.8)	63.2(30.9)	81.5(0.0)	44.3(15.2)	73.8(2.8)	84.2(4.6)
Trace	80.7(0.7)	100.0(0.0)	100.0(0.0)	96.0(1.8)	35.4(27.7)	24.0(0.0)	86.3(5.4)	95.0(2.5)	95.9(1.9)
TwoLeadECG	76.2(1.3)	100.0(0.0)	100.0(0.0)	86.3(2.6)	50.0(0.0)	50.0(0.0)	76.0(16.8)	87.2(2.1)	85.2(11.5)
Two_Patterns	94.6(0.3)	87.1(0.3)	100.0(0.0)	100.0(0.0)	40.3(31.1)	25.9(0.0)	97.8(0.6)	99.2(0.3)	87.1(1.1)
UWaveGestureLibraryAll	95.5(0.2)	81.7(0.3)	86.0(0.4)	95.4(0.1)	28.9(34.7)	12.8(0.2)	92.9(1.1)	91.8(0.4)	55.6(2.5)
Wine	56.5(7.1)	58.7(8.3)	74.4(8.5)	50.0(0.0)	50.0(0.0)	50.0(0.0)	50.0(0.0)	51.7(5.1)	75.9(9.1)
WordsSynonyms	59.8(0.8)	56.4(1.2)	62.2(1.5)	61.3(0.9)	28.4(13.6)	21.9(0.0)	46.3(6.1)	56.6(0.8)	49.0(3.0)
Worms	45.7(2.4)	76.5(2.2)	79.1(2.5)	57.1(3.7)	42.9(0.0)	42.9(0.0)	42.6(5.5)	38.3(2.5)	46.6(4.5)
WormsTwoClass	60.1(1.5)	72.6(2.7)	74.7(3.3)	63.9(4.4)	57.1(0.0)	55.7(4.5)	57.0(1.9)	53.8(2.6)	57.0(2.3)
synthetic_control	97.6(0.4)	98.5(0.3)	99.8(0.2)	99.6(0.3)	29.8(27.8)	16.7(0.0)	98.3(1.2)	99.0(0.4)	87.4(1.6)
uWaveGestureLibrary_X	76.7(0.3)	75.4(0.4)	78.0(0.4)	78.6(0.4)	18.9(21.3)	12.5(0.4)	71.1(1.5)	71.1(1.1)	60.6(1.5)
uWaveGestureLibrary_Y	69.8(0.2)	63.9(0.6)	67.0(0.7)	69.6(0.6)	23.7(24.0)	12.1(0.0)	63.6(1.2)	62.6(0.7)	52.0(2.1)
uWaveGestureLibrary_Z	69.7(0.2)	72.6(0.5)	75.0(0.4)	71.1(0.5)	18.0(18.4)	12.1(0.0)	65.0(1.8)	64.2(0.9)	56.5(2.0)
wafer	99.6(0.0)	99.7(0.0)	99.9(0.1)	99.6(0.0)	91.3(4.4)	89.2(0.0)	99.2(0.3)	96.1(0.1)	91.4(0.5)
yoga	85.5(0.4)	83.9(0.7)	87.0(0.9)	82.0(0.6)	53.6(0.0)	53.6(0.0)	76.2(3.9)	78.1(0.7)	60.7(1.9)
Average_Rank	4.611765	2.682353	1.994118	3.682353	8.017647	8.417647	5.376471	4.970588	5.247059
Wins	4	18	41	10	0	0	3	4	1

The following table contains the averaged accuracy over 10 runs of each implemented model on the MTS archive, with the standard deviation between parentheses.

Datasets	MLP	FCN	ResNet	Encoder	MCNN	t-LeNet	MCDCNN	Time-CNN	TWIESN
AUSLAN	93.3(0.5)	97.5(0.4)	97.4(0.3)	93.8(0.5)	1.1(0.0)	1.1(0.0)	85.4(2.7)	72.6(3.5)	72.4(1.6)
ArabicDigits	96.9(0.2)	99.4(0.1)	99.6(0.1)	98.1(0.1)	10.0(0.0)	10.0(0.0)	95.9(0.2)	95.8(0.3)	85.3(1.4)
CMUsubject16	60.0(16.9)	100.0(0.0)	99.7(1.1)	98.3(2.4)	53.1(4.4)	51.0(5.3)	51.4(5.0)	97.6(1.7)	89.3(6.8)
CharacterTrajectories	96.9(0.2)	99.0(0.1)	99.0(0.2)	97.1(0.2)	5.4(0.8)	6.7(0.0)	93.8(1.7)	96.0(0.8)	92.0(1.3)
ECG	74.8(16.2)	87.2(1.2)	86.7(1.3)	87.2(0.8)	67.0(0.0)	67.0(0.0)	50.0(17.9)	84.1(1.7)	73.7(2.3)
JapaneseVowels	97.6(0.2)	99.3(0.2)	99.2(0.3)	97.6(0.6)	9.2(2.5)	23.8(0.0)	94.4(1.4)	95.6(1.0)	96.5(0.7)
KickvsPunch	61.0(12.9)	54.0(13.5)	51.0(8.8)	61.0(9.9)	54.0(9.7)	50.0(10.5)	56.0(8.4)	62.0(6.3)	67.0(14.2)
Libras	78.0(1.0)	96.4(0.7)	95.4(1.1)	78.3(0.9)	6.7(0.0)	6.7(0.0)	65.1(3.9)	63.7(3.3)	79.4(1.3)
NetFlow	55.0(26.1)	89.1(0.4)	62.7(23.4)	77.7(0.5)	77.9(0.0)	72.3(17.6)	63.0(18.2)	89.0(0.9)	94.5(0.4)
UWave	90.1(0.3)	93.4(0.3)	92.6(0.4)	90.8(0.4)	12.5(0.0)	12.5(0.0)	84.5(1.6)	85.9(0.7)	75.4(6.3)
Wafer	89.4(0.0)	98.2(0.5)	98.9(0.4)	98.6(0.2)	89.4(0.0)	89.4(0.0)	65.8(38.1)	94.8(2.1)	94.9(0.6)
WalkvsRun	70.0(15.8)	100.0(0.0)	100.0(0.0)	100.0(0.0)	75.0(0.0)	60.0(24.2)	45.0(25.8)	100.0(0.0)	94.4(9.1)
Average_Rank	5.208333	2.000000	2.875000	3.041667	7.583333	8.000000	6.833333	4.625000	4.833333
Wins	0	5	3	0	0	0	0	0	2

These results should give an insight of deep learning for TSC therefore encouraging researchers to consider the DNNs as robust classifiers for time series data.

If you would like to generate the critical difference diagrams using Wilcoxon Signed Rank test with Holm's alpha correction, check out the cd-diagram repository.

Reference

If you re-use this work, please cite:

@article{IsmailFawaz2018deep,
  Title                    = {Deep learning for time series classification: a review},
  Author                   = {Ismail Fawaz, Hassan and Forestier, Germain and Weber, Jonathan and Idoumghar, Lhassane and Muller, Pierre-Alain},
  journal                  = {Data Mining and Knowledge Discovery},
  Year                     = {2019},
  volume                   = {33},
  number                   = {4},
  pages                    = {917--963},
}

Acknowledgement

We would like to thank the providers of the UCR/UEA archive. We would also like to thank NVIDIA Corporation for the Quadro P6000 grant and the Mésocentre of Strasbourg for providing access to the cluster. We would also like to thank François Petitjean and Charlotte Pelletier for the fruitful discussions, their feedback and comments while writing this paper.

dl-4-tsc's Issues

about folder tree

dear author, why don't you provide the folder tree so it will be easier for us to run your great work

The version of UCR

Hi,
Your work is great. I have a little question, the results in your table seem to be different from the 'Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline' and I read the issue 'can't reproduce the results with FordB dataset'. So I suspect that the version of UCR is responsible for the difference.
Originally, I thought UCR2018 simply increased the number of datasets. Now it does not seem that way, I do not know if the data sets in UCR 2015 were all reprocessed in UCR 2018. This may lead to differences in the classification accuracy of the same data set under the same network model. Thanks for your help.

Results on the UEA multivariate archive?

Hi Hassan,
Do you have, by any chance, the results of ResNet on https://arxiv.org/abs/1811.00075?
Just wondering :)
François

30 UEA mts datasets

Hi, can you update the accuracies of 30 UEA MTS datasets?

ModuleNotFoundError: No module named 'utils.utils'

I am not able to find out how to import the utils.utils module. I tried a virtual environment and was not able to import the module and any help is greatly appreciated.

run time

Hi Dr Fawaz,
Thanks for providing the deep learning tsc codes. I am about to begin my work which is also on the similar track. So, i downloaded the code from your page https://github.com/hfawaz/dl-4-tsc and cofee data (both in .zip) from UCR archive to my machine and ran the coffee data as mentioned in this page under Code section.
Could your please tell me how long this one line (python3 main.py TSC Coffee fcn _itr_8) code should take to execute? if each prerequisite is completed already. Thank you.

Regards,
Vibha

FileNotFoundError: [Errno 2] No such file or directory: '...............x_train.npy'

请问x_train.npy怎么得到的？

can't reproduce the results with FordB dataset

Hi there,

congratulations for this amazing repository!

I'm trying to reproduce some of the results, but I was going crazy with the FordB dataset because I was obtaining a much lower accuracy with every model. Then I checked the results in the file: "dl-4-tsc/results/results-ucr-128.csv" (below a sample of that file)

They don't match with the results table, where you get 87.8 accuracy with the FCN model as instance in stead of 77.x that is shown in the results file.

Is that all right? Am I missing something?

Thanks in advance,
Jorge

also

classify new sample

What is the purpose of the 'predict' function in the classifiers files? it's look that this function not in use at all.

In addition, I would appreciate your help with making classification for a new sample with the pre-trained model.

Suggestion: Add TCN to benchmark

keras-tcn

Miss MTS dataset

Thanks for the code, maybe it has been a long time now and the MTS link is invalid.
Would you please provide an alternative link for the dataset?

Thank you so much and have a great day.

can be your code used for multivariate time series classification

great code thanks
only can you clarify if
can be your code used for multivariate time series classification
when data is mixter of continues and categorical variables for each time sample?
like
https://github.com/alan-turing-institute/sktime/blob/master/examples/multivariate_time_series_classification.ipynb

What is the version number of tensorflow and keras used

First, thank for your variable work.
When I run the model, many Errors in tensorflow have appeared, so want to know the version number of tensorflow and keras you used.

will it work for multivariate time series prediction both regression and classification

great code thanks
may you clarify :
will it work for multivariate time series prediction both regression and classification
1
where all values are continues values
weight height age target
1 56 160 34 1.2
2 77 170 54 3.5
3 87 167 43 0.7
4 55 198 72 0.5
5 88 176 32 2.3

2
or even will it work for multivariate time series where values are mixture of continues and categorical values
for example 2 dimensions have continues values and 3 dimensions are categorical values

color        weight     gender  height  age  target 

1 black 56 m 160 34 yes
2 white 77 f 170 54 no
3 yellow 87 m 167 43 yes
4 white 55 m 198 72 no
5 white 88 f 176 32 yes

OSError: /b/home/uha/hfawaz-datas/dl-tsc//archives/TSC/Coffee/Coffee_TRAIN not found.

Method: TSC Coffee fcn _itr_8
Traceback (most recent call last):
File "main.py", line 150, in
datasets_dict = read_dataset(root_dir, archive_name, dataset_name)
File "/content/dl-4-tsc/utils/utils.py", line 105, in read_dataset
x_train, y_train = readucr(file_name + '_TRAIN')
File "/content/dl-4-tsc/utils/utils.py", line 33, in readucr
data = np.loadtxt(filename, delimiter=',')
File "/usr/local/lib/python3.6/dist-packages/numpy/lib/npyio.py", line 981, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/usr/local/lib/python3.6/dist-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/usr/local/lib/python3.6/dist-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: /b/home/uha/hfawaz-datas/dl-tsc//archives/TSC/Coffee/Coffee_TRAIN not found.

best_model.hdf5

Does it generate the best_model? or how is it going to work?

OSError: Unable to open file (unable to open file: name = '/data1/prjs/code/ABTS/dl_4_tsc//results/fcn/UCRArchive_2018_itr_8/Coffee/best_model.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

The link to Multivariate time series datasets is now missing

Thank you for your works!

It's a great contribution for the study of time series classification.
I'm very interested on your works but I noticed that the link to Multivariate time series datasets is now missing.

Can you help me fix that or give me another link？
Thank you very much

y_true vs y_test

Hi,
just a quick question. In twiesn.py, the fit method of the classifier asks for y_true and y_test but doesn't seem to use the latter for anything (it's used to define self.y_test but nothing else). Is there a difference between the two? I haven't looked in detail but I think there was a similar point in inception.py?
Cheers

How to prepare data for model

Greetings. First of all thanks for such wonder full work in time series domain. I want to know how to prepare data that can feed to model you have listed.
What i am doing is reshaping them as follows

72x18x30000

where 72 are number of subjects/examples. Each example has 18 channels, and each channel has 30,000 data points. I have add channels_first in keras layer.
Let me know, please, is my input shape is correct

Also some error on utils.py

First, thank you for your program shared.
Then in lines: 313 314 338 339 in utils.py.
['acc'] and ['val_acc'] should be modified to ['accuracy'] and ['val_accuracy'].
If not it will report that : Error : ' acc '.

Non-stationary timeseries

Hello,
great work!

I would like to know how the code handles non-stationary timeseries (based on p-value criteria / Dickey-Fuller-Test). Is there any stationization involved? / or do i need to stationize experiment data beforehand?

best regards

Inception Time Model Predicts all 1's

Hello,

Thank you for sharing the repo. I am trying to implement your InceptionTime model and I'm running into some issues. All of my predicted values are exactly the same, they are all 1.

Perhaps there is something I am doing wrong, although I believe to have followed your code exactly. I am even getting good validation results when the model is being fitted.

Here are some snippets of code, please let me know if you need any more information.

As you can see in the images, y_pred is all 1's although this doesn't make too much sense as fitting the model before doing the prediction tells a different story.

some error in utils.py

In the viz_cam function，“unresolved reference ‘spline’” error occurred.

data format of UCR2018

In utils.py, there is "pd.read_csv(..._TRAIN.tsv)",but there is now only data of ts format provided on the official website. Then there is obvious error for "y_train = df_train.values[:,0]" if data of ts format is used. Can you add some comment about the data shape? I'm really quite confused.

How do you load the MTS archive for Multivariate

Hey Hassan,
AsSalaamu Alaykum, can you explain to me how you formatted the MTS archive set to be ingested by your scripts? When I download it, it's in the form of a matlab file and I don't have matlab to extract the data. Please advise. Thanks,

Sincerely,
Sohaib

AttributeError: 'Tensor' object has no attribute 'assign'

First of all, thank you for your marvellous publication and also for open-sourcing your code.

I've been trying out your code with my own datasets and came across this error when using the MCNN model:

Method:  my_archive tmp_bs16_binarySecond_offset_kmeans mcnn
C:\Users\<username>\Anaconda3\envs\tf_gpu\lib\site-packages\sklearn\preproces
sing\_encoders.py:368: FutureWarning: The handling of integer data will change i
n version 0.22. Currently, the categories are determined based on the range [0,
max(values)], while in the future they will be determined based on the unique va
lues.
If you want the future behaviour and silence this warning, you can specify "cate
gories='auto'".
In case you used a LabelEncoder before this OneHotEncoder to convert the categor
ies to integers, then you can now use the OneHotEncoder directly.
  warnings.warn(msg, FutureWarning)
Using TensorFlow backend.
________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to

================================================================
input_1 (InputLayer)            (None, 14, 1)        0

________________________________________________________________________________
input_2 (InputLayer)            (None, 7, 1)         0

________________________________________________________________________________
input_3 (InputLayer)            (None, 5, 1)         0

________________________________________________________________________________
input_4 (InputLayer)            (None, 4, 1)         0

________________________________________________________________________________
input_5 (InputLayer)            (None, 10, 1)        0

________________________________________________________________________________
conv1d_1 (Conv1D)               (None, 14, 256)      256         input_1[0][0]

________________________________________________________________________________
conv1d_2 (Conv1D)               (None, 7, 256)       256         input_2[0][0]

________________________________________________________________________________
conv1d_3 (Conv1D)               (None, 5, 256)       256         input_3[0][0]

________________________________________________________________________________
conv1d_4 (Conv1D)               (None, 4, 256)       256         input_4[0][0]

________________________________________________________________________________
conv1d_5 (Conv1D)               (None, 10, 256)      256         input_5[0][0]

________________________________________________________________________________
max_pooling1d_1 (MaxPooling1D)  (None, 2, 256)       0           conv1d_1[0][0]

________________________________________________________________________________
max_pooling1d_2 (MaxPooling1D)  (None, 2, 256)       0           conv1d_2[0][0]

________________________________________________________________________________
max_pooling1d_3 (MaxPooling1D)  (None, 2, 256)       0           conv1d_3[0][0]

________________________________________________________________________________
max_pooling1d_4 (MaxPooling1D)  (None, 2, 256)       0           conv1d_4[0][0]

________________________________________________________________________________
max_pooling1d_5 (MaxPooling1D)  (None, 2, 256)       0           conv1d_5[0][0]

________________________________________________________________________________
concatenate_1 (Concatenate)     (None, 2, 1280)      0           max_pooling1d_1
[0][0]
                                                                 max_pooling1d_2
[0][0]
                                                                 max_pooling1d_3
[0][0]
                                                                 max_pooling1d_4
[0][0]
                                                                 max_pooling1d_5
[0][0]
________________________________________________________________________________
conv1d_6 (Conv1D)               (None, 2, 256)       256         concatenate_1[0
][0]
________________________________________________________________________________
max_pooling1d_6 (MaxPooling1D)  (None, 2, 256)       0           conv1d_6[0][0]

________________________________________________________________________________
flatten_1 (Flatten)             (None, 512)          0           max_pooling1d_6
[0][0]
________________________________________________________________________________
dense_1 (Dense)                 (None, 256)          131328      flatten_1[0][0]

________________________________________________________________________________
dense_2 (Dense)                 (None, 2)            514         dense_1[0][0]

===============================================================
Total params: 133,378
Trainable params: 133,378
Non-trainable params: 0
________________________________________________________________________________
Traceback (most recent call last):
  File "main.py", line 112, in <module>
    fit_classifier()
  File "main.py", line 41, in fit_classifier
    classifier.fit(x_train,y_train,x_test,y_test, y_true)
  File "C:\Users\<username>\Desktop\Abschlussprojekt\Analysis\dl-4-tsc-master
\classifiers\mcnn.py", line 454, in fit
    y_test,y_true,pool_factor,filter_size)
  File "C:\Users\<username>\Desktop\Abschlussprojekt\Analysis\dl-4-tsc-master
\classifiers\mcnn.py", line 281, in train
    cost_ij, accuracy = model.train_on_batch(x,y)
  File "C:\Users\<username>\Anaconda3\envs\tf_gpu\lib\site-packages\keras\eng
ine\training.py", line 1216, in train_on_batch
    self._make_train_function()
  File "C:\Users\<username>\Anaconda3\envs\tf_gpu\lib\site-packages\keras\eng
ine\training.py", line 509, in _make_train_function
    loss=self.total_loss)
  File "C:\Users\<username>\Anaconda3\envs\tf_gpu\lib\site-packages\keras\leg
acy\interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\<username>\Anaconda3\envs\tf_gpu\lib\site-packages\keras\opt
imizers.py", line 505, in get_updates
    self.updates.append(K.update(m, m_t))
  File "C:\Users\<username>\Anaconda3\envs\tf_gpu\lib\site-packages\keras\bac
kend\tensorflow_backend.py", line 973, in update
    return tf.assign(x, new_x)
  File "C:\Users\<username>\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflo
w\python\ops\state_ops.py", line 222, in assign
    return ref.assign(value, name=name)
AttributeError: 'Tensor' object has no attribute 'assign'

I am not sure what exactly is causing this error. I have tried the MCNN model with the UCR/Adiac dataset and it worked fine. I can only imagine it being due to the different input dataset. I have used the same input dataset for other models of your study and it worked fine on them as well.

It would be much appreciated if you could lead me in the right direction. If you need any more information, please let me know.

Best regards,

Some errors occured when running mcdd and twiesn

When I try to run the model of mcdd and twiesn, something wrong occured, can you have a look. Thank you.

use for multiple id and static categorical feature

I have multivariate time series (13 series) for all around 100 id. the data also have static category data. is the repo can handle my dataset ? Do I need to include static category as new channel in data set (but the value would be same for every time step)?

please advice
thank you

regards

Validation of Models on Loss Function

Hello

Thanks again for sharing your code.

I successfully applied your code on my datasets and found, when inspecting the results in e.g. df_best_model.csv and df_metrics, that you deem the model as the best model, by monitoring the loss function of the training set. For example, in the FCN model you are setting a ModelCheckpoint by monitoring the training loss. But in the mentioned metrics and your paper you rate the model by its validation accuracy, is that correct?

file_path = self.output_directory+'best_model.hdf5'

model_checkpoint = keras.callbacks.ModelCheckpoint(filepath=file_path, 
    monitor='loss', save_best_only=True)

self.callbacks = [reduce_lr,model_checkpoint]

hist = self.model.fit(x_train, y_train, batch_size=mini_batch_size, 
    epochs=nb_epochs, verbose=self.verbose, validation_data=(x_val,y_val), 
    callbacks=self.callbacks)

I have run through multiple model / dataset variations and found that sometimes Model A with parameters A is better, measured in training loss, than Model A with parameters B. However, in these cases Model A with parameters A overfitted and its validation loss is worse than that of Model A with parameters B. When measuring the performance of a model by its validation accuracy, wouldn't it be more accurate to set the ModelCheckpoint to monitor val_loss or val_acc instead of train_loss?

Thanks

unable to find best_model

Hi,

For some datasets (i.e. DodgerLoopDay), I got an error when running FCN classifier.
Actually, I got the error "OSError: Unable to open file (unable to open file: name = 'xxx/results/fcn/UCRArchive_2018/DodgerLoopDay/best_model.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)"
It seems that there is no best_model.hdf5 saved.

Do you know how to manage/fix this issue?

Thank you

mts command

Can you please share the command to run a mts dataset?

Tensorflow compatibility issues

Hello,

When trying to run the FCN/Resnet models I get cudnn errors:

2019-09-20 10:07:17.847490: E tensorflow/stream_executor/cuda/cuda_dnn.cc:319] Loaded runtime CuDNN library: 7.0.5 but source was compiled with: 7.3.0. CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.

tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found. (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node conv1d_4/convolution}}]] [[loss/mul/_939]] (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node conv1d_4/convolution}}]]

I might need to upgrade my CudNN version but before I try this I wanted to ask which version of Tensorflow-gpu did you use? (I am assuming you used a GPU...?)

Thanks

does it mean your code works for classification with mixture of categorical and continues data?

great code thanks
you give link to
The MTS archive, which contains the 13 multivariate time series datasets
does it mean your code works with mixture of categorical and continues data?
for example at time t1 we have observation: red, 2.4 , 5, 12.456 and time t2: green, 3.5, 2, 45.78; time t3: black, 5.6, 7, 23.56; t4: red, 2.1, 5, 12.6 ?
etc

by the way do you know not NN github code for classification of mixture of categorical and continues data? let say data is not long enough to use NN?

USE CUSTOM DATA

hi, how can i efficiently use the code by ingesting my personal data?

vs SVM('rbf')?

How do I use GPU to run the code?

How do I use GPU to run the code?

How do you deal with data of varying length?

such as "AllGestureWiimoteX" in UCRArchive_2018, it includes Nan in the train.tsv and test.tsv. How do you deal with this?
Thanks a lot to inform me.

ValueError: Unknown layer: InstanceNormalization

First of all: Thank you very much for providing this code. It really helps a lot not to reinvent the wheel every time.

Unfortunately, I'm greeted with the following error whenever I attempt to run the "Encoder" model:

ValueError: Unknown layer: InstanceNormalization

I did not change anything in the way the layers are imported. Could you please have a look into this issue?

The input size of the twiesn?

Hello, I am very confused about this network model. What is the input size of the time series model? In other words, how many time steps are used as an input to the network? I would be very grateful if I can get your answer！！！

About MCNN

WARNING:tensorflow:From C:\Users\MOTNOG\anaconda3\lib\site-packages\tensorflow\python\ops\init_ops.py:97: calling GlorotUniform.init (from tensorflow.python.ops.i
nit_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From C:\Users\MOTNOG\anaconda3\lib\site-packages\tensorflow\python\ops\init_ops.py:97: calling Zeros.init (from tensorflow.python.ops.init_ops)
with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
C:\Users\MOTNOG\anaconda3\lib\site-packages\sklearn\metrics_classification.py:1272: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels w
ith no predicted samples. Use zero_division parameter to control this behavior.
_warn_prf(average, modifier, msg_start, len(result))

Issue using class activation map

Hi,

Thank you so much for sharing the code.

I'm having some issues with using class activation map on a dataset trained using FCN:

x_train.reshape(x_train.shape[0], x_train.shape[1], 1)

ValueError: cannot reshape array of size 1272000 into shape (424,750,1)

My x_train has the size (343, 750, 4) in which I have 343 samples with 750 points and 4 channels. What am I doing wrong?

Thank you in advance

How to improve validation score

I tried with my data, looks poor, I guess it is overfitting, can you please discuss what is wrong happening and what can be changed ?

https://colab.research.google.com/drive/1dSvzTLVk65i0Q0yUL_4O3xOoJILCz8wa?usp=sharing

Having issues in regression

II am using univariate time-series data that i simulated from my models for this dsc-4-tsc codes. I modified the codes according to the regression for me. Some codes worked well. But mcdcnn, cnn, and mcnn are not providing expected results. I mean mcdcnn and cnn are throwing the same loss and metric values (like mape, mae, etc regression) throughout all the epochs, . I checked the given classification codes with the provided coffee data, it worked well for classification. Could you please suggest for modification in the codes for regression? if possible. thanks

Question on validation dataset for TSC

Hi, thanks for you good work.
I am new in TSC and try to follow your work. But I am confued about the validation dataset code in this repo.
It seems that some classifiers lack validation datasets and just use the test dataset instead, e.g., encoder and inception.
In this way, how do I determine the best model to get the results of test datasets in TSC problem?

Thank you.

Is the data being pulled in N,T,F

Hey Hassan,

Is the data that is being pulled in for the multivariate MTS data in the shape of (Number of Samples, Timesteps, Number of Features)? Please inform, Thanks,

Sincerely,
Sohaib

How to analyze the CD diagram

Hi,
I read your paper (Deep learning for time series classification: a review ) these days . I am always confused about how to analyze the CD diagram. For example, the FIg.7 in your paper. There are connections between MLP, Time-CNN, and TWIESN, as well as between TWIESN and MCDCNN. That means there's no statistical difference between MLP, Time-CNN and TWIESN, and there's no statistical difference between TWIESN and MCDCNN.
So, there is no statistical difference between MLP and MCDCNN? I doubt the correctness of this conclusion. Because these four methods are not connected by a single connection. But I do not know what the right conclusion is, could you please give me some help? Thanks a lot.