vsainteuf / lightweight-temporal-attention-pytorch

A PyTorch implementation of the Light Temporal Attention Encoder (L-TAE) for satellite image time series. classification

Lightweight Temporal Self-Attention (PyTorch)

A PyTorch implementation of the Light Temporal Attention Encoder (L-TAE) for satellite image time series classification. (see preprint here)

The increasing accessibility and precision of Earth observation satellite data offers considerable opportunities for industrial and state actors alike. This calls however for efficient methods able to process time-series on a global scale. Building on recent work employing multi-headed self-attention mechanisms to classify remote sensing time sequences, we propose a modification of the Temporal Attention Encoder. In our network, the channels of the temporal inputs are distributed among several compact attention heads operating in parallel. Each head extracts highly-specialized temporal features which are in turn concatenated into a single representation. Our approach outperforms other state-of-the-art time series classification algorithms on an open-access satellite image dataset, while using significantly fewer parameters and with a reduced computational complexity.

(see preprint here)

Requirements

• PyTorch + Torchnet
• Numpy + Scipy + scikit-learn

(see requirements.txt)

The code was developed in python 3.6.10 with pytorch 1.5.0

Downloads

Sentinel-Agri dataset

We use the Sentinel-Agri dataset available on this github repository. The dataset is comprised of time series of satellite images of agricultural parcels. Check the "Data Format" section of the repository for more details on the data.

Pre-trained weights

Pre-trained weights of the PSE+LTAE model available here

Use the models.stclassifier.PseLTae_pretrained class to instanciate the pre-trained model.

Code

This repo contains all the necessary scripts to reproduce the figure below. The implementations of the L-TAE, TAE, GRU and TempCNN temporal modules can be found in models. These four modules are combined with a Pixel-Set Encoder to form a spatio-temporal classifier, directly applicable on the Sentinel-Agri PixelSet dataset. The four architectures are found in models.stclassifier.

Use the train.py script to train the 150k-parameter L-TAE based classifier (by default). You will only need to specify the path to the dataset folder:

python train.py --dataset_folder path_to_sentinelagri_pixelset_dataset

You can use the same script to play around with the model's hyperparameters, or train an instance of a competing architecture.

To train the precise configurations that were used to produce the figure, add the arguments that are listed in the config_fig2.json file. For example, the following command will train the 9k-parameter L-TAE instance:

python train.py --dataset_folder path_to_sentinelagri_pixelset_dataset --n_head 8 --d_k 8 --mlp3 [128]

Credits

• The Lightweight Temporal Attention Encoder is heavily inspired by the works of Vaswani et al. on the Transformer, and this pytorch implementation served as code base for the ltae.py script.
• Credits to github.com/clcarwin/ for the pytorch implementation of the focal loss

Reference

Please include a citation to the following paper if you use the L-TAE.

@article{garnot2020ltae,
  title={Lightweight Temporal Self-Attention  for Classifying Satellite Images Time Series},
  author={Sainte Fare Garnot, Vivien  and Landrieu, Loic},
  journal={arXiv preprint arXiv:2007.00586},
  year={2020}
}

Make sure to also include a citation to the PSE+TAE paper below if you are using the Pixel-Set Encoder.

@article{garnot2020psetae,
  title={Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention},
  author={Sainte Fare Garnot, Vivien  and Landrieu, Loic and Giordano, Sebastien and Chehata, Nesrine},
  journal={CVPR},
  year={2020}
}

lightweight-temporal-attention-pytorch's Issues

Question about Time axis

Hello Vivien,

Sorry to bother again, I have a quick question about time index:

Is it correct that you need to have precisely the same time sampling to score the model ? For example if the model was trained on 28 dates in 2017, every input tensor must have a sample set of pixels at each of these dates: (28, C, S) ?

Thanks a lot !