asrafulashiq / hamnet

PyTorch implementation of AAAI 2021 paper: A Hybrid Attention Mechanism for Weakly-Supervised Temporal Action Localization

HAM-Net

This repository contains code for the AAAI 2021 paper:

A Hybrid Attention Mechanism for Weakly-Supervised Temporal Action Localization

Overview

Prerequisites

• PyTorch 1.7.1
• pytorch-lightning 1.1.2
• loguru, colorama, etc.

Older versions of PyTorch(1.3+) and pytorch-lightning(0.9+) should also work but not tested.

You can create a new conda environment with all the dependencies using:

conda env create -f environment.yml

How to Run

Download Data

The ground-truth and I3D features for THUMOS14 and ActivitiNet1.2 dataset can be downloaded from here:

Box Download Link

Please put the downloaded files/folders under data/ directory.

Training

To train HAM-Net on Thumos14 dataset:

python main.py

Please check options.py to know more about the available cli arguments.

Testing

To evaluate on Thumos14 dataset:

python main.py --test --ckpt [checkpoint_path]

For ActivityNet-1.2, use main_anet.py script.

Citation

If you find this repo useful for your research, please consider citing the paper:

@misc{islam2021hybrid,
      title={A Hybrid Attention Mechanism for Weakly-Supervised Temporal Action Localization}, 
      author={Ashraful Islam and Chengjiang Long and Richard J. Radke},
      year={2021},
      eprint={2101.00545},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgement

• BaSNet
• ActivityNet
• WSAD

hamnet's Issues

About ablation study

When I only remain BCL Loss and SAL Loss，my AVG mAP can only reach 0.38%，I don't know the reason.
Which part of the code do I need to adjust?

你好，这个问题怎么解决啊

Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound:

• setuptools==49.6.0=py37he5f6b98_2
• certifi==2020.12.5=py37h89c1867_0
• sqlite==3.34.0=h74cdb3f_0
• ld_impl_linux-64==2.35.1=hea4e1c9_1
• libgcc-ng==9.3.0=h5dbcf3e_17
• python==3.7.9=hffdb5ce_0_cpython
• libstdcxx-ng==9.3.0=h2ae2ef3_17
• zlib==1.2.11=h516909a_1010
• tk==8.6.10=h21135ba_1
• _libgcc_mutex==0.1=conda_forge
• _openmp_mutex==4.5=1_gnu
• openssl==1.1.1i=h7f98852_0
• libffi==3.3=h58526e2_2
• ncurses==6.2=h58526e2_4
• readline==8.0=he28a2e2_2
• libgomp==9.3.0=h5dbcf3e_17
• ca-certificates==2020.12.5=ha878542_0
• xz==5.2.5=h516909a_1

labels.npy and labels_all.npy

May I know what is the difference between labels.npy and labels_all.npy? My understanding is that labels.npy contains labels for each action segment in a video, whereas labels_all.npy contains labels for unique actions in a video. However, when I try to generate unique action labels for each video from labels.npy, there seem to be some discrepancies between what I'm getting vs what is in labels_all.py

.

hi

about feature_fps ~ what is it ?

pytorch_lightning.py is missing Activity1.2 feature

First, thank you for your excellent work!
I cloned your hamnet code and found that the pytorch_lightning.py was missing.
Could you share this file?

And would you share the I3D feature about THUMOS14 and ActivityNet1.2, especially ActivityNet1.2?

Thanks a lot!

How to extract features by myself?

Hello, could you plz release the more detailed code/processing of extracting feature?

About adding new branches

Hi, I tried to add a new branch, but when I tested, the mAP all became 0. Do I need to make any changes to other code files in the project?

Can not reproduce the performance in your paper.

First, thanks for your great work on the weakly-supervised temporal action localization task.
Sorry to bother you, I was curious about why the released codes are not the same as the description in your paper?

Eg-1. In your parse_args.py for training THUMOS-14 datasets, you set the num_class = 20 which is actually the action class number. However, in your paper, you said that your snippet level CAS's dimension is class number + 1.

Eg-2. In your paper, you said that you introduced $$L_{BCL}$$ , $$L_{SAL}$$, $$L_{SSAL}$$, $$L_{HAL}$$ losses for training the temporal attention scores. However, in your released codes, you do not implement the $$L_{BCL}$$ , $$L_{HAL}$$ losses.

Could you tell us the reasons or share the full codes? 😂