xiaofanustc / me-sampler

[ECCV2020] Motion-excited Sampler: Video Adversarial Attack with Sparked Prior

Motion-Excited Sampler: Video Adversarial Attack with Sparked Prior

This repository contains code for the paper:

Motion-Excited Sampler: Video Adversarial Attack with Sparked Prior

Hu Zhang, Linchao Zhu, Yi Zhu, Yi Yang

[Arxiv] [Slides] [Demo Video]

ReLER, University of Technology Sydney, NSW; Amazon Web Services

Abstract

Deep neural networks are known to be susceptible to ad- versarial noise, which is tiny and imperceptible perturbations. Most of previous work on adversarial attack mainly focus on image models, while the vulnerability of video models is less explored. In this paper, we aim to attack video models by utilizing intrinsic movement pattern and regional relative motion among video frames. We propose an effective motion- excited sampler to obtain motion-aware noise prior, which we term as sparked prior. Our sparked prior underlines frame correlations and uti- lizes video dynamics via relative motion. By using the sparked prior in gradient estimation, we can successfully attack a variety of video clas- sification models with fewer number of queries. Extensive experimental results on four benchmark datasets validate the efficacy of our proposed method.

Requirements:

• python 3.6
• pytorch 1.0.1
• coviar
• mxnet-gpu 1.5.0: install it by pip install mxnet-cu100==1.5.0 for cuda 10.0
• gluoncv 0.6.0: install it by pip install gluoncv==0.6.0

Dataset

Something-Something v2: video is split into frames by video2frames.py and change the path in run_smth_i3d.sh.

Attacked models

We use existing I3D and TSN2D models from gluoncv, download [here]. You can replace this part with other models.

Mpeg video generation

When use coviar to extract motion vector, first convert original video to mpeg format:

run bash reencode_smth_smth.sh.

Attacking

run bash run_smth_i3d.sh or bash run_smth_tsn.sh

Reminder: when attacking, we impose noise after normalize pixels to 0-1 but before mean and std normalization, thus we need to split previous operations of transformation.

License

This project is licensed under the license found in the LICENSE file in the root directory.

me-sampler's Issues

I think mxnet with cuda and gluoncv should be included in the requirements

Can't run run_smth_i3d.sh for --val-list: command not found

Hi, sorry for interrupt,
When I try to run run run_smth_i3d.sh, some error happened and report: --val-list: command not found. Since I'm not familiar with linux，I hope you can give me some advice.
Thank you!

attack_list

Excuse me. I want to know what is the meaning of the values in your i3d_index_attack.txt? I mean, if I want to creat an attack_list using my own dataset, what are the necessities to be listed like this?

Runscript for UCF101 or HMDB51 datasets

Hi,
I am interested in your work. I would like to test the suggested method on UCF101 or HMDB51 datasets.
Can you please share the runscripts for these datasets? i.e) like run_ucf_i3d.sh, reencode_ucf.sh, congits/.json files, and attack list file?
Thank you.

Broadcasting error

Hello, Xiao.

Thank you for your kind comments on issues.

Now, I faced another error.

Please, look into it and let me know if you need more inforamtion.

Thank you.

Cannot run your code

Hi,
I've been trying to run your code, but it does not run because of so many errors.
I fixed all the problems mentioned by previous issues, but yet another error came out..
I think you manually modified the gluoncv toolkit too much.
Can you please share all your modifications?

Thank you.

Cannot run `bash run_smth_i3d.sh` because `batch[2]` is not accessible.

When I enumerate over val_loader, the length of batch is 2 (batch[0] is an input clip and batch[1] is a target label).
In your batch_fn, however, you access batch[2] to get item_id.
It makes an IndexError as below.