jnzs1836 / era-vsum

ERA: Entity–Relationship Aware Video Summarization with Wasserstein GAN

Project Structure

./cache # cahce for the object detection result
./data # data loaders and video name mapping files
./deployment # code for deployment of the models, e.g. reading the inputting videos.
./evaluation # code for evaluating the results
./factory # factory mode for the solvers and models
./loggers # code for logging the training progress
./notebooks # notebooks for performing the qualitative analysis
./solvers # training solvers based on different settings, i.e. W-GAN and vanilla GAN.
./scripts # scripts for running the training
./models # models used in the project
./utils # utility code for the video summarization

Installation

pip install requirements.txt

If you encounter the errors regarding Detectron2, please check the document.

Running

The entrypoint of our project is the file train_avs.py. We also provide two bash scripts in scripts directory.

bash ./scripts/train_tvsum.sh # train models on TVSum

bash ./scripts/train_summe.sh # train models on SumMe

Evaluation

We offer a trained model checkpoint in the chcekpoints directory. You could test the model on your own dataset and splits. The model is trained on the SumMe split-3. Due to the file size limit, we are only able to add one checkpoint file in the submission.

python generate_scores.py \
	--ckpt_path /your/checkpoint/dir/split-x.pkl \
	--model_name custom_name_for_saving_the_result \
    --output_dir /your/output/dir \
    --split_index 0

Acknowledgement

We thank to j-min for providing the implementation of the original SUM-GAN.

era-vsum's Issues

Unable to find "h5 files"

Hello. I'm a university student in Japan, and researching unsupervised video summarization.
Thanks to your research and codes, I could learn about state-of-the-art method of video summarization.
Then, I have trouble in finding "h5 files", which is necessary for "tvsum_h5_path" in Qualitative Analysis.ipynb.
Also, what is the purpose of "h5 files"? Or, how can I make the files?
If possible, I'd like you to tell me about how to find the files.

OS : Ubuntu 18.04
python version : 3.8.12
virtual environment : venv (virtualenv)

Sincerely yours,
achievementtoo

About "Qualitative Analysis.ipynb"

Hello Guande

I'm a Japanese university student, who posted an issue a few days ago.
Now, I have three questions in your code.

First, I couldn't find "score_directory" and "video_directory" in the second cell.
Also, "read_video_meta function" is not defined.

Second, I had KeyError about "video_key = "video_42"" in the fifth cell.
Where can I find this "video_42"?

Third, where is the "checkpoints_path" in "generate_scores.py", script directory?

I'm sorry for asking many questions.
I'd like you to answer my questions, when it's convenient for you.

Sincerely,
achievementtoo