finspire13 / diffact

Code for Diffusion Action Segmentation (ICCV 2023)

diffact's Issues

How those videos been extracted?

Hi Author,

Thanks for your work. I found that we can download the datasets with pre calculated dataset. Almost all the papers, they will use those pre calclulated data.

Do you know how those features are been extracted? For example, which model that used to extract? And does your model could accept different features from different models?

I heard that there is a widely used repo: https://github.com/v-iashin/video_features
There are many models support for frame-wise features such as ViT or CLIP or ResNet.

Do you know which features that your code use and did you use some code or tools to calculate them? Thanks

How to use your trained model.

Thanks for sharing your code.
I want to test on your trained model.
The model size is too small, only ~4 MB. Is it just a placeholder?
If yes, could you provide your models for testing?
If no, could you give me some suggestions how to use your provided models.

Many thanks!

Is there code that execute only evaluation?

Hello, @Finspire13 , first of all, thank you for sharing your work!

In your given code, I think main.py does both training and evaluation,

However, is there an code only for evaluation? (like https://github.com/ChinaYi/ASFormer/blob/main/eval.py)

Thank you in advance!

batch_size Can I set it to a different value?

Hi Daochang
thank you for sharing the code. I noticed an issue in line 69 of the main.py file, where batch_size=1 is hard-coded, and it doesn't utilize the batch_size value specified in the command-line arguments. I understand that when I try to manually modify the batch_size value, the code does not run properly. Typically, the batch_size value is not limited to 1. I am wondering if there is a version of the code that supports a batch_size value other than 1.

two differemt seeds

Hello,

Thank you for sharing your code, it has been very useful, but currently I am having trouble to understand why you are using 2 different seeds in the function ddim_sample, first in

DiffAct/model.py

Line 323 in a223e0c

if seed is not None:

and the in https://github.com/Finspire13/DiffAct/blob/a223e0c27a6f02adfb321a518cb0f7dd0d0896b6/model.py#L369C1-L369C9

Thank you in advance,
Summy

why is time_emb added to the second dimension rather than the last dimension of the x?

Thank you for sharing this work.

I got a question that I wish I can get your answer.
In MixedConvAttModule and MixedConvAttModuleV2, why would you add the time_embed to the second dimension, the feature dimension, rather than the last dimension, the last dimension?

Thank you in advance.

Seed

Thanks for sharing your interesting work! I was wondering if there is something missing in order to always getting the same results. I already tried to make some changes to the seed. I saw your code lines related to it and I also tried adding torch.manual_seed(0) but in every execution I get different results. Please let me know if I am missing something.

Regards,
Andrea

Test Code and Reproduce the results

Hi Daochang,

Could you also update the inference instruction?

Based on the training script, I didn't obtain a result similar to what was reported in the paper.

For gtea dataset, results for 2 of 4 splits.

Epoch 10000 - decoder-agg-Test-Acc 79.82436882546652
Epoch 10000 - decoder-agg-Test-Edit 81.8243661100804
Epoch 10000 - decoder-agg-Test-F1@10 87.76978417266189
Epoch 10000 - decoder-agg-Test-F1@25 87.05035971223022
Epoch 10000 - decoder-agg-Test-F1@50 80.5755395683453

Epoch 10000 - decoder-agg-Test-Acc 78.19410319410319
Epoch 10000 - decoder-agg-Test-Edit 79.28571428571429
Epoch 10000 - decoder-agg-Test-F1@10 85.39325842696628
Epoch 10000 - decoder-agg-Test-F1@25 83.89513108614231
Epoch 10000 - decoder-agg-Test-F1@50 77.90262172284645

Any suggestions?

Thanks.