wzmsltw / bsn-boundary-sensitive-network Goto Github PK

Hi Tianwwei,
I download the data from GoogleCloud and when I ran the TEM_train.py, there is ValueError: cannot reshape array of size 16 into shape (16,100,newaxis). Do i unzip the dataset incorrectly?
Thanks

I download the tsn_feature files for many times from the link you provided: https://pan.baidu.com/s/19GI3_-uZbd_XynUO6g-8YQ (i also tried the google drive download link, but will be abruptly ended every time) but every time when i unzip the BaiduYun files,

$ unzip -d ../BMN-Boundary-Matching-Network/data/activitynet_feature_cuhk/ csv_mean_100.zip

i got hundreds of bad zipfile offset errors in my terminal like:

Archive:  csv_mean_100.zip
error: End-of-centdir-64 signature not where expected (prepended bytes?)
  (attempting to process anyway)
warning [csv_mean_100.zip]:  zipfile claims to be last disk of a multi-part archive;
  attempting to process anyway, assuming all parts have been concatenated
  together in order.  Expect "errors" and warnings...true multi-part support
  doesn't exist yet (coming soon).
warning [csv_mean_100.zip]:  5242880000 extra bytes at beginning or within zipfile
  (attempting to process anyway)
file #1:  bad zipfile offset (local header sig):  5242880004
  (attempting to re-compensate)
...
file #3998:  bad zipfile offset (local header sig):  5293978810
file #3999:  bad zipfile offset (local header sig):  5294255897
file #4000:  bad zipfile offset (local header sig):  5294532787
file #4001:  bad zipfile offset (lseek):  5294809088
file #4002:  bad zipfile offset (lseek):  5295079424
file #4003:  bad zipfile offset (lseek):  5295357952
file #4004:  bad zipfile offset (lseek):  5295636480
file #4005:  bad zipfile offset (lseek):  5295915008
file #4006:  bad zipfile offset (lseek):  5296185344
file #4007:  bad zipfile offset (lseek):  5296463872
file #4008:  bad zipfile offset (lseek):  5296742400
file #4009:  bad zipfile offset (lseek):  5297004544
file #4010:  bad zipfile offset (lseek):  5297283072
file #4011:  bad zipfile offset (lseek):  5297553408
file #4012:  bad zipfile offset (lseek):  5297831936
file #4013:  bad zipfile offset (lseek):  5298110464

...

Also, when i run BMN program in pytorch(https://github.com/JJBOY/BMN-Boundary-Matching-Network), i will find some files are missing:

like:

FileNotFoundError: [Errno 2] File b'./data/activitynet_feature_cuhk/csv_mean_100/v_sx_npA4wRrw.csv' does not exist: b'./data/activitynet_feature_cuhk/csv_mean_100/v_sx_npA4wRrw.csv'

Could you please tell me how to re-generate the tsn feature files? So i can make them by myself...

same issue at: JJBOY/BMN-Boundary-Matching-Network#11

Hello, thank you for your contribution.
In the paper, On ActivityNet-1.3, we use top-1 video-level class generated by classification model [44] for all proposals in a video. ([44] means UTS at ActivityNet 2016). But I can't find the source code.
So my question is how do you actually do action detection with proposals?
Thank you very much!

Hi Tianwei,

Thanks for your great work! Could we also have extracted features of THUMOS'14? It would be great if the rescaled and original features for THUMOS'14 are provided. Thank you so much!

There are some inconsistencies between the paper and the implementation when it comes to sampling data for predicting the iou in the PEM. According to the paper you sample with a 1 to 2 ratio between negative and positives. according to the code, you measure according to the ratio between positive and negatives and according to the ratio between medium and positive. can you add more information about that?
Thanks,
Ilan

"x = torch.sigmoid(0.01*self.conv3(x))" why multi 0.01 before sigmoid function?

Hello,

In the paper, you mentioned the threshold of Soft-NMS is set as 0.65. While in the code, you use a if statement as

I was wondering whether you could provide some details about this part.

Thanks

Hi,

This is nice work, I would like to apply this for video, How this applies to video?

Hello,

I'm trying to experiment with this model using original features.
But I'm in trouble of acquiring video features from "CUHK & ETH & SIAT Solution to ActivityNet Challenge 2016" model.
I want to get the original features used in your paper. (2048-dim I think, and whose length is not rescaled to 100)
Will you able to provide the 2048-dim original two stream features not rescaled?

Thank you.

Hello,

Can you please also provide the rescaled features for Thumos14 similar to ActivityNet dataset?

Hi Tianwei,
Thanks for your great work！ Can you provide the C3D features for THUMOS14？It would be great if the original features for THUMOS'14 are provided. Thanks!

Hi, Tianwei Lin, Thank you for your contribution! I have do some experiments on thumos14 based on your training code for ActivityNet! During feature encoding, I rescale the feature sequence to 100, but I got poor results that AR@100 is about 22%. I notice that you do not rescale feature on Thumos14.
Have you do experiments based on rescaled feature on Thumos14 and what about the results? Does the operation of rescaling have so much impact on the results?
Thank you very much!

Hi Tianwei,

After extracting the features, you rescaled them into size (100,400) via linear interpolation. Could you please tell us that what kind of method did you use for that?

Hi,
Mentioned in your ActivityNet report improvement B that you adopt TSN and P3D pretrained on Kinetics-400 dataset for video feature extraction. I wonder whether the model fine-tuned on the ActivityNet dataset. If not, are you using the feature of the last pooling layer?

According to the "Implementation Details" part in Section 4 of the original paper, you use the model pre-trained on the training set of ActivityNet-1.3 as the features extractor. And I don't make anything wrong, your BMN uses the output of the last layer as the feature. That's why you get 400-dimensions length features (200 classes, 2 streams).

But the output of the last layer should represent the predicted class scores of the input frame of 200 ActivityNet actions. This kind of feature should be meaningless for THUMOS14 because half of the action classes in THUMOS14 don't belong to ActivityNet. How can you detect an unseen type of action using the classification scores of 200 irrelative actions?

Thanks for your great work.
I want to get the evaluation results. So I followed the readme in Evaluation and use the command python get_classification_performance.py data/activity_net.v1-3.min.json sample_classification_prediction.json. But I got the error urllib2.URLError: <urlopen error [Errno 110] Connection timed out>. Then I checked the url: http://ec2-52-11-11-89.us-west-2.compute.amazonaws.com/challenge16/api.py?action=get_blocked. Ant it is not accessible. Could you please help me with the probelm ?
Thank you !

Hi,

I got this problem of data/activitynet_feature_cuhk/csv_mean_100/v_nt4Ag91306U.csv does not exist. It seems that some of the files are missing.Can you please tell me how to solve this problem?

Hi tianwei!
Thanks a lot about BSN code release.
I tried to train BSN on THUMOS14, do u have annotations of THUMOS14 used in BSN training procedure?

@wzmsltw Lin, thanks for your repo.

I have downloaded the original features you provided, but how to unpack and use the features, any ideas?

i use command cat original_spatial_csv.tar.gz.0* > original_spatial_csv.tar.gz, and then tar -zxvf original_spatial_csv.tar.gz, but error occurred

tar: Unrecognized archive format
tar: Error exit delayed from previous errors.

As described in your paper, you perform action detection by classifying proposals using Untrimmed Net or SCNN. I am trying to do this with UntrimmendNet but don't know how.
Can you provide more details? Do I need to re-train UntrimmedNet based on BSN generated proposals or the "shots" described in the UntrimmedNet paper? And how about testing? What are the inputs and outputs?

where is _image_labels.bin,_temporal_labels.bin,*_feature_data.bin?

Hi Tianwei,

After extracting the features, you rescaled them into size (100,400) via linear interpolation. Could you please tell us that what kind of method did you use for that?

Thank you for your great help!
I have another question. About THUMOS14, may I ask what is the way you take to extract features?
The FC layer output of anet is 200 classes. Did you modify the FC layer and then train on THUMOS14?

Hi Will,
Could you please tell me how to control the different numbers of proposals precisely, e.g. 50, 100, 200?
I know it maybe by setting the threshold, but I have no idea how to do it and make the number of proposals are excatly the amount you want.
Thanks!

Hi, thanks for the great work.

For my project, I would like to visualize the the probability actionness score for input video. How can I visualize that sequence? Any help will be appreciated.

Thank you!

Hi, Tianwei Lin
Thanks for your great sharing.
When I use the thumos14 features you shared, I find the row number of rgb and flow features of the same videos are different,why so and how can I do?

this repo will get ActivityNet v1.3 result according to your README,but in your paper,we can also see THUMOS2014 result, so could you please release some pre-trained model on THUMOS2014 dataset?
I believe the model in ./models/TEM or PEM is pre-trained model on ActivityNet ?

Hi~
When i run the code python TEM_train.py, i meet a error about feature,like this:
IOError:File ./data/activitynet_feature_chunk/csv_mean_100/v_QOlsCBRmfWY.csv does not exit
I check the zip_csv_mean_100.zip and cannot find this file...
What's wrong with it? Is it the feature missing?

Excusme ,could you tell me how you get video_info.csv and anet_anno_action.csv files ?
Is the two files you created from activity-net.v1-3.min.json ? Then could you please send the script to me?
Thank you very much!!!
My email is [email protected]

Hi,

(1) You mentioned that "For convenience of training and testing, we rescale the feature length of all videos to same length 100", if I understand correctly, this means that you only extract 100 snippets per video clip? Also would you mind elaborating on how the video resizing is done?

(2) Also there are many improvements for the winning solution compared to BSN baseline, will you consider releasing the code for the winning solution?

Thank you

Hi, thanks for your great contribution!
However I have two questions while I am downloading the dataset activitynet-1.3 as follows:

1. It shows that some of the youtube accountis terminated so the video is not available either.
2. Due to some unknown reasons, the video is unable to extract, also the webpage cannot be reached.
   

Now I have downloaded just 18449 videos, and I notice that there are 19228 csv files instead of original 19994. So we could not download them all, right?

Thank you for your contribution! I've learned a lot from your code. The caffe files about anet_16 and UntrimmedNet can't compiled on my computer. Could you please send me the classification results of anet and UntrimmedNet on ActivityNet and THUMOS data sets respectively?
email : [email protected]

Hi, I think there is some inconsistency between the code and your paper in terms of how ground truth regions are been generated. In the TEM_load_data.py file, you have:

gt_lens=gt_xmaxs-gt_xmins
gt_len_small=np.maximum(tgap,0.1*gt_lens)
gt_start_bboxs=np.stack((gt_xmins-gt_len_small/2,gt_xmins+gt_len_small/2),axis=1)
gt_end_bboxs=np.stack((gt_xmaxs-gt_len_small/2,gt_xmaxs+gt_len_small/2),axis=1)

I believe that this correspond to 3.4 part of the paper where you describe TEM training as:

For ground truth action instance φ g = (t s , t e ) in Ψ ω , we denote its region
as action region r ... as r = [t s − d g /10, t s + d g /10]
and r = [t e − d g /10, t e + d g /10] separately, where d g = t e − t s ...

So it seems that in your paper the start and end regions are generated by +/- one-tenth of the ground truth duration. But in your code it is +/- duration/20 (divide by 10 and then divide by 2 again).

It will great if you can help clarify this part. Thank you.

Hi Tianwei,

I would like to ask for the -s(step) setting you used in this command.

Did you change the -s 1 to -s 8 or -s 16? If not, how do you skip the sampling of frames?

Thanks! ;)

Hi! Thanks a lot for the code! I am referring to this model as well as BMN model for my own work and I need to test my model on the Thumos dataset too. Can you please share the annotation files for the THUMOS dataset? I would really appreciate it if you do!
Thanks!

Hi~ I've downloaded the tsn feature from Baidu Yun, but I found the total number of .csv files in the csv_mean_100 directory is 3992 after unzip. I want to know where are the video features for all the videos?

Thank you for your contribution.Limited by the computer, I could only do experiments on the small data set THUMOS14.However, there was something I did not understand about the training process of THUMOS14, which caused me to fail to achieve the effect in the paper.For example, the setting of the TEM training set in THUMOS14 (the sliding window size is 100, so how to regularize time for each group) and the extraction of BSP features.
Could you please send the code to me? E-mail: 349249680 @qq.com
Thank you very much!

Thanks for sharing your work on github!
Is it possible to share the THUMOS14 proposal results? We do not need the code or model, just a json file with the proposals (video-id, start, end, score) would be fine. We want to compare our proposal methods with this and do some analysis.
Many thanks.

Hello, Tianwei~ I tried to reimplement your action detection experiments on activity-net, but only get a poor result. Here is my results using different iou thresh on NMS:

I found average number of proposals is far smaller than 100. Does it matter?

I have checked my code several times, but can not find any mistake. Can you help me? Or share your code which generates the detection result. Many thanks.

I've been looking at this for the past few days and I can't seem to figure out how I can use this to train on any other dataset than Thumos or ActivityNet. How would you train this in the end-to-end fashion proposed in the article please ?

Hi, as you mentioned in the paper, RGB and optical flow features are extracted by two-stream network.
So I am wondering what is the meaning of 400-d vector in .csv file?
Are they ordered? For example, the first 200-d means RGB feature and the rest means optical?

Your job is excellent, but where can I get the two-stream and C3D features for comparative experiments? thankes！！！

the file download from Baidu as your guide, would you please help? thanks.

Hi,

Thanks for releasing your code. I was wondering if you could also share the proposal predictions on activitynet training and validation sets? Thanks so much!

Hi, Tianwei Lin
Thanks for your BSN codes sharing.
From paper and the function poolData code in this project, I noticed that there is another operation before you pass TSN features to BSN network.
That operation does interpolation from var-length TSN feature to fixed-length processed feature.

I have read your BSN paper and ActivityNet 2017 paper, but can't find much instruction about that.
Can you give me some reference for it? Any paper or forum discussion will be appreciated.

Another question, as BSN network accepted fix-length feature, how about extract fixed number of frames (and optical flow) for every single video (comparison to extract 1 snippet every 16 frames)?
I think the poolData function, after the TSN feature extraction stage, is the reason why we need to use a pre-trained TSN network. otherwise, we can train the feature extraction network (RGB and Flow) together with the BSN (and BMN ) network online, which may achieve better results.