annbless / dutcode Goto Github PK
View Code? Open in Web Editor NEWPytorch implementation of DUT: Learning Video Stabilization by Simply Watching Unstable Videos
Pytorch implementation of DUT: Learning Video Stabilization by Simply Watching Unstable Videos
Dear Xu,May I ask when the complete training code you mentioned earlier can be released? I am eager to learn about it,please!
Traceback (most recent call last):
File "./scripts/StabNetStabilizer.py", line 36, in
model.load_state_dict(r_model)
File "/home/nikesh/stabl/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", l
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for stabNet:
Missing key(s) in state_dict: "resnet50.resnet_v2_50_block1_unit_1_bottleneck_v2_p_bottleneck_v2_preact_FusedBatchNorm.running_var", "resnet50.resnet_v2_50_block1_unit_1_boet_v2_50_block1_unit_1_bottleneck_v2_conv1_BatchNorm_FusedBatchNorm.running_var", "resnet5m.running_mean", "resnet50.resnet_v2_50_block1_unit_1_bottleneck_v2_conv2_BatchNorm_FusedB2_preact_FusedBatchNorm.running_mean", "resnet50.resnet_v2_50_block1_unit_2_bottleneck_v2__bottleneck_v2_conv1_BatchNorm_FusedBatchNorm.running_mean", "resnet50.resnet_v2_50_block1et50.resnet_v2_50_block1_unit_2_bottleneck_v2_conv2_BatchNorm_FusedBatchNorm.running_mean"edBatchNorm.running_var", "resnet50.resnet_v2_50_block1_unit_3_bottleneck_v2_preact_FusedBv2_preact_FusedBatchNorm.running_var", "resnet50.resnet_v2_50_block1_unit_3_bottleneck_v2_ck1_unit_3_bottleneck_v2_conv1_BatchNorm_FusedBatchNorm.running_var", "resnet50.resnet_v2_an", "resnet50.resnet_v2_50_block1_unit_3_bottleneck_v2_conv2_BatchNorm_FusedBatchNorm.runedBatchNorm.running_mean", "resnet50.resnet_v2_50_block2_unit_1_bottleneck_v2_preact_Fusedv2_conv1_BatchNorm_FusedBatchNorm.running_mean", "resnet50.resnet_v2_50_block2_unit_1_bottv2_50_block2_unit_1_bottleneck_v2_conv2_BatchNorm_FusedBatchNorm.running_mean", "resnet50.running_var", "resnet50.resnet_v2_50_block2_unit_2_bottleneck_v2_preact_FusedBatchNorm.runsedBatchNorm.running_var", "resnet50.resnet_v2_50_block2_unit_2_bottleneck_v2_conv1_BatchNottleneck_v2_conv1_BatchNorm_FusedBatchNorm.running_var", "resnet50.resnet_v2_50_block2_un50.resnet_v2_50_block2_unit_2_bottleneck_v2_conv2_BatchNorm_FusedBatchNorm.running_var", "running_mean", "resnet50.resnet_v2_50_block2_unit_3_bottleneck_v2_preact_FusedBatchNorm.ruchNorm_FusedBatchNorm.running_mean", "resnet50.resnet_v2_50_block2_unit_3_bottleneck_v2_co_unit_3_bottleneck_v2_conv2_BatchNorm_FusedBatchNorm.running_mean", "resnet50.resnet_v2_50, "resnet50.resnet_v2_50_block2_unit_4_bottleneck_v2_preact_FusedBatchNorm.running_mean", .running_var", "resnet50.resnet_v2_50_block2_unit_4_bottleneck_v2_conv1_BatchNorm_FusedBat_conv1_BatchNorm_FusedBatchNorm.running_var", "resnet50.resnet_v2_50_block2_unit_4_bottlen_50_block2_unit_4_bottleneck_v2_conv2_BatchNorm_FusedBatchNorm.running_var", "resnet50.res", "resnet50.resnet_v2_50_block3_unit_1_bottleneck_v2_preact_FusedBatchNorm.running_var", BatchNorm.running_mean", "resnet50.resnet_v2_50_block3_unit_1_bottleneck_v2_conv1_BatchNorleneck_v2_conv2_BatchNorm_FusedBatchNorm.running_mean", "resnet50.resnet_v2_50_block3_unitresnet_v2_50_block3_unit_2_bottleneck_v2_preact_FusedBatchNorm.running_mean", "resnet50.re", "resnet50.resnet_v2_50_block3_unit_2_bottleneck_v2_conv1_BatchNorm_FusedBatchNorm.runniNorm_FusedBatchNorm.running_var", "resnet50.resnet_v2_50_block3_unit_2_bottleneck_v2_conv2nit_2_bottleneck_v2_conv2_BatchNorm_FusedBatchNorm.running_var", "resnet50.resnet_v2_50_bl.resnet_v2_50_block3_unit_3_bottleneck_v2_preact_FusedBatchNorm.running_var", "resnet50.renning_mean", "resnet50.resnet_v2_50_block3_unit_3_bottleneck_v2_conv1_BatchNorm_FusedBatchnv2_BatchNorm_FusedBatchNorm.running_mean", "resnet50.resnet_v2_50_block3_unit_3_bottlenec_block3_unit_4_bottleneck_v2_preact_FusedBatchNorm.running_mean", "resnet50.resnet_v2_50_b.resnet_v2_50_block3_unit_4_bottleneck_v2_conv1_BatchNorm_FusedBatchNorm.running_mean", "rtchNorm.running_var", "resnet50.resnet_v2_50_block3_unit_4_bottleneck_v2_conv2_BatchNorm_Fneck_v2_conv2_BatchNorm_FusedBatchNorm.running_var", "resnet50.resnet_v2_50_block3_unit_5_0_block3_unit_5_bottleneck_v2_preact_FusedBatchNorm.running_var", "resnet50.resnet_v2_50_b "resnet50.resnet_v2_50_block3_unit_5_bottleneck_v2_conv1_BatchNorm_FusedBatchNorm.runningm_FusedBatchNorm.running_mean", "resnet50.resnet_v2_50_block3_unit_5_bottleneck_v2_conv2_B_6_bottleneck_v2_preact_FusedBatchNorm.running_mean", "resnet50.resnet_v2_50_block3_unit_60_block3_unit_6_bottleneck_v2_conv1_BatchNorm_FusedBatchNorm.running_mean", "resnet50.resning_var", "resnet50.resnet_v2_50_block3_unit_6_bottleneck_v2_conv2_BatchNorm_FusedBatchNor2_BatchNorm_FusedBatchNorm.running_var", "resnet50.resnet_v2_50_block4_unit_1_bottleneck_vt_1_bottleneck_v2_preact_FusedBatchNorm.running_var", "resnet50.resnet_v2_50_block4_unit_1esnet_v2_50_block4_unit_1_bottleneck_v2_conv1_BatchNorm_FusedBatchNorm.running_var", "resnNorm.running_mean", "resnet50.resnet_v2_50_block4_unit_1_bottleneck_v2_conv2_BatchNorm_Fusk_v2_preact_FusedBatchNorm.running_mean", "resnet50.resnet_v2_50_block4_unit_2_bottleneck_t_2_bottleneck_v2_conv1_BatchNorm_FusedBatchNorm.running_mean", "resnet50.resnet_v2_50_bloesnet50.resnet_v2_50_block4_unit_2_bottleneck_v2_conv2_BatchNorm_FusedBatchNorm.running_meFusedBatchNorm.running_var", "resnet50.resnet_v2_50_block4_unit_3_bottleneck_v2_preact_Fusck_v2_preact_FusedBatchNorm.running_var", "resnet50.resnet_v2_50_block4_unit_3_bottleneck_block4_unit_3_bottleneck_v2_conv1_BatchNorm_FusedBatchNorm.running_var", "resnet50.resnet__mean", "resnet50.resnet_v2_50_block4_unit_3_bottleneck_v2_conv2_BatchNorm_FusedBatchNorm.n", "resnet50.resnet_v2_50_postnorm_FusedBatchNorm.running_var", "resnet50.resnet_v2_50_
Hello, I'm trying to use the example that you put on google colab however, I face an error as I try to download model files for the the models.the error says :
Cannot retrieve the public link of the file. You may need to change
the permission to 'Anyone with the link', or have had many accesses.
You may still be able to access the file from the browser:
https://drive.google.com/uc?id=12R5wW2yKGMyOY7h7zz3j8gwAFQt1MSJC
but even the lind doesn't works .
Hello,
Thanks for your great work for video stabilization!
How can I inference with video with smaller resolution, such as 114*180? I modified WIDTH and HEIGHT in config file, but when I tried on my data, it occured some error, like ValueError: cannot reshape array of size 216 into shape (9, 11, 2)
. It seems like have something wrong in MotionPro.py.
Looking forward to your reply, thanks!
So far, I am not seeing in the comments posted here that anyone has successfully run this code through to the end. I was able to make headway with an A100 40GB GPU and 600 2k images, however the code fails towards the end due to missing models. It is looking for both sintel.pytorch and stabNet.pth in the ckpt folder, both of which are not included via your download link. Please let us know where to find these models or if you would be able to update your archive to include them. Without them, it is not possible to run this code.
I'm having a problem, is there code for the loss function definition?Thanks!
The pre-training model compression package is corrupted. Specifically, the rfdet_640. path.tar file in the CKPT folder cannot be opened. I want to ask if this is normal. Finally, can you provide the download address of your data set?
Hi there, 1st of all thanks for this awesome work !
Since we've 'doxed' it in our HyMPS project (under VIDEO section \ AI-based page \ Stabilizers), can you please add a GH-compliant license file for it ?
As you know, expliciting licensing terms is extremely important to let anyone better/faster understand how to reuse/adapt/modify sources (and not only) in other open projects and vice-versa.
Although it may sounds like a minor aspect, license file omission obviously causes an inconsistent generation of the relative badge too:
(badge-generator URL: https://badgen.net/github/license/Annbless/DUTCode)
You can easily set a standardized one through the GH's license wizard tool.
Last but not least, let us know how we could improve - in your opinion - our categorizations and links to resources in order to favor collaboration between developers (and therefore evolution) of listed projects.
Hope that helps/inspires !
Excellent job, but I have a question. When there is a large area of sky in the picture with little texture, can the key point detection model based on CNN be able to effectively extract the key points?
Dear Annbless
Thanks for your great work for video stabilization!
I am very respect your work.
Do you have any plans to release the training code and metrics calculation for this project in the near future?
Thank you for an early answer!
Problem with cupy 7.5.0 installation
This is the following error log:
ERROR: Could not find a version that satisfies the requirement cupy==7.5.0
ERROR: No matching distribution found for cupy==7.5.0
Can anybody help?
Thx for your awesome paper and code!
I am trying rewriting training code, however I have some question for my bad understanding QQ
In "def generateSmooth"
why temp_smooth with kernel and abskernel together? cuz in paper, there is no absolute for kernel
My training result, kernel becomes zero and thus Lts become converging XDDD
Is there any constraints for kernel in training?
Dear Annbless
Thanks for your great work for video stabilization! The result is amazing!
Could you release the training code for this project?
Thank you for an early answer!
There is no download link for your data set [18]. Could you please give it to me?
Thank you very much.
Namespace(InputBasePath='images/', OutputBasePath='results/', cuda=True, desiredHeight=480, desiredWidth=640, modelPath='ckpt/DIFNet2.pth', n_iter=3, skip=2, temp_file='./DIFRINT_TEMP/')
Iter: 1
C:\Users\Milan\miniconda3\envs\dutcode\lib\site-packages\torch\nn\functional.py:2705: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
warnings.warn("Default grid_sample and affine_grid behavior has changed "
Frame: 10/4673
Dut stabilizer not working, difrint works fine...
on windows 10, miniconda envionment...these videos are not in classic res....they are smaller than 640x480pix
please tell me where I am wrong?
Hi author, thanks for the Pytorch version of the StabNet inference model, do you have the training code for StabNet now?
Traceback (most recent call last):
File "./scripts/DUTStabilizer.py", line 111, in
generateStable(model, inPath, outPath, outPrefix, maxlength, args)
File "./scripts/DUTStabilizer.py", line 57, in generateStable
origin_motion, smoothPath = model.inference(x.cuda(), x_RGB.cuda(), repeat=args.Repeat)
File "/home/gary/vs/deeplearn/DUTCode/models/DUT/DUT.py", line 273, in inference
for i in range(len(kpts) - 1)]
File "/home/gary/vs/deeplearn/DUTCode/models/DUT/DUT.py", line 273, in
for i in range(len(kpts) - 1)]
File "/home/gary/vs/deeplearn/DUTCode/models/DUT/MotionPro.py", line 105, in inference
motion, gridsMotion, _ = self.homoEstimate(concat_motion, kp)
File "/home/gary/vs/deeplearn/DUTCode/utils/ProjectionUtils.py", line 170, in multiHomoEstimate
pred_Y = KMeans(n_clusters=2, random_state=2).fit_predict(motion_numpy)
File "/home/gary/anaconda3/envs/DUTCode/lib/python3.6/site-packages/sklearn/cluster/kmeans.py", line 1122, in fit_predict
return self.fit(X, sample_weight=sample_weight).labels
File "/home/gary/anaconda3/envs/DUTCode/lib/python3.6/site-packages/sklearn/cluster/_kmeans.py", line 1033, in fit
accept_large_sparse=False)
File "/home/gary/anaconda3/envs/DUTCode/lib/python3.6/site-packages/sklearn/base.py", line 420, in _validate_data
X = check_array(X, **check_params)
File "/home/gary/anaconda3/envs/DUTCode/lib/python3.6/site-packages/sklearn/utils/validation.py", line 72, in inner_f
return f(**kwargs)
File "/home/gary/anaconda3/envs/DUTCode/lib/python3.6/site-packages/sklearn/utils/validation.py", line 645, in check_array
allow_nan=force_all_finite == 'allow-nan')
File "/home/gary/anaconda3/envs/DUTCode/lib/python3.6/site-packages/sklearn/utils/validation.py", line 99, in _assert_all_finite
msg_dtype if msg_dtype is not None else X.dtype)
ValueError: Input contains NaN, infinity or a value too large for dtype('float32').
I use default dataset in folder images, thanks~
Whether real-time effects can be achieved
I cant install the cupy version specified. I tried many other versions but had other problems infront.Please help me out
Hi,
I tried to run inference of your model following your instructions, I got stuck at
Namespace(InputBasePath='images/', MaxLength=1200, MotionProPath='ckpt/MotionPro.pth', OutNamePrefix='', OutputBasePath='results/', PWCNetPath='ckpt/network-default.pytorch', RFDetPath='ckpt/RFDet_640.pth.tar', Repeat=50, SingleHomo=False, SmootherPath='ckpt/smoother.pth')
-------------model configuration------------------------
using RFNet ...
using PWCNet for motion estimation...
using Motion Propagation model with multi homo...
using Deep Smoother Model...
------------------reload parameters-------------------------
reload Smoother params
successfully load 12 params for smoother
reload RFDet Model
any suggestions?
Why is the DUT output video cropped,Is it because of the model?How should this problem be solved?
is there any progress in releasing code?
Hello, I started ‘Run the three models on the provided unstable sequences and see the results’, and the following error occurred, how can I solve it?
`From https://github.com/Annbless/DUTCode
Traceback (most recent call last):
File "/content/DUTCode/models/DUT/PWCNet.py", line 14, in
from .correlation import correlation # the custom cost volume layer
ModuleNotFoundError: No module named 'models.DUT.correlation'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/content/DUTCode/models/DIFRINT/pwcNet.py", line 17, in
from models.correlation import correlation # the custom cost volume layer
File "/content/DUTCode/models/correlation/correlation.py", line 273, in
@cupy.util.memoize(for_each_device=True)
File "/usr/local/lib/python3.7/dist-packages/cupy/init.py", line 875, in getattr
f"module 'cupy' has no attribute {name!r}")
AttributeError: module 'cupy' has no attribute 'util'
During handling of the above exception, another exception occurred:
inference with [1, 2, 4, 8, 16, 32]
totally 480 frames for stabilization
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at /pytorch/c10/core/TensorImpl.h:1156.)
return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
length: 10
fps=12.950710894327822
length: 20
fps=15.40031679614381
length: 30
fps=16.42688958201805
length: 40
fps=17.125610864624488
length: 50
fps=17.591498310390687
length: 60
fps=17.907956614992536
length: 70
fps=18.141366974465086
length: 80
fps=18.295022273905573
length: 90
fps=18.413580126169684
length: 100
fps=18.530250318733433
length: 110
fps=18.601868551162436
length: 120
fps=18.713303068323516
length: 130
fps=18.773824996850426
length: 140
fps=18.827789863414793
length: 150
fps=18.84869852294256
length: 160
fps=18.894238953017897
length: 170
fps=18.94839989998453
length: 180
fps=18.98515587712094
length: 190
fps=19.032914027050115
length: 200
fps=19.066544372029053
length: 210
fps=19.109932728253188
length: 220
fps=19.125262363111936
length: 230
fps=19.12811980772301
length: 240
fps=19.142416627959022
length: 250
fps=19.16557507573052
length: 260
fps=19.157523566592534
length: 270
fps=19.17786571572879
length: 280
fps=19.179405902201264
length: 290
fps=19.191366861751973
length: 300
fps=19.1906491851741
length: 310
fps=19.179568576982227
length: 320
fps=19.195247026502646
length: 330
fps=19.208179441868264
length: 340
fps=19.215569600713355
length: 350
fps=19.226686902273496
length: 360
fps=19.232920607360814
length: 370
fps=19.236069136892027
length: 380
fps=19.24671430176637
length: 390
fps=19.257166548749606
length: 400
fps=19.264782372366604
length: 410
fps=19.27072526922419
length: 420
fps=19.26644721940641
length: 430
fps=19.27420648287631
length: 440
fps=19.280633224313547
length: 450
fps=19.292573304446215
length: 460
fps=19.302845257515294
length: 470
fps=19.305520153209212
total length=480
0
0
10
0
20
0
30
0
40
0
50
45792
60
70452
70
70452
80
70452
90
70452
100
70452
110
70452
120
70452
130
70452
140
70452
OpenCV: FFMPEG: tag 0x5634504d/'MP4V' is not supported with codec id 12 and format 'mp4 / MP4 (MPEG-4 Part 14)'
OpenCV: FFMPEG: fallback to use tag 0x7634706d/'mp4v'`
You did a good job.
I want code for Video Stabilization Quality Evaluation ,can you supply it?
Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.