chenyangqiqi / bsvd

[ACM MM 2022] Real-time Streaming Video Denoising with Bidirectional Buffers

BSVD

Official implementation for Real-time Streaming Video Denoising with Bidirectional Buffers (ACM MM 2022)

project website | paper

This branch is for synthetic noise removal on the DAVIS dataset. We also provide the code for real raw noise removal here.

---

Dependencies and Installation

Environment

This code is based on PyTorch and BasicSR repo. It has been tested on Ubuntu 18.04 LTS and CUDA 11.1.

Create a new conda environment with:

conda create --name bsvd python=3.7
conda activate bsvd

We recommend you install this repo locally as:

pip install -r requirements.txt
cd BasicSR
python setup.py develop
cd ..

if you use CUDA 11 and get a runtime error. Reinstall torch as:

pip uninstall torch
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

We use a Nvidia DALI loader to load video frames.

pip install --extra-index-url https://developer.download.nvidia.com/compute/redist/ nvidia-dali-cuda110==1.12.0

Our test environment is CUDA 11.1. You may find the DALI loader for other CUDA versions from their github

Data

Download the DAVIS (for train and test) and Set8 dataset (for test) from onedrive. Keep the folder structure as

--bsvd
 |--BasicSR
 |--datasets
    |--DAVIS-training-mp4
    |--DAVIS-2017-test-dev-480p
    |--Set8
 |--Experimental_root
 |--***.py

---

Test

You may also download the pretrained checkpoint Put bsvd_64.pth under .experiments/pretrained_ckpt. Then, run the command to produce the result in the Table 2 from paper

python ./run_test.py -opt ./options/test/bsvd_c64.yml

Two checkpoints are also provided for blind and unidirection

---

Train

CUDA_VISIBLE_DEVICES=0,1 python ./run.py -opt ./options/train/bsvd_c64_unblind.yml

We train the model on 2 RTX 3090 GPUs for 2 days. Here is one example log and tensorboard for our training.

---

Profile the model

Type in the command line

python profile.py

Example output

__file__ profile.py
Disable distributed.
device_name CUDA
os.environ["CUDA_VISIBLE_DEVICES"] 2
device cuda
load from ./experiments/pretrained_ckpt/bsvd-64.pth
<Experimental_root.models.denoising_model.DenoisingModel object at 0x7f3187110710>
size of the tensor torch.Size([1, 10, 4, 540, 960])
use device  cuda
output shape is torch.Size([1, 10, 3, 540, 960])
size of the tensor torch.Size([1, 10, 4, 540, 960])
use device cuda

test function name: <class 'Experimental_root.archs.bsvd_arch.BSVD'>
10 loops, mean of best 1: 0.353594 sec per loop
max memory required              2.26GB

Codes Development

We support all functions provided by BasicSR natively. You may check the docs of BasicSR for details of file structure and installation.

To extend functions, please create your codes here:

/Experimental_root
|- models       # Custom Training and validation Behaviour
|- archs        # Custom network architectures
|- data         # Custom dataloader

Citation

If you find this work useful for your research, please cite:

@inproceedings{qi2022BSVD,
    title={Real-time Streaming Video Denoising with Bidirectional Buffers},
    author={Chenyang Qi and Junming Chen and Xin Yang and Qifeng Chen},
    booktitle = {ACM MM},
    year={2022}
    }

Contact

Please contact us if there is any question (Chenyang QI, [email protected]; Junming CHEN, [email protected])

bsvd's Issues

How to run on my images/videos?

Could PLEASE write a mode detailed README for non-researchers, who aren't interested in reproducing your results, but would like to see the algorithm on my own test data. It's so frustrating.

How to validate on raw datas(CRVD)?

I see quantitative comparisons of PSNR and SSIM on the CRVD test set for raw image denosing. How can we test this ? Thank you~

inference memory

in the paper, "Unlike MIMO frameworks, our buffers work in a pipeline that consumes frames one by one. Therefore, our method can keep a constant run-time memory, which depends only on the number of Bidirectional Buffer Blocks 𝑁."
but,when testing,for 20 input images, the memory takes about 10G;for 40 input images, the memory takes about 14G.
Please, What's the reason?

about noise_map

In bsvd_arch.py ,
if noise_map != None:
input = torch.cat([input, noise_map], dim=2)
please, what does the noise_map do?

Log Files from Training

Thank you for your awesome code!

I am hoping you might open-source the log files you have from training. Maybe the training and validation loss as a function of epoch (and/or batch) with an estimate of the runtime?

How to inference ?

You've done excellent research, I'm having trouble inference right now, can you provide an denoising example program for a noise reduction video stream？thanks

inference on noisy videos

Hello,
You have done excellent work,
I'm trying to run your code on my noisy video sequences. Im using this command line:
python ./run_test.py -opt ./options/test/bsvd_c64.yml
I changed the images path to my noisy images and the system doesn't seem to clean the noise. Im providing some examples of my images.
please let me know what is wrong with my attempts, and what checkpoint is the best to be used in my case (blind, unidirection or other), and what parameter setting should I be using on my images.

Thanks a lot !

here is some of my images:
sequence 1:

sequence 2:

Do other pretrained checkpoints exist?

Hi,

BSVD-32?

In the following, “BSVD-32” denotes our model whose smallest feature channel is “32”.

However, in the repository, I cannot find the checkpoint of BSVD-32. Can you provide it?
I wanna insert this model into my realtime pipeline, however BSVD-64 seems bit slower than realtime requirement.

Optimization?

Took about 110ms for a 1x3x4x960x540 image with 3080@FP16; I wanna reduce this by at least about 15ms. I did not apply TensorRT yet.
For my requirement is it good to train a new model with much lower channels? (about 16??)

How to generate input?

Also I think I implement the wrong model input, and I cannot find good example in the code. Can you suggest any example?

Thank you for your great work!

Question about Bidirectional Buffer Block

you said in the paper： For time step 𝑖 < 𝑁, each input frame activates one deeper Bidirectional Buffer Block 𝐵0,𝑖 in our network. Therefore, when the first denoised frame 𝑦0 comes out, there are already 𝑁 + 1 frames fed into our inference block.

What is the number of Bidirectional Buffer Block's? 16？

Why can a frame of input also produce output when I test it?

Thank you very much！

Uni-directional model

Hello,

Is it possible to provide the uni-directional pre-trained model?

Best regards

Can I use in windows?

I would like to test a new model, but I am unable to install Nvidia-DALI in a Windows environment. Is there a way to use a model with DALI in a Windows environment?

Request for a checkpoint without noise_map for blind video test

Hi,
Thank you so much for a great job.It really help me a lot!
I want to do some learning based on your code，and try to test some videos with nature noise.I see you have tested:Blind denoising on DAVIS and Set8 datasets in your paper.

Can you help send a link of a checkpoint without noise map input?
Thank you very much!