- Author : Pablo Arias [email protected], see
AUTHORS
- Copyright : (C) 2019, Pablo Arias [email protected]
This code provides an implementation of the video denoising methods described in:
Please cite the paper if you use results obtained with this code in your research.
The code is in C with some BASH helper scripts. Known dependencies are:
- OpenMP: parallelization [optional, but recommended]
- libpng, libtiff and libjpeg: image i/o
- GNU parallel: parallelization in some helper scripts
Compilation was tested on Ubuntu Linux 16.04 and 18.04. Configure and compile the source code using cmake and make. It is recommended that you create a folder for building:
$ mkdir build; cd build
$ cmake ..
$ make
NOTE: By default, the code is compiled with OpenMP multithreaded
parallelization enabled (if your system supports it). Use the
OMP_NUM_THREADS
enviroment variable to control the number of threads
used.
The compilation populates build/bin
with the following binaries:
nlkalman-flt
non-local Kalman filtering of a framenlkalman-smo
RTS smoother of a frametvl1flow
compute TV-L1 optical flow between two imagesawgn
add noise to an imageiion
convert image to a different formatimprintf
display statistics of an image in printf formatplambda
evaluate lambda expression at all pixels of an image.decompose
DCT pyramid decompositionrecompose
recomposition from a DCT pyramid
In addition, the following helper scripts will be installed in bin/
nlkalman-seq.sh
computes NL-Kalman filtering (and optionally) the smoothing over a noisy image sequence.nlkalman-seq-gt.sh
given a clean sequence, adds noise, runsnlkalman-seq.sh
and computes PSNR.msnlkalman-seq.sh
multiscale version of nlkalman-seq.sh (experimental)msnlkalman-seq-gt.sh
given a clean sequence, adds noise, runsmsnlkalman-seq.sh
and computes PSNR.psnr.sh
computes MSE/RMSE/PSNR between two images
Denoising a noisy sequence
The simplest use is via the helper scripts:
nlkalman-seq.sh /my/video/frames-%03d.png first-frame last-frame sigma out-folder [filt-params] [smoo-params] [flow-params]
The method reads the video as a sequence of images. The sequence of images is passed
as a pattern in printf format, thus frame-%03d.png
means that frames have the following
filenames: frame-001.png
, frame-002.png
, etc. The first and last frame
numbers have to given, as well as the standard deviation of the noise.
The denoising results are stored in the out-folder. The script produces the following
output sequences:
bflo_%03d.flo
: backward optical flow (ie flow from frame t to t-1)bocc_%03d.png
: masks of backwards occluded pixelsflt1_%03d.tif
: output of 1st NL-Kalman filtering iterationflt2_%03d.tif
: output of 2nd NL-Kalman filtering iteration (if 2nd iteration is enabled)
If smoothing is performed, the following additional sequences will also be left in out-folder
fflo_%03d.flo
: forward optical flow (from from frame t to t+1)focc_%03d.png
: masks of forward occluded pixelssmo1_%03d.tif
: output of the smoothing pass
You can pass options to the filtering and the smoothing thought the optional
arguments [filt-params]
and [smoo-params]
. For a list of
all parameters run nlkalman-flt -h
and nlkalman-smo -h
. If no parameters are
given, the parameters are set automatically based on the noise level sigma
. The
filtering and smoothing parameters have to be passed between quotes.
Some examples:
# Run the denoising with automatic parameters from frame 3 to 56 with noise 10.
nlkalman-seq.sh /my/video/frames-%03d.png 3 56 10 out/path
# Set patch size during both filtering iterations at 12x12, toggle verbose output:
nlkalman-seq.sh /my/video/frames-%03d.png 3 56 10 out/path "--f1_p 12 --f2_p 12 -v 1"
# Filter with automatic parametes, smoothing with a patch size of 6x6
nlkalman-seq.sh /my/video/frames-%03d.png 3 56 10 out/path "" "--s1_p 6"
# Filter with automatic parametes, do not enable smoothing
nlkalman-seq.sh /my/video/frames-%03d.png 3 56 10 out/path "" "no"
Finally, you can also provide a string with parameters for the optical flow and occlusions detection. The string has to have 6 numbers, three parameters for the backward optical flow computed during filtering and three for the forward flow computed for the smoothing pass:
"fscale-filt data-weight-filt occl-th-filt fscale-smoo data-weight-smoo occl-th-smoo"
fscale
finest scale of the multiscale TV-L1:0
means the finest scale, and1
means that the optical flow is computed at half resolution and then upscaled (default is1
).data-weight
data-attachment weight to control the smoothness of the flow (default is0.25
)occl-th
threshold on the divergence of the flow use to compute occlusions (default is0.75
)
For example, to run the denoising with automatic filtering and smoothing parameters but with custom parameters for the optical flows
nlkalman-seq.sh /my/video/frames-%03d.png 3 56 10 out/path "" "" "1 0.2 .75 0 0.2 0.75"
Add noise, denoising and compute PSNR
Finally, if you want to compute the flow on a sequence with synthetic noise and then compute the PSNR on the result, you can use:
nlkalman-seq.sh /my/clean/video/frames-%03d.png first-frame last-frame sigma out-folder [filt-params] [smoo-params] [flow-params]
In addition to the previous outputs, you will find in out-folder
:
out-folder/%03d.tif
: frames with noise added (as tif floating point images)out-folder/measures
: text file with RMSE and PSNR computed globally and per-frame
The following libraries are also included as part of the code:
- For computing the optical flow: the IPOL implementation of the TV-L1 optical flow method of Zack et al..
- For image I/O: Enric Meinhardt's iio.
- For basic image manipulation: a reduced version of Enric Meinhardt's imscript.
- For command line parsing: Yecheng Fu's argparse.
- For multiscale denoising: Pierazzo and Facciolo's DCT multiscaler
The project is organized as follows
root/
├── lib/ 3rd party libraries
├── scripts/ helper scripts
└── src/ kalman filtering and smoothing code
The code of BNLK is licensed under the GNU Affero General Public License v3.0,
see LICENSE
. The 3rd party libraries are distributed under their own licences
specified inside each folder.