coupriec / videopredictioniclr2016 Goto Github PK

Hi,

I'm having problems using the GDL criterion. The error stack states
../torch/install/share/lua/5.1/nn/CSubTable.lua:11: bad argument #2 to 'add' (sizes do not match at /tmp/luarocks_cutorch-scm-1-7235/cutorch/lib/THC/generated/../generic/THCTensorMathPointwise.cu:216)
stack traceback:
[C]: in function 'add'
../torch/install/share/lua/5.1/nn/CSubTable.lua:11: in function 'func'
../torch/install/share/lua/5.1/nngraph/gmodule.lua:346: in function 'neteval'
../torch/install/share/lua/5.1/nngraph/gmodule.lua:381: in function 'updateOutput'
./GDLCrit.lua:40: in function 'forward'

The size of the output does indeed change (it's smaller by one pixel). Before the target label and the output of the network is forwarded through the gdl loss they are both the same size. I haven't been able to figure out where the size change occurs. Any help would be really appreciated.

@MichaelMathieu:
Though not a code issue - but my query is like this:
I have an existing MATLAB (say, something.m file) and I need that MATLAB file to be called from the torch file:
th train_iclr_model.lua.

To be more specific: I need a function within train_iclr_model.lua - that will call something.m and will use something.m results (like: mat file) as a part of Video Prediction.

For example: th train_iclr_model.lua ---> within train_iclr_model.lua I need a function `CALL_MATLAB_M_FILE()

end`

May you please help me to figure out that..

Regards

Hi @MichaelMathieu , I have a small doubt: Is your generator G an autoencoder (X->z->X') or its a z-->X likelihood network. I mean generator of Adversarial Network.
Here z: sample from the distribution, and, X: generated future frame.

Hi,
I tried to train network, that you provide its code (train_iclr_model.lua), on the UCF101 dataset, then I tried to evaluate the trained model using your test code but I got the following error:

/home/user/torch/install/bin/luajit: ...torch/install/share/lua/5.1/cudnn/SpatialConvolution.lua:173: attempt to call method 'isContiguous' (a nil value)
stack traceback:
...torch/install/share/lua/5.1/cudnn/SpatialConvolution.lua:173: in function 'makeContiguous'
...torch/install/share/lua/5.1/cudnn/SpatialConvolution.lua:187: in function 'func'
/home/user/torch/install/share/lua/5.1/nngraph/gmodule.lua:345: in function 'neteval'
/home/user/torch/install/share/lua/5.1/nngraph/gmodule.lua:380: in function 'func'
/home/user/torch/install/share/lua/5.1/nngraph/gmodule.lua:345: in function 'neteval'
/home/user/torch/install/share/lua/5.1/nngraph/gmodule.lua:380: in function 'forward'
test-frame-prediction-on-ucf-rec_gdl.lua:204: in main chunk
[C]: in function 'dofile'
...user/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50

I would really appreciate it if you can tell me what's going on.

Thanks for sharing this fantastic code! I'm having trouble understanding the part of psnr calculation for motion region in PSNR function. The denominator for calculation of MSE used in the code is the number of all pixels in one image and shouldn't it be the number of pixels only in motion region?

Hi,
I am trying to run test code but I get the following error:

/home/user/torch/install/bin/luajit: /home/user/torch/install/share/lua/5.1/trepl/init.lua:389: /home/user/torch/install/share/lua/5.1/trepl/init.lua:389: module 'fb.util' not found:No LuaRocks module found for fb.util
no field package.preload['fb.util']
no file '/home/user/.luarocks/share/lua/5.1/fb/util.lua'
no file '/home/user/.luarocks/share/lua/5.1/fb/util/init.lua'
no file '/home/user/torch/install/share/lua/5.1/fb/util.lua'
no file '/home/user/torch/install/share/lua/5.1/fb/util/init.lua'
no file './fb/util.lua'
no file '/home/user/torch/install/share/luajit-2.1.0-beta1/fb/util.lua'
no file '/usr/local/share/lua/5.1/fb/util.lua'
no file '/usr/local/share/lua/5.1/fb/util/init.lua'
no file '/home/user/.luarocks/lib/lua/5.1/fb/util.so'
no file '/home/user/torch/install/lib/lua/5.1/fb/util.so'
no file '/home/user/torch/install/lib/fb/util.so'
no file './fb/util.so'
no file '/usr/local/lib/lua/5.1/fb/util.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
no file '/home/user/.luarocks/lib/lua/5.1/fb.so'
no file '/home/user/torch/install/lib/lua/5.1/fb.so'
no file '/home/user/torch/install/lib/fb.so'
no file './fb.so'
no file '/usr/local/lib/lua/5.1/fb.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
[C]: in function 'error'
/home/user/torch/install/share/lua/5.1/trepl/init.lua:389: in function 'require'
test-frame-prediction-on-ucf-rec_gdl.lua:11: in main chunk
[C]: in function 'dofile'
...user/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50

then I tried to install fblualib by using the one-step installation script as provided in the fblualib github page here: https://github.com/facebook/fblualib/blob/master/INSTALL.md and https://github.com/facebookarchive/fblualib/blob/master/install_all.sh but it stops and shows this massage:

• echo Building fbthrift
  Building fbthrift

  • echo

  • cd /tmp/fblualib-build.8taTVB/fbthrift/thrift

  • autoreconf -ivf
    autoreconf: 'configure.ac' or 'configure.in' is required

I would really appreciate it if you can tell me what's going on.

Is it possible to share the model fine-tuned on UCF101?
Thanks

Hi @MichaelMathieu: I got the error;
error loading module 'libthffmpeg' from file '/home/intern_vnb/torch/install/lib/lua/5.1/libthffmpeg.so'

The error is, when I tried:
require 'thffmpeg'
/home/intern_vnb/torch/install/share/lua/5.1/trepl/init.lua:384: /home/intern_vnb/torch/install/share/lua/5.1/trepl/init.lua:384: ...tern_vnb/torch/install/share/lua/5.1/luarocks/loader.lua:117: error loading module 'libthffmpeg' from file '/home/intern_vnb/torch/install/lib/lua/5.1/libthffmpeg.so': /home/intern_vnb/torch/install/lib/lua/5.1/libthffmpeg.so: undefined symbol: av_frame_free stack traceback: [C]: in function 'error' /home/intern_vnb/torch/install/share/lua/5.1/trepl/init.lua:384: in function 'require' [string "_RESULT={require 'thffmpeg'}"]:1: in main chunk [C]: in function 'xpcall' /home/intern_vnb/torch/install/share/lua/5.1/trepl/init.lua:652: in function 'repl' ..._vnb/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:199: in main chunk [C]: at 0x00406670

I have changed the permission of: libthffmpeg.so at that location (i.e. '/home/intern_vnb/torch/install/lib/lua/5.1/libthffmpeg.so').
Another thing: I can see libthffmpeg.so file at THFFmpeg/build location too

Dear @MichaelMathieu sorry to bother you, but this time I got the following error when I was trying:
luarocks make at THFFmpeg folder:

`
Warning: unmatched variable LUALIB
cmake -E make_directory build && cd build && cmake .. -DLUALIB= -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH="/home/intern_vnb/torch/install/bin/.." -DCMAKE_INSTALL_PREFIX="/home/intern_vnb/torch/install/lib/luarocks/rocks/thffmpeg/1.0.alpha-0" && make

-- The C compiler identification is GNU 4.7.3
-- The CXX compiler identification is GNU 5.2.0
-- Check for working C compiler: /usr/local/bin/gcc
-- Check for working C compiler: /usr/local/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/local/bin/c++
-- Check for working CXX compiler: /usr/local/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found Torch7 in /home/intern_vnb/torch/install
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.26")
-- checking for module 'libavcodec'
-- found libavcodec, version 57.48.101
-- checking for module 'libavformat'
-- found libavformat, version 57.41.100
-- checking for module 'libavutil'
-- found libavutil, version 55.28.100
-- Found FFMPEG or Libav: /usr/local/lib/libavcodec.a;/usr/local/lib/libavformat.a;/usr/local/lib/libavutil.a, /usr/local/include
-- Configuring done
-- Generating done
CMake Warning:
Manually-specified variables were not used by the project:

LUALIB

-- Build files have been written to: /home/intern_vnb/Arghya/VideoPredictionICLR2016/THFFmpeg/build
Scanning dependencies of target thffmpeg
[100%] Building C object CMakeFiles/thffmpeg.dir/thffmpeg.c.o
/home/intern_vnb/Arghya/VideoPredictionICLR2016/THFFmpeg/thffmpeg.c: In function ‘AV_open’:
/home/intern_vnb/Arghya/VideoPredictionICLR2016/THFFmpeg/thffmpeg.c:145:5: warning: ‘codec’ is deprecated (declared at /usr/local/include/libavformat/avformat.h:880) [-Wdeprecated-declarations]
/home/intern_vnb/Arghya/VideoPredictionICLR2016/THFFmpeg/thffmpeg.c:153:3: warning: ‘codec’ is deprecated (declared at /usr/local/include/libavformat/avformat.h:880) [-Wdeprecated-declarations]
/home/intern_vnb/Arghya/VideoPredictionICLR2016/THFFmpeg/thffmpeg.c:172:3: warning: ‘avpicture_get_size’ is deprecated (declared at /usr/local/include/libavcodec/avcodec.h:5406) [-Wdeprecated-declarations]
/home/intern_vnb/Arghya/VideoPredictionICLR2016/THFFmpeg/thffmpeg.c:172:37: error: ‘PIX_FMT_RGB24’ undeclared (first use in this function)
/home/intern_vnb/Arghya/VideoPredictionICLR2016/THFFmpeg/thffmpeg.c:172:37: note: each undeclared identifier is reported only once for each function it appears in
/home/intern_vnb/Arghya/VideoPredictionICLR2016/THFFmpeg/thffmpeg.c:190:3: warning: ‘avpicture_fill’ is deprecated (declared at /usr/local/include/libavcodec/avcodec.h:5391) [-Wdeprecated-declarations]
/home/intern_vnb/Arghya/VideoPredictionICLR2016/THFFmpeg/thffmpeg.c: In function ‘AV_read_frame’:
/home/intern_vnb/Arghya/VideoPredictionICLR2016/THFFmpeg/thffmpeg.c:203:7: warning: ‘avcodec_decode_video2’ is deprecated (declared at /usr/local/include/libavcodec/avcodec.h:4753) [-Wdeprecated-declarations]
/home/intern_vnb/Arghya/VideoPredictionICLR2016/THFFmpeg/thffmpeg.c:212:2: warning: ‘av_free_packet’ is deprecated (declared at /usr/local/include/libavcodec/avcodec.h:4414) [-Wdeprecated-declarations]
/home/intern_vnb/Arghya/VideoPredictionICLR2016/THFFmpeg/thffmpeg.c:217:5: warning: ‘av_free_packet’ is deprecated (declared at /usr/local/include/libavcodec/avcodec.h:4414) [-Wdeprecated-declarations]
/home/intern_vnb/Arghya/VideoPredictionICLR2016/THFFmpeg/thffmpeg.c: In function ‘AV_skip_frame’:
/home/intern_vnb/Arghya/VideoPredictionICLR2016/THFFmpeg/thffmpeg.c:227:7: warning: ‘avcodec_decode_video2’ is deprecated (declared at /usr/local/include/libavcodec/avcodec.h:4753) [-Wdeprecated-declarations]
/home/intern_vnb/Arghya/VideoPredictionICLR2016/THFFmpeg/thffmpeg.c:229:2: warning: ‘av_free_packet’ is deprecated (declared at /usr/local/include/libavcodec/avcodec.h:4414) [-Wdeprecated-declarations]
/home/intern_vnb/Arghya/VideoPredictionICLR2016/THFFmpeg/thffmpeg.c:233:5: warning: ‘av_free_packet’ is deprecated (declared at /usr/local/include/libavcodec/avcodec.h:4414) [-Wdeprecated-declarations]
make[2]: *** [CMakeFiles/thffmpeg.dir/thffmpeg.c.o] Error 1
make[1]: *** [CMakeFiles/thffmpeg.dir/all] Error 2
make: *** [all] Error 2

Error: Build error: Failed building.
`

I have done:
Deleted build folder and tried again "luarocks make", but no luck

My Ubuntu version is 14.04
I have cloned THFFmpeg from "https://github.com/MichaelMathieu/THFFmpeg.git"

Regards,
Arghya.

As I understand absolute values of L2 loss and GDL loss will have large difference, but in paper lambda coefficient is 1.0 for both. Is it normal?

Here I see lambda 1.0 for L2 loss and lambda 0.01 for adversarial loss:

Also looks like alpha is 1 for GDL loss:

Looks related:
#7 (comment)

@MichaelMathieu

Your link does't work. Can you check it?
http://perso.esiee.fr/~coupriec/MathieuICLR16TestCode.zip

@MichaelMathieu

When I tried th test-frame-prediction-on-ucf-rec_gdl.lua

I got the error:

gm (GraphicsMagick) binary not found, please install (see www.graphicsmagick.org) THCudaCheck FAIL file=/home/dl-box/torch/extra/cutorch/lib/THC/generic/THCStorage.cu line=65 error=2 : out of memory /home/dl-box/torch/install/bin/luajit: /home/dl-box/torch/install/share/lua/5.1/torch/File.lua:351: cuda runtime error (2) : out of memory at /home/dl-box/torch/extra/cutorch/lib/THC/generic/THCStorage.cu:65 stack traceback: [C]: in function 'read' /home/dl-box/torch/install/share/lua/5.1/torch/File.lua:351: in function </home/dl-box/torch/install/share/lua/5.1/torch/File.lua:245> [C]: in function 'read' /home/dl-box/torch/install/share/lua/5.1/torch/File.lua:351: in function 'readObject' /home/dl-box/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject' /home/dl-box/torch/install/share/lua/5.1/nn/Module.lua:184: in function 'read' /home/dl-box/torch/install/share/lua/5.1/torch/File.lua:351: in function 'readObject' /home/dl-box/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject' /home/dl-box/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject' /home/dl-box/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject' /home/dl-box/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject' ... /home/dl-box/torch/install/share/lua/5.1/torch/File.lua:353: in function 'readObject' /home/dl-box/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject' ...e/dl-box/torch/install/share/lua/5.1/nngraph/gmodule.lua:495: in function 'read' /home/dl-box/torch/install/share/lua/5.1/torch/File.lua:351: in function 'readObject' /home/dl-box/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject' /home/dl-box/torch/install/share/lua/5.1/torch/File.lua:409: in function 'load' test-frame-prediction-on-ucf-rec_gdl.lua:59: in main chunk [C]: in function 'dofile' ...-box/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk [C]: at 0x00406670

I think the problem is that, I am out of memory.
Now, from where I can reduce the network size?

Hi!
I'm trying out the code, It woks fine, but I found something weird in line 144 of train_iclr_model.lua where it states that:
local w, h = opt.h, opt.w
Which seems to me is inverting the dimensions. Is that intended behavior?
Thanks in advance.