How to run inference on larger frames e.g. 360p? about vrt

[JingyunLiang]

If out-of-memory, try to reduce --tile at the expense of slightly decreased performance. For example, --tile 40 128 128 means dividing the video into 40x128x128 (40 frames, height 128, width 128) clips in testing.

[machinelearnear]

Hi @JingyunLiang ! Thanks for the quick reply. Unfortunately, regardless of the tile size that I use, I'm still getting an OOM error. Here's some sample code to replicate, first to download a video from YT and extract frames, and then to run main_test_vrt.py against that set of frames, with different tile sizes. I used the lowest value == args.window_size as an example. When I run this against anything 180p or below, it works no problem. Any advice would be appreciated, thanks!

from pathlib import Path
import shutil, os

if not os.path.exists("data"): os.makedirs("data")
YouTubeID = 'kgCbG0q4jmc' # random 360p video
OutputFile = 'data/video.mp4'
!youtube-dl -o $OutputFile $YouTubeID --restrict-filenames -f 'best[filesize<50M]'
!rm -r testsets/*
!rm -r results/*
inputFolder = 'testsets/uploaded/000'
Path(inputFolder).mkdir(parents=True, exist_ok=True)
print(f'extracting the video as frames to `{str(inputFolder)}`')
os.system(f'ffmpeg -i {OutputFile} -qscale:v 1 -qmin 1 -qmax 1 -vsync 0  {inputFolder}/frame%08d.png')

Then running your test code

!python main_test_vrt.py --task 001_VRT_videosr_bi_REDS_6frames --folder_lq testsets/uploaded --tile 6 8 8 --tile_overlap 2 20 20

[JingyunLiang]

Maybe loading the whole video into GPU consumes too much GPU memory?

[machinelearnear]

You were right. Loading up more than 5 seconds clips was killing the GPU. Using --tile 32 64 64, I can now process 125 frames (5 seconds) with size 480x360 in a little bit over 1hr on an NVIDIA T4. Is that expected?

[JingyunLiang]

Yes, it is slow is if you test it patch by patch. Testing different patches in parallel can help.