baidut / patchvq

Patch-VQ: ‘Patching Up’ the Video Quality Problem

PatchVQ

Patch-VQ: ‘Patching Up’ the Video Quality Problem

Please email yingzhenqiang at gmail dot com for any questions. Thank you!

Demo

Please follow this to test the Patch VQ model pretrained on the LSVQ database. Please follow this to test our Patch VQ model on your database.

Note

• Due to the breaking changes in fastai 2.0.18 --> 2.1, the code is incompatible with the latest fastai! Please make sure the following versions are installed:

  fastai 2.0.18
  fastcore 1.2.5
  torch 1.6.0
  torchvision 0.7.0
  

​ a new version is still work-in-progress. See here.

• To reproduce the results in the paper, make sure to use the pretrained weights provided here:
  • RoIPoolModel-fit.10.bs.120.pth for PaQ2PiQ (don't use weights from other sources)
  • fastai-r3d18_K_200ep.pth for resnet 3d (don't use pytorch builtin r3d_18)

Common Issues:

• NameError: name '_C' is not defined --> pip3 install Cython
• RuntimeError: Could not infer dtype of PILImage --> downgrade: pip install pillow==8.2
• AttributeError: module 'PIL.Image' has no attribute 'Resampling' --> check your fastai and fastcore version
• ImportError: cannot import name 'default_generator' from 'torch._C' (unknown location) --> pip install torch==1.6.0 torchvision==0.7.0

Download LSVQ database

Description

No-reference (NR) perceptual video quality assessment (VQA) is a complex, unsolved, and important problem to social and streaming media applications. Efficient and accurate video quality predictors are needed to monitor and guide the processing of billions of shared, often imperfect, user-generated content (UGC). Unfortunately, current NR models are limited in their prediction capabilities on real-world, "in-the-wild" UGC video data. To advance progress on this problem, we created the largest (by far) subjective video quality dataset, containing 39, 000 real-world distorted videos and 117, 000 space-time localized video patches ("v-patches"), and 5.5M human perceptual quality annotations. Using this, we created two unique NR-VQA models: (a) a local-to-global region-based NR VQA architecture (called PVQ) that learns to predict global video quality and achieves state-of-the-art performance on 3 UGC datasets, and (b) a first-of-a-kind space-time video quality mapping engine (called PVQ Mapper) that helps localize and visualize perceptual distortions in space and time. We will make the new database and prediction models available immediately following the review process.

Investigators

• Zhenqiang Ying ([email protected]) -- Graduate Student, Dept. of ECE, UT Austin
• Maniratnam Mandal ([email protected]) -- Graduate Student, Dept. of ECE, UT Austin
• Deepti Ghadiyaram ([email protected]), Facebook Inc.
• Alan Bovik ([email protected]) -- Professor, Dept. of ECE, UT Austin

Download

We are making the LSVQ Database available to the research community free of charge. If you use this database in your research, we kindly ask that you reference our papers listed below:

• Z. Ying, M. Mandal, D. Ghadiyaram and A.C. Bovik, "Patch-VQ: ‘Patching Up’ the Video Quality Problem," arXiv 2020.[paper]
• Z. Ying, M. Mandal, D. Ghadiyaram and A.C. Bovik, "LIVE Large-Scale Social Video Quality (LSVQ) Database", Online:https://github.com/baidut/PatchVQ, 2020.

Please fill THIS FORM to download our database.

1. follow 'download_from_internetarchive.ipynb' to download Internet archive videos
2. download YFCC videos from Box (The password will be sent to your email after you submit the request form.)
3. download label files (coordinates and scores).
   • labels_test_1080p.csv 1.05 MB
   • labels_train_test.csv 10.8 MB (is_test column denotes if a video is in the train set or the test set )
4. [optional] follow this crop patches from videos

Copyright Notice

-----------COPYRIGHT NOTICE STARTS WITH THIS LINE------------ Copyright (c) 2020 The University of Texas at Austin All rights reserved.

Permission is hereby granted, without written agreement and without license or royalty fees, to use, copy, modify, and distribute this database (the images, the results and the source files) and its documentation for any purpose, provided that the copyright notice in its entirety appear in all copies of this database, and the original source of this database, Laboratory for Image and Video Engineering (LIVE, http://live.ece.utexas.edu ) at the University of Texas at Austin (UT Austin, http://www.utexas.edu ), is acknowledged in any publication that reports research using this database.

The following papers are to be cited in the bibliography whenever the database is used as:

• Z. Ying, M. Mandal, D. Ghadiyaram and A.C. Bovik, "Patch-VQ: ‘Patching Up’ the Video Quality Problem," arXiv 2020.[paper]
• Z. Ying, M. Mandal, D. Ghadiyaram and A.C. Bovik, "LIVE Large-Scale Social Video Quality (LSVQ) Database", Online:https://github.com/baidut/PatchVQ, 2020.

IN NO EVENT SHALL THE UNIVERSITY OF TEXAS AT AUSTIN BE LIABLE TO ANY PARTY FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OF THIS DATABASE AND ITS DOCUMENTATION, EVEN IF THE UNIVERSITY OF TEXAS AT AUSTIN HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

THE UNIVERSITY OF TEXAS AT AUSTIN SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE DATABASE PROVIDED HEREUNDER IS ON AN "AS IS" BASIS, AND THE UNIVERSITY OF TEXAS AT AUSTIN HAS NO OBLIGATION TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.

-----------COPYRIGHT NOTICE ENDS WITH THIS LINE------------

patchvq's Issues

How can I download database?

'download_from_internetarchive.ipynb'cannot open.

Number of video and video duration

Hi @baidut,

Thanks for your great effort in constructing the excellent UGC VQA dataset. I encounter some problems when I use the dataset. I appreciate any help you can provide.

1. As reported in the manuscript, the number of unique contents is 38,811. However, 35,499 videos in the labels_train_test.csv plus 3,573 videos in the labels_test_1080p.csv is 39,072, which is not equal to the given number in the paper. Thus, could you please help me figure out which one is the correct number, or maybe there is something I missed or understood incorrectly?
2. In the published labels_train_test.csv, I notice that the frame number of some videos (from the row 35,241 to the end) are 0 or 1 that the duration of these video are not 5-12s. I am wondering whether these videos belong to the LSVQ dataset.

Best regards

How can I download LSVQ dataset.

Since the form has been disabled.

Error running "test_PVQ_on_new_datasets.ipynb"

There is an error when I run "test_PVQ_on_new_datasets.ipynb" on Colab. I only changed the "dir" parameter sent to SingleVideo2MOS.from_dict. Could you help me with this? Thank you very much.

WARNING:root:setattr roi_col not existed
WARNING:root:roipool: size = (2, 2), scale = 1/32
WARNING:root:setattr vid_id not existed

0%| | 0/22 [00:00<?, ?it/s]WARNING:root:setattr split_mode not existed
Loading data...G005

AttributeError Traceback (most recent call last)
in ()
32 e = IqaExp('exp_features', gpu=0, seed=None)
33 e[name] = learn
---> 34 e.run(lambda x: [get_features(x, name, bs=bs, vid_id=vid_id) for vid_id in tqdm(vid_list)])

7 frames
/content/PatchVQ/fastiqa/iqa_exp.py in run(self, func, destroy, append)
149
150 if type(self.gpu) is int:
--> 151 func(learn)
152 else: # None or True/False (cpu or multi-gpus)
153 ctx = learn.parallel_ctx

in (x)
32 e = IqaExp('exp_features', gpu=0, seed=None)
33 e[name] = learn
---> 34 e.run(lambda x: [get_features(x, name, bs=bs, vid_id=vid_id) for vid_id in tqdm(vid_list)])

in (.0)
32 e = IqaExp('exp_features', gpu=0, seed=None)
33 e[name] = learn
---> 34 e.run(lambda x: [get_features(x, name, bs=bs, vid_id=vid_id) for vid_id in tqdm(vid_list)])

in get_features(x, name, bs, vid_id)
9 x.dls.bs = bs
10 x.dls.set_vid_id(vid_id)
---> 11 x.extract_features(name=name, skip_exist=True)
12 except RuntimeError:
13 tmp_bs = bs

/content/PatchVQ/fastiqa/learn.py in extract_features(self, on, name, cache, skip_exist)
122 if hasattr(self.model, 'rois'):
123 self.model.rois = None
--> 124 preds = self.get_preds() # get preds on one video with several batches
125 self.model.output_features = old_setting
126

/usr/local/lib/python3.7/dist-packages/fastai/learner.py in get_preds(self, ds_idx, dl, with_input, with_decoded, with_loss, act, inner, reorder, cbs, **kwargs)
225 def get_preds(self, ds_idx=1, dl=None, with_input=False, with_decoded=False, with_loss=False, act=None,
226 inner=False, reorder=True, cbs=None, **kwargs):
--> 227 if dl is None: dl = self.dls[ds_idx].new(shuffled=False, drop_last=False)
228 if reorder and hasattr(dl, 'get_idxs'):
229 idxs = dl.get_idxs()

/content/PatchVQ/fastiqa/bunch.py in getitem(self, index)
156 if self._loading or self._bunching:
157 raise AttributeError(f"index={index}: '{self}' data not loaded yet. loading={self._loading}, bunching={self._bunching}")
--> 158 return self._data[index]
159
160 def getattr(self, k: str):

/content/PatchVQ/fastiqa/bunch.py in getattr(self, k)
161 # fail to get attributes, fetch it from self.data
162 if self._loading or self._bunching:
--> 163 raise AttributeError(f"key={k}: '{self}' data not loaded yet. loading={self._loading}, bunching={self._bunching}")
164 return getattr(self._data, k)
165

AttributeError: key=_data: 'G005' data not loaded yet. loading=True, bunching=False

I have extract the frames for G005
!ls data/LIVE_VQC_tiny/jpg/G005 | wc -l
#return: 301

Dataset download

Hi,

I have filled the google form and have been waiting for few hours, but I still haven't received the password to access the box. May I know how long will it take for me to get the password? Thanks!

请问可以提供一下训练代码嘛

Fill form is disabled.

Hi, Thanks for great LSVQ dataset.

I noticed, the fill form link,

https://docs.google.com/forms/d/e/1FAIpQLSeElJQswYZCM-4NaWfEZDR-FmmoI3-UObTkuOs8Gxw9jSCeMA/viewform?usp=send_form

is disabled. So I can not find password of LSVQ dataset.

Can you help me to how to fill the form and get a passwords?

Thanks in advance.

MSU Video Quality Metrics Benchmark Invitation

Hello! We kindly invite you to participate in our video quality metrics benchmark. You can submit PatchVQ (or any other your metrics) to the benchmark, following the submission steps, described here. The dataset distortions refer to compression artifacts on professional and user-generated content. The full dataset is used to measure methods overall performance, so we do not share it to avoid overfitting. Nevertheless, we provided the open part of it (around 1,000 videos) within our paper "Video compression dataset and benchmark of learning-based video-quality metrics", accepted to NeurIPS 2022.

Missing Videos in backup2/ folder

I think some of the videos are missing under backup2/ folder. The folder only contains 27 videos and some of the videos in the csv are not available (e.g. backup2/b3.mp4 and backup2/b4.mp4)

Cannot reproduce the PVQ numbers in test_PVQ_on_new_datasets.ipynb

Hi,

Very interesting work!

So for fun I tried following every step in test_PVQ_on_new_datasets.ipynb (the only difference being that I'm running a bunch of smaller Python programs instead a Jupyter notebook), but I'm getting different values on LIVE_VQC_tiny_all:

PVQ 0.704,0.605

Instead of 0.548,0.515 as stated in test_PVQ_on_new_datasets.ipynb. Are the numbers in the Jupyter notebook outdated perhaps?

I verified every intermediate step and everything seems to be OK in terms of tensor sizes and so on.

I'm curious if anyone has been able to reproduce the stated PVQ numbers successfully from the Jupyter notebook.

How can I download data? no reply from google form

Is there any other way to download data?

'download_from_internetarchive.ipynb' can not open

this web cannot open

Patch-VQ数据集下载咨询

学长好，我看到数据集的第一步里面运行之后不知道如何下载，好像第二步里面的谷歌云盘里面包含里第一步的数据集对嘛，请问百度云盘上有完整的数据集嘛，谢谢学长

请问可以提供训练代码和运行说明嘛

请问可以提供训练代码和运行说明嘛

name 'Accelerator' is not defined

In your demo of colab,

I got this error... T.T help me...

---

NameError Traceback (most recent call last)
in ()
----> 1 from fastiqa.vqa import *
2
3 # modify the json files to point to your database locations
4 LSVQ = load_json('json/LSVQ.json')
5 LIVE_VQC = {

2 frames
/usr/local/lib/python3.7/dist-packages/fastai/distributed.py in DistributedTrainer()
142 "Wrap model in DistributedDataParallel and dls in DistributedDL"
143 order = 11
--> 144 @Delegates(Accelerator, but=_hidden_params)
145 def init(self,
146 sync_bn=True, # Whether to replace all batch norm with nn.SyncBatchNorm

NameError: name 'Accelerator' is not defined

use docker to prevent errors in test_PVQ_on_new_datasets

hi
can u make docker to prevent dependencies errors in test_PVQ_on_new_datasets
maybe it solve the problem

PaQ2PiQ features error, test on one video doesnt work

hi, thanks for your great work
I get below error at # PaQ2PiQ features part

WARNING:root:setattr roi_col not existed
WARNING:root:roipool: size = (2, 2), scale = 1/32
WARNING:root:setattr vid_id not existed

100%|██████████| 1/1 [00:00<00:00, 2744.96it/s]exp_features

---

AttributeError Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/IPython/core/formatters.py in call(self, obj)
336 method = get_real_method(obj, self.print_method)
337 if method is not None:
--> 338 return method()
339 return None
340 else:

/usr/local/lib/python3.7/dist-packages/fastcore/foundation.py in getattr(self, k)
134 attr = getattr(self,self._default,None)
135 if attr is not None: return getattr(attr,k)
--> 136 raise AttributeError(k)
137 def dir(self): return custom_dir(self,self._dir())
138 # def getstate(self): return self.dict

AttributeError: path
{'paq2piq': <fastiqa.learn.TestLearner at 0x7ff098218210>}

can you fix it ???