@IceClear Any chance to get the code without using OpenMMLab? Would make it easier for experimenting. Thanks for the awesome work by the way.

Hi there, thank you for publishing such a great work. May I ask what is the resolution for CLIP-IQA? The resolution I found in coopclipiqa.py file is 224x224, but the resize transformation in config files are all commented out. So does this mean that there are some code for image resizing somewhere else or the image resolution is kept the same? The resolution are also not included in the paper as well.

when initing the model the error
*** urllib.error.URLError: <urlopen error CLIPIQAFixed: <urlopen error [Errno 104] Connection reset by peer>>

Is there a straightforward way to run the model on CPU?

Hello,
do you have plans to provide the ONNX model weights and inference code at a later time?
Thanks.

python tools/train.py configs/clipiqa/clipiqa_coop_koniq.py

2023-10-16 08:47:54,004 - mmedit - INFO - workflow: [('train', 1)], max: 500 iters
2023-10-16 08:47:54,004 - mmedit - INFO - Checkpoints will be saved to C:\Users\pdo-labuser\CLIP-IQA\work_dirs\clipiqa_coop_koniq by HardDiskBackend.
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 2015/2015, 17.2 task/s, elapsed: 117s, ETA: 0sTraceback (most recent call last):
File "tools/train.py", line 146, in
main()
File "tools/train.py", line 135, in main
train_model(
File "c:\users\pdo-labuser\clip-iqa\mmedit\apis\train.py", line 106, in train_model
_non_dist_train(
File "c:\users\pdo-labuser\clip-iqa\mmedit\apis\train.py", line 362, in _non_dist_train
runner.run(data_loaders, cfg.workflow, cfg.total_iters)
File "C:\Users\pdo-labuser\anaconda3\envs\clipiqa\lib\site-packages\mmcv\runner\iter_based_runner.py", line 134, in run
iter_runner(iter_loaders[i], **kwargs)
File "C:\Users\pdo-labuser\anaconda3\envs\clipiqa\lib\site-packages\mmcv\runner\iter_based_runner.py", line 67, in train
self.call_hook('after_train_iter')
File "C:\Users\pdo-labuser\anaconda3\envs\clipiqa\lib\site-packages\mmcv\runner\base_runner.py", line 309, in call_hook
getattr(hook, fn_name)(self)
File "c:\users\pdo-labuser\clip-iqa\mmedit\core\evaluation\eval_hooks.py", line 48, in after_train_iter
self.evaluate(runner, results)
File "c:\users\pdo-labuser\clip-iqa\mmedit\core\evaluation\eval_hooks.py", line 57, in evaluate
eval_res = self.dataloader.dataset.evaluate(
TypeError: evaluate() got an unexpected keyword argument 'gpu_collect'

Please help with resolution

Hi @IceClear , I am trying to run the demo for a single image, but the pretrained model provided is taking forever to load. Infact, doesn't load at all.

This is the line with the issue:

model = init_model(
        args.config, args.checkpoint, device=torch.device('cuda', args.device))

This is the output I get:

Initializing a generic context
Initial context: "X X X X X X X X X X X X X X X X"
Number of context words (tokens): 16
2023-02-20 03:16:18.505897: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-20 03:16:18.939280: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.

After which it just doesn't complete loading.
Im using config file: ./configs/clipiqa/clipiqa_coop_koniq.py
and checkpoint: ./iter_80000.pth as provided and mentioned.

Please advice.

Hi,

Im doing different tests with demo/clipiqa_single_image_demo.py and the attribute_list = ['Quality', 'Brightness', 'Sharpness', 'Noisiness', 'Colorfulness', 'Contrast'].

First, I’ve seen that fitting a good size to the input image is essential because in some cases the result is NaN. Is there a fixed size that should be adjusted in the inbound image?

In the tests I’m doing, regardless of the resize, I see that it doesn’t matter the input image the result is always the same, you know what might be?

Example:

MSI@DESKTOP-FEG9P7H MINGW64 /e/2. Projects/Image Quality/CLIP-IQA (v2-3.8)
$ python demo/clipiqa_single_image_demo.py --config configs/clipiqa/clipiqa_attribute_test.py --file_path dataset/good_1.jpg 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
['Quality', 'Brightness', 'Sharpness', 'Noisiness', 'Colorfulness', 'Contrast', 'Quality']
[0.9892578  0.98876953 0.99853516 0.06512451 0.74316406 0.66796875]
(clipiqa) 
MSI@DESKTOP-FEG9P7H MINGW64 /e/2. Projects/Image Quality/CLIP-IQA (v2-3.8)
$ python demo/clipiqa_single_image_demo.py --config configs/clipiqa/clipiqa_attribute_test.py --file_path dataset/bad_3.jpeg 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
['Quality', 'Brightness', 'Sharpness', 'Noisiness', 'Colorfulness', 'Contrast', 'Quality']
[0.9892578  0.98876953 0.99853516 0.06512451 0.74316406 0.66796875]
(clipiqa) 
MSI@DESKTOP-FEG9P7H MINGW64 /e/2. Projects/Image Quality/CLIP-IQA (v2-3.8)
$ python demo/clipiqa_single_image_demo.py --config configs/clipiqa/clipiqa_attribute_test.py --file_path dataset/blur_2.tif
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
['Quality', 'Brightness', 'Sharpness', 'Noisiness', 'Colorfulness', 'Contrast', 'Quality']
[0.9892578  0.98876953 0.99853516 0.06512451 0.74316406 0.66796875]
(clipiqa) 

Hi IceClear,
I just try to train CLIP-IQA+ on KonIQ-10k as your instruct:

python tools/train.py configs/clipiqa/clipiqa_coop_koniq.py

But encounter the following error, it seems only PSNR/SSIM supported.
Did this mean I have to change the config option L1DIS to PSNR/SSIM?

[                                                  ] 0/2015, elapsed: 0s, ETA:Traceback (most recent call last):
  File "tools/train.py", line 146, in <module>
    main()
  File "tools/train.py", line 135, in main
    train_model(
  File "d:\code\clip-iqa\mmedit\apis\train.py", line 106, in train_model
    _non_dist_train(
  File "d:\code\clip-iqa\mmedit\apis\train.py", line 362, in _non_dist_train
    runner.run(data_loaders, cfg.workflow, cfg.total_iters)
  File "C:\Users\A\.conda\envs\clipiqa\lib\site-packages\mmcv\runner\iter_based_runner.py", line 134, in run
    iter_runner(iter_loaders[i], **kwargs)
  File "C:\Users\A\.conda\envs\clipiqa\lib\site-packages\mmcv\runner\iter_based_runner.py", line 67, in train
    self.call_hook('after_train_iter')
  File "C:\Users\A\.conda\envs\clipiqa\lib\site-packages\mmcv\runner\base_runner.py", line 309, in call_hook
    getattr(hook, fn_name)(self)
  File "d:\code\clip-iqa\mmedit\core\evaluation\eval_hooks.py", line 42, in after_train_iter
    results = single_gpu_test(
  File "d:\code\clip-iqa\mmedit\apis\test.py", line 43, in single_gpu_test
    result = model(
  File "C:\Users\A\.conda\envs\clipiqa\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\A\.conda\envs\clipiqa\lib\site-packages\mmcv\parallel\data_parallel.py", line 50, in forward
    return super().forward(*inputs, **kwargs)
  File "C:\Users\A\.conda\envs\clipiqa\lib\site-packages\torch\nn\parallel\data_parallel.py", line 166, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "C:\Users\A\.conda\envs\clipiqa\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\A\.conda\envs\clipiqa\lib\site-packages\mmcv\runner\fp16_utils.py", line 110, in new_func
    return old_func(*args, **kwargs)
  File "d:\code\clip-iqa\mmedit\models\restorers\basic_restorer.py", line 79, in forward
    return self.forward_test(lq, gt, **kwargs)
  File "d:\code\clip-iqa\mmedit\models\restorers\clipiqa.py", line 190, in forward_test
    results = dict(eval_result=self.evaluate(output, gt))
  File "d:\code\clip-iqa\mmedit\models\restorers\clipiqa.py", line 156, in evaluate
    value = self.allowed_metrics[metric](output_img, gt_img)
KeyError: 'L1DIS'

Hello, can this code be tested and trained on non-public datasets, such as one's own dataset? How to set up and process data?

Welcome update to OpenMMLab 2.0

I am Vansin, the technical operator of OpenMMLab. In September of last year, we announced the release of OpenMMLab 2.0 at the World Artificial Intelligence Conference in Shanghai. We invite you to upgrade your algorithm library to OpenMMLab 2.0 using MMEngine, which can be used for both research and commercial purposes. If you have any questions, please feel free to join us on the OpenMMLab Discord at https://discord.gg/A9dCpjHPfE or add me on WeChat (ID: van-sin) and I will invite you to the OpenMMLab WeChat group.

Here are the OpenMMLab 2.0 repos branches:

Attention: please create a new virtual environment for OpenMMLab 2.0.

Hello,

I have been doing tests with the clipiqa_single_image_demo.py and the "test degradation attributes" and sometimes the result is nan.
It is true that for some images it returns scores, but usually not.

python demo/clipiqa_single_image_demo.py --config configs/clipiqa/clipiqa_attribute_test.py --checkpoint checkpoint/iter_80000.pth --file_path dataset/img.jpg

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
['Quality', 'Brightness', 'Sharpness', 'Noisiness', 'Colorfulness', 'Contrast', 'Quality']
[0.9892578  0.98876953 0.99853516 0.06488037 0.7416992  0.66796875]

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
['Quality', 'Brightness', 'Sharpness', 'Noisiness', 'Colorfulness', 'Contrast', 'Quality']
[nan nan nan nan nan nan]

Do you know why this can be and how to fix it?

HI~ Thank you for your work！
I want to train a model that recognizes cars and boats, is the train config and CSV file labeled correctly?

Thank you for your work
In the paper you mentioned interpolation scheme -- do you have code to interpolate embedding?
And whats best way to customize CLIP? why you choose mmedit?

how to infer image directly not by load image form path ，it means that I just input a image array to the main function not input a image path？

Hi @IceClear,

Thank you for open sourcing this impressive work! I have a question regarding the clipiqa_single_image_demo.py script. When I use this script to do inference on a single image, I notice that only the first value of the attribute list is returned. The output looks something like this: [0.74316406] (using the example rock image).

The print info would be something like:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[0.74316406]

Could you please provide some guidance on how to obtain all the attribute values, such as 'Brightness', 'Sharpness', 'Noisiness', 'Colorfulness', and 'Contrast', which may differ from each other?

Thank you very much for your help!

After running this code：output, attributes = restoration_inference(model, os.path.join(args.file_path), return_attributes=True)

output and attributes are the same. Why return the same two data?

Code running result：

attributes>>>>>>>>>>>>>>>>>>
tensor([[0.9644, 0.3345, 0.9995, 0.0357, 0.2766, 0.3577]])
output>>>>>>>>>>>>>>>>>>
[[0.96435547 0.33447266 0.9995117 0.03567505 0.27661133 0.35766602]]

When we try to run clipiqa_single_image_demo.py and load pre-trained model, there will be the following problems:

unexpected key in source state_dict: generator.clipmodel_0.logit_scale, generator.clipmodel_0.prompt_learner.ctx,.......
missing keys in source state_dict: generator.clip_model.positional_embedding, generator.clip_model.text_projection,......

How should I solve it?

Hi!
I am just wondering about the CLIPIQA+ experiment mentioned in the paper. In Table 1, it says that the model was trained on Koniq and tested on Koniq, LIVE-itW, and SPAQ. So, I was curious about the specifics of the training process. Did it involve training on the official Koniq-train dataset, selecting the best parameters on Koniq-valid, and testing on Koniq-test, LIVE-itW, and SPAQ?

Hi IceClear,
Thank you for your impressive work! I encounter OOM issue when handle large image with clipiqa_single_image_demo.py, such as 8M/12M pictures. I want to know is there any memery limited for the CLIP-IQA model. Or whether my env runtime had configuration issue?

Here's the detail output logs:

clipiqa) PS D:\code\CLIP-IQA> python .\demo\clipiqa_single_image_demo.py --file_path=../dataset/IMG_20230316_105935577.jpg
Traceback (most recent call last):
File ".\demo\clipiqa_single_image_demo.py", line 62, in
main()
File ".\demo\clipiqa_single_image_demo.py", line 38, in main
output, attributes = restoration_inference(model, os.path.join(args.file_path), return_attributes=True)
File "d:\code\clip-iqa\mmedit\apis\restoration_inference.py", line 79, in restoration_inference
result = model(test_mode=True, **data)
File "C:\Users\A.conda\envs\clipiqa\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\A.conda\envs\clipiqa\lib\site-packages\mmcv\runner\fp16_utils.py", line 110, in new_func
return old_func(*args, **kwargs)
File "d:\code\clip-iqa\mmedit\models\restorers\basic_restorer.py", line 79, in forward
return self.forward_test(lq, gt, **kwargs)
File "d:\code\clip-iqa\mmedit\models\restorers\clipiqa.py", line 182, in forward_test
output, attribute_prob = self.generator(lq)
File "C:\Users\A.conda\envs\clipiqa\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "d:\code\clip-iqa\mmedit\models\backbones\sr_backbones\coopclipiqa.py", line 314, in forward
logits_per_image, logits_per_text = self.clip_model(image, self.tokenized_prompts[i].to(image.device), self.pos_embedding)
File "C:\Users\A.conda\envs\clipiqa\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "d:\code\clip-iqa\mmedit\models\components\clip\model.py", line 373, in forward
image_features = self.encode_image(image, pos_embedding)
File "d:\code\clip-iqa\mmedit\models\components\clip\model.py", line 355, in encode_image
return self.visual(image.type(self.dtype), pos_embedding=pos_embedding)
File "C:\Users\A.conda\envs\clipiqa\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "d:\code\clip-iqa\mmedit\models\components\clip\model.py", line 158, in forward
x = self.attnpool(x, return_token, pos_embedding)
File "C:\Users\A.conda\envs\clipiqa\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "d:\code\clip-iqa\mmedit\models\components\clip\model.py", line 72, in forward
x, _ = F.multi_head_attention_forward(
File "C:\Users\A.conda\envs\clipiqa\lib\site-packages\torch\nn\functional.py", line 5101, in multi_head_attention_forward
attn_output, attn_output_weights = _scaled_dot_product_attention(q, k, v, attn_mask, dropout_p)
File "C:\Users\A.conda\envs\clipiqa\lib\site-packages\torch\nn\functional.py", line 4847, in _scaled_dot_product_attention
attn = softmax(attn, dim=-1)
File "C:\Users\A.conda\envs\clipiqa\lib\site-packages\torch\nn\functional.py", line 1680, in softmax
ret = input.softmax(dim)
RuntimeError: CUDA out of memory. Tried to allocate 8.86 GiB (GPU 0; 12.00 GiB total capacity; 9.59 GiB already allocated; 0 bytes free; 9.70 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Could u please give me some advice?

Hi!
I try to run clipiqa_single_image_demo.py, but it tells me "CLIPIQA is not in the model registry".
Could you please tell me how to fix this problem? Thanks!