xindongzhang / ecbsr Goto Github PK

There are instructions for training, however, no test script is available. Can you provide the test script?

感谢作者的贡献！ECBSR模型在DIV2K的bicubic的数据上PSNR能确实到达很高。为了更好的落地作者的算法，我们尝试用同样的模型训练一批仿真的DIV2K多帧Y数据（退化方案不仅是Bicubic，而是复杂多样的，并通过亚像素位移融入多帧信息），得出来的PSNR结果却降到26左右且很难上升到30以上，Loss也很难像bicubic一样降到3左右：
##===========Epoch: 8=============##
Epoch:008, 0003200/3933696, loss: 7.0499, time: 13.528, psnr: 26.5676, ssim: 0.8217
Epoch:008, 0006400/3933696, loss: 6.9462, time: 8.626, psnr: 26.0487, ssim: 0.8200
Epoch:008, 0009600/3933696, loss: 6.9424, time: 8.789, psnr: 25.4504, ssim: 0.8032
Epoch:008, 0012800/3933696, loss: 6.9694, time: 8.869, psnr: 25.2612, ssim: 0.8418
Epoch:008, 0016000/3933696, loss: 6.9461, time: 8.921, psnr: 26.3836, ssim: 0.8434
Epoch:008, 0019200/3933696, loss: 6.9384, time: 8.902, psnr: 24.9899, ssim: 0.7983
Epoch:008, 0022400/3933696, loss: 6.9713, time: 8.835, psnr: 26.5746, ssim: 0.8638
Epoch:008, 0025600/3933696, loss: 6.9641, time: 8.790, psnr: 24.8029, ssim: 0.8114
Epoch:008, 0028800/3933696, loss: 6.9686, time: 8.926, psnr: 25.2221, ssim: 0.8379
Epoch:008, 0032000/3933696, loss: 6.9657, time: 8.830, psnr: 27.0723, ssim: 0.8332
Epoch:008, 0035200/3933696, loss: 6.9636, time: 9.089, psnr: 28.2336, ssim: 0.8760
想请教作者能否给些建议呢？感谢作者！

为了让模型可以输入rgb图，我将下面的两行代码改了，如下：

  # y = self.backbone(x) + x
  # y = self.upsampler(y)

  up_x = F.upsample_bilinear(x, scale_factor=self.scale_factor)
  y = up_x + self.upsampler(self.backbone(x))

模型的参数是一样的，使用的模型是ECBSR-M16C64，
set5上的psnr为30.6(Y通道)，相比原文中直接超分Y通道少了1.3db(31.92->30.6)，请问这种现象正常吗，按照我的理解，直接超分RGB的结果相比超分Y，结果应该不会差太多？

作者您好，我想问下如果训练数据的HR图片尺寸非常大，比如6000*4000这种，在训练的时候需要对图片尺寸做一些调整或者配置参数修改吗？？还是和div2k数据一样训练即可？？

I have trained the network in pytorch, the network only have three parts, pixel unshuffle, pixel shuffle and ECB blocks.
And the input channel and output channel are RGB.
Is it possible to convert the pytorch network to tensorflow? I have tried the convert.py，but I failed.

请问为什么我用
flops, params = profile(model, inputs=(input,)) print('计算量:{}'.format(flops)) print('参数量:{}'.format(params))

统计ecb 参数量和计算量显示是0呢？ 换了其他统计方法也是.

请问你们是用什么工具统计参数量和计算量的

Thank you for your work.
default colors set as 1, when I tried to train 3(RGB) image, error occurred...
maybe x need concat more times?

train the model from scratch!
##===========Epoch: 0=============##
Traceback (most recent call last):
File "/code/ECBSR/train.py", line 143, in
sr = model(lr)
File "/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/code/ECBSR/models/ecbsr.py", line 28, in forward
y = self.backbone(x) + x
RuntimeError: The size of tensor a (12) must match the size of tensor b (3) at non-singleton dimension 1

感谢作者的贡献！我正尝试训练3通道图片，目前遇到几个问题：
1、当训练ECBSR-M4C16时，训练基本正常，但是psnr比论文中低不少，我用了IMDN代码中的测试方法，将图片转成Ycbcr之后，针对Y通道计算psnr和ssim，但是测试结果在set14上psnr只有31.2x左右，在set5上psnr只有34.96左右。如果直接计算3通道图片的psnr和ssim，结果会更低。
2、当训练ECBSR-M16C64时，训练loss波动异常，loss会不定期的变得非常大：

##===========Epoch: 15=============##
Epoch:015, 03200/51200, loss: 4.5040, time: 35.135
Epoch:015, 06400/51200, loss: 4.3543, time: 30.960
Epoch:015, 09600/51200, loss: 4.3308, time: 30.984
Epoch:015, 12800/51200, loss: 4.3714, time: 30.946
Epoch:015, 16000/51200, loss: 4.3591, time: 31.149
Epoch:015, 19200/51200, loss: 498924.5582, time: 31.234
Epoch:015, 22400/51200, loss: 529279.5449, time: 31.021
Epoch:015, 25600/51200, loss: 463260.1697, time: 31.044
Epoch:015, 28800/51200, loss: 411850.2629, time: 30.985
Epoch:015, 32000/51200, loss: 370703.8992, time: 31.002
Epoch:015, 35200/51200, loss: 337032.6480, time: 31.208
Epoch:015, 38400/51200, loss: 308969.3531, time: 31.480
Epoch:015, 41600/51200, loss: 285220.5471, time: 31.479
Epoch:015, 44800/51200, loss: 264861.9574, time: 30.958
Epoch:015, 48000/51200, loss: 247216.6885, time: 30.925
Epoch:015, 51200/51200, loss: 231775.5556, time: 31.046
[set5-X2], PSNR/SSIM: 7.86/0.0751 (Best: 32.21/0.9010, Epoch: 14/14)
[set14-X2], PSNR/SSIM: 9.28/0.1133 (Best: 28.74/0.8311, Epoch: 14/14)

隔几个epoch后loss就会变得异常大，但是ECBSR-M4C16不会。其他训练参数都是默认的，训练数据是DIV2K，只修改了训练时的colors为3，然后forward里y = self.backbone(x) + x进行相应的修改。目前没看出来哪里的问题。

Would you provide the pretrained model?
Thx.

Thanks for your great work. I have some questions about the ECB block structure diagram in the paper.
The ECB block in the paper has conv-33 and sobel-Dy branch but conv-11 and sobel-Dy int the code. Is there someting wrong in the structure diagram (3.1 b) of the paper?

It's better to do something with the pretrained model.

Hi~ one more question..
As you know, the y channel super resolution result has to merge back to all channels' image, which are simple resized up.
so, when I compute the PSNR using the color image, result is not good as the evaluation result during training.
the x3 model test using DIV2K val DB. the PSNR result drop from 31.0x to 29.x...

anything wrong? thank you.

Thank you very much for your work, it is very helpful to my study!
I did not find the code for the inference where shows how to implement the re-parameterizable building during testing?

Hello. Thanks for your awesome work. I have some questions.

1. Can I get a weight file (.pth) for converted model after training? (not front-end file like onnx) It looks like you can get that after modifying convert.py (before 80 lines). By the way, there are simple code for converted model's weight file in RepVGG github. So it looks so good if you upload relevant code and I'll appreciate it.

2. Can you upload another block (RepVGG block, AC block, ...) codes like ecb.py and ecbsr.py? I want to know how can you drop the BN for layer fusion in RepVGG.

3. In the Table 1 in the paper, does the # of parameters means inference model's? I calculated M4C8's # of parameters in training mode. And it is more than 2.8K, not 2.8K as mentioned in paper.

Thank you.

Basically, each branch of the model is 1*1conv followed by 3*3conv, and then combined in series；If it is replaced by a 3*3conv and then a 3*3conv, can it be combined in series?

我在手机上， （高通的小米9） 死活调用不了dsp

Hi，
How wonderful is your work！
I have a question ''How much is the cpu's memory overhead when data loading?"
Only 15G I have, But it shows this is not enough.

If I don't need deploy the model to mobile phone, only want to remcover SR pictures from LR ,which are RGB pictures, what should Ido?
How to train and recover?
Thanks!

这里用到了4个重参数化的块，想问下这种快是越多越好吗，比如把repvgg等的块也加上来

WARNING: Params backbone.0.conv1x1_sbx.mask will not be optimized.

I changed the ecbsr to 3 channel，but the result was bad. Therefore, I would appreciate it if you give me method to solve the problem.

i wirte the code to merge Ychannel by sr with CbCr upsampled by bicubic, but the final rgb mat contains many blue or green or red pixels.

将超分后的Y通道和直接只用bicubic插值上采样的CbCr转换为rgb格式图片保存，发现有很多红绿蓝的像素点，怎么解决？能分享一下代码吗

你好，最近我将SR模型转换为SNPE DLC模型的时候出现GPU/DSP runtime不支持pixelshuffle的问题，请教一下如何解决？
我的具体环境如下：
芯片：865
snpe版本：1.55.0
系统：aarch64_ubuntu1804_rb5

具体问题表述：将pytorch模型转换为onnx，然后将onnx模型转成dlc模型的时候，snpe会自动将pixelshuffle操作分解为reshape、permute等操作，在模型输入尺寸超过320*320的时候转换模型就会出现dsp/gpu runtime不支持的情况。

在您的论文中有提到在DSP上实现了1080p的实时超分，请问一下您有没有遇到这个问题？然后您是怎么解决的呢？
非常感谢！！！

Hi, Thanks for your sharing. I tested the model you shared, the performance looked good. when will you release the quantization tool？

Hello, In your paper Figure 2(b) : Before "Sobel-Dy",the block shoud be "conv-1x1" rather than "conv-3x3"。
Would you please check it right or not?

What is this file used for? Convolution parameter fusion in the inference stage is not directly read from the training model, but after the training model is converted by convert.py, then inference is performed?

我的数据集是一维码，图像是灰度图，修改了数据集加载方式，直接加载灰度图，其他代码不变，但是超分后图像中出现0,255这样的随机噪点，这是什么原因引起的呢?

I used the pt file that came out through the learning process you provided.
But, the results converted to onnx and executed using pt files are different.
The results of the same pre-processing and the same post-processing are different.

Here is Picture
[pytorch inference]

[onnx inference]

Looking at it, the onnx model shows better results.
If it's the opposite situation, I'll understand, but I don't understand that onnx is a better result.

请问能提供预训练模型吗，非常感谢

idt_ecbsr: 0

why the Y channel of YCbCr was the only input into model?

Hi, it is a good job to incorporate the Re-parameterization into SR model. I still have some questions about the code and paper.

• In Section 2, Computation Reduction, the paper says

However, quantization of SR models often can hardly maintain the SR quality because of the precision requirement of pixel-wise prediction

The original RepVGG model does not perform well in quantization. How about the error of quantization in low-level Re-parameterization quantization? It seems that there is no further explanation or results for the quantization error in the paper.

• Common SR models are trained on RGB channel and tested on Matlab version Y of YCbCr. However, in the provided code
  

  ECBSR/datas/div2k.py

  Line 10 in eac2ff1

  import skimage.color as sc

  
  

  ECBSR/datas/div2k.py

  Line 54 in eac2ff1

  lr_image, hr_image = sc.rgb2ycbcr(lr_image)[:, :, 0:1], sc.rgb2ycbcr(hr_image)[:, :, 0:1]

  
  How about the error between different versions of Ycbcr?
  Thanks!