speedinghzl / pytorch-deeplab Goto Github PK

how to train in multiple GPUs？

Hi,
By not changing anything on the code, i ve ran train.py for the augmented pascal voc dataset. But im getting CUDA error. Might you have a suggestion on how i can solve this?

A detailed error message is as below.

Thanks in advance.
sinem.

 File "train.py", line 234, in <module>
   main()
 File "train.py", line 213, in main
   pred = interp(model(images))
 File "/home/sinem/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
   result = self.forward(*input, **kwargs)
 File "/media/sinem/LENOVO/Pytorch-Deeplab-master/deeplab/model.py", line 261, in forward
   x = self.layer5(x)
 File "/home/sinem/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
   result = self.forward(*input, **kwargs)
 File "/media/sinem/LENOVO/Pytorch-Deeplab-master/deeplab/model.py", line 115, in forward
   out += self.conv2d_list[i+1](x)
 File "/home/sinem/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
   result = self.forward(*input, **kwargs)
 File "/home/sinem/anaconda3/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
   self.padding, self.dilation, self.groups)
RuntimeError: CUDA error: out of memory

Hi, I am the author of this repository. Your repository was brought to my notice recently. It seems that you have used a lot of my code and have not cited my repository. My repository has an MIT license. I would really appreciate if you cite my repository in your readme and mention that code in repo has been borrowed from my repo.
Thanks.

Hi @speedinghzl ,

I used your pytorch deeplab v2 implementation, same setting, with torch version of 0.4.0, I used batch size of 16 to train the multi-scale model and evaluated at evaluate.py. However I checked iteration at 10000, 20000, 30000... the meanIOU performance remains at 0.55~0.59.
Could you give me any comments for what I did wrong?
Here is the parameter I used:

BATCH_SIZE = 16
ITER_SIZE = 10
IGNORE_LABEL = 255
INPUT_SIZE = '321,321'
LEARNING_RATE = 2.5e-4
NUM_STEPS = 500000
POWER = 0.9
RANDOM_SEED = 1234
RESTORE_FROM = './model/MS_DeepLab_resnet_pretrained_COCO_init.pth'
SAVE_NUM_IMAGES = 2
SAVE_PRED_EVERY = 50
SNAPSHOT_DIR = './snapshots_msc/'
WEIGHT_DECAY = 0.0005

Thank you for your help!

First of all, thanks for providing your code.
When I use your uploaded VOC12_scenes_20000.pth model to evaluate the single scale model on 1449 validation data, I get 71.8% which is much lower than reported 74.0%.
Moreover, when I try to train the single scale model by running python train.py --random-mirror --random-scale --gpu 0, and then evaluate the trained model, I get 73.1%, which is still lower than 74%.
Do you know where this discrepancy might come from?
Everything is on default settings and I also use MS_DeepLab_resnet_pretrained_COCO_init.pth as the pre-trained model.

Hi, @speedinghzl

Thanks for your sharing. I am a little curious why you set 'cudnn.enabled = False'? I set it to True and find it is faster and use less GPU memory. Will it have a bad influence on accuray? Thank you.

Could you upload a .pth file for the validation 74.0%?

i change NUM_CLASSES =2 and load model "MS_DeepLab_resnet_pretrained_COCO_init.pth"

I am trying to load VOC12_scenes_20000.pth onto Res_Deeplab
Running this,

    saved_state_dict = torch.load(args.restore_from)
    new_params = model.state_dict().copy()
    for i in saved_state_dict:
        #Scale.layer5.conv2d_list.3.weight
        i_parts = i.split('.')
        # print i_parts
        if not args.num_classes == 21 or not i_parts[1]=='layer5':
            print(i)
            new_params['.'.join(i_parts[1:])] = saved_state_dict[i]
    model.load_state_dict(new_params)

gives me
*** KeyError: 'unexpected key "weight" in state_dict'

https://github.com/speedinghzl/Pytorch-Deeplab/blob/master/deeplab/model.py#L116

Should it be

def forward(self, x):
    out = self.conv2d_list[0](x)
    for i in range(len(self.conv2d_list)-1):
        out += self.conv2d_list[i+1](x)
    return out

instead of

def forward(self, x):
    out = self.conv2d_list[0](x)
    for i in range(len(self.conv2d_list)-1):
        out += self.conv2d_list[i+1](x)
        return out

I download this weight and share on google drive https://drive.google.com/open?id=19_PGrzARnJugSKULbTyRJ0txlI2UPvm6.

讲真，装百度网盘的时候真的超级怕被在系统里开个后门。

Could anybody share VOC12_scenes_20000.pth on googledrive? I couldnot install it from the link given here: https://pan.baidu.com/s/1bP52R8

in chinese and it requires me to install a exe file.. baudi..something ..

thanks in advance
sinem.

Hi @speedinghzl ,

I used your pytorch deeplab v2 implementation, same setting, with pytorch version of 1.0.0. The validation mIoU of VOC_scenes_20000.pth is 71.1. Is this degradation due to randomness? Could you give me any comments?
Here is the parameter I used:

BATCH_SIZE = 10
DATA_DIRECTORY = './dataset/voc12'
DATA_LIST_PATH = './dataset/list/train_aug.txt'
IGNORE_LABEL = 255
INPUT_SIZE = '321,321'
LEARNING_RATE = 2.5e-4
MOMENTUM = 0.9
NUM_CLASSES = 21
NUM_STEPS = 20000
POWER = 0.9
RANDOM_SEED = 1234
RESTORE_FROM = './dataset/MS_DeepLab_resnet_pretrained_COCO_init.pth'
SAVE_NUM_IMAGES = 2
SAVE_PRED_EVERY = 1000
SNAPSHOT_DIR = './snapshots/'
WEIGHT_DECAY = 0.0005

Thank you for your help!

Hi, can I run the code on my laptop where I don't have a GPU that supports CUDA? Can you guide me how to make it work? Cheers.

Thanks for your code, I have learned a lot from it. But there are some question that troubles me.
When I run code, I set args.gpu =1, but I find the program always occupy memory in gpu0.

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 6721 C python 399MiB |
| 1 6721 C python 7215MiB |
+-----------------------------------------------------------------------------+

I don't know where is the problem, can you solve this? Thanks a lot.

Hello, Do you implement "frozen BN layers"? I find that the running means and running vars still keep changing during training.

I notice that the normalization of images in your code uses "image - mean," rather than the normal "(image - mean)/std". Is there a specific reason for using this kind of normalization?

For the scale of pixel values, your code is in the scale of [-128,128], rather than [-0.5,0.5] that are used in other works. Which one is better?

Also, if I use only the ImageNet pre-trained model, which kind of image normalization and pixel scale should I use?

Hi, RANDOM_SEED = 1234 was defined in train.py (Line 36, Line 78-79), but where did you use this random seed? Thanks.

How to change parameters for deeplabV3 training and what parameters can be used .

i chang NUM_CLASSES=21 to NUM_CLASSES=2 ,but i meet a problem, as
RuntimeError: inconsistent tensor size, expected tensor [2 x 2048 x 3 x 3] and src [21 x 2048 x 3 x 3] to have the same number of elements, but got 18432 and 387072 elements respectively at /pytorch/torch/lib/TH/generic/THTensorCopy.c:86_.
it seems that pre-trained model weight doesn't match 'NUM_CLASSES=2' network,

Hi, i am sorry but where can i find the crf implementation?

HI! @speedinghzl , thanks for you code!
I used the VOC12_scenes_20000.pth model you provided to run the test code. I only change the path to load data, any other thing was changed in my operation. The Miou results achieved 0.693386318233 in the single scale model , when i use the multi scale model you provided , the result is also 69%. Can you help me ? or give me some idea about it.
Thank you very much! Good luck to u.

Hi, thanks for your code.

In train.py, function 'loss_calc ' compute log_softmax for parameter 'pred' , but in CrossEntropy2d, the code use F.cross_entropy which will compute log_softmax again, so I think this maybe a bug. Maybe we should delete line 103 and 105 of train.py?

hi, i have a questions about the label maps used in the code.
the folder "SegmentationClassAug" seems only have boundary information rather than class information. So, only boundary information is enough or i have a false understanding of the code and the original paper?

hi, i use it to train for my own data set, but it run error, can u give me some suggestion? (my segmentation class img is gray image).

/opt/conda/conda-bld/pytorch_1532581333611/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [3,0,0], thread: [190,0,0] Assertion t >= 0 && t < n_classes failed.
/opt/conda/conda-bld/pytorch_1532581333611/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:99: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T *, T *, T *, long *, T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [3,0,0], thread: [191,0,0] Assertion t >= 0 && t < n_classes failed.
Traceback (most recent call last):
File "train.py", line 234, in
main()
File "train.py", line 215, in main
loss.backward()
File "/home/zhangjunyi/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/tensor.py", line 93, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/zhangjunyi/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/init.py", line 90, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: CuDNN error: CUDNN_STATUS_INTERNAL_ERROR

Not quite sure about this, but I think we should write optimizer.zero_grad() in the for loop?

I think this part of code is wrong since it will yeild same parameters multiple times.

Hello, I am a master student in Beijing. I am reading your codes and I am confused about some questions.

In datasets.py, your 'Class VOCDataSet' reads image using 'cv2.imread', so your image should be 'BGR' channel order. But then you use 'image = image.transpose((2, 0, 1))', so the image should be 'RBG' channel order. This really confused me a lot because I think that it should be 'RGB'.

Also, in datasets.py, when you show image, you use 'img = np.transpose(img, (1, 2, 0)), img = img[:, :, ::-1]'. And I also think that this way may generate an image that is not in 'RGB' channel order.

It may just be my misunderstanding. But I am really looking forward to your reply. Thanks!!!

Since pytorch 0.4.0, the default paramter for align_corners for Upsample in False, that's quite different from older version. To keep the excutation the same for different versions of pytorch, I highly advise you explicitly set it.

Hi,
thanks for the code sharing
have a few questions about the implementation.

1. why load a net pre trained on coco instead of using a pre trained net on Imagenet which is inbuilt in pytorch
   2.why swap dimensions of 'R' and 'B'