clovaai / ext_portrait_segmentation Goto Github PK

Thanks for your share~
I have some problems about the training~
When i try to change the input_image size from 224224 to 448448, it occurs AttributeError :
AttributeError: 'ExtremeC3NetCoarse' object has no attribute '_parameters'
Could you help me?

@HYOJINPARK Thank you for your hard work,

If I want to train baidu_V2, what exactly should I change?

Because I tried training baidu_V2 dataset, and got error:

FileNotFoundError: [Errno 2] No such file or directory: './baidu_V2//Portrait/train.txt'

So I placed the the files within Portrait folder, like this:

baidu_V2
└── Portrait
    ├── input
    ├── target
    ├── train.txt
    └── val.txt

And now I am getting error:

ValueError: invalid literal for int() with base 10: '07517.jpg\n'

I am unable to use the lovasz loss b/c of a channel mismatch:
the output I receive is 2 channels vs lovasz expects 1.
I saw in the comments that we have to adjust this as needed, but unclear where to do it?

I thought the chnn was how to control it but changing that from 1 - 4 does not affect the final tensor output?
Could you kindly clarify how to toggle between lovasz and CE outputs?
Thanks!

The number of labels is 1887.
The number of pictures is 1630.

Hello @HYOJINPARK ,

I successfully trained the ExtremeC3Net. Now I want to test new images to see the testing accuracy. I am not sure what to do next. I tried putting images in Link512DATA folder and run Test_model.py. But there is no option to show images results.

1. Where I should put my testing images? I mean which folder.
2. And how to see segmented results? I can't see any of the following. I am not sure where?
   cv2.imwrite("%s.jpg" %idx,np.asarray(segmented))
   fig, axes = plt.subplots(1, 2, figsize=(12, 10))
   axes[0].imshow(image)

Your response would be highly appreciated. Thank you

Setting --visualize to True doesn't appear to do anything.

I am trying to convert the pth model to onnx and I get the following error:

Traceback (most recent call last):
File "", line 1, in
File "/home/rsalasch/miniconda3/envs/tf1_4/lib/python3.6/site-packages/torch/onnx/init.py", line 26, in _export
result = utils._export(*args, **kwargs)
File "/home/rsalasch/miniconda3/envs/tf1_4/lib/python3.6/site-packages/torch/onnx/utils.py", line 488, in _export
fixed_batch_size=fixed_batch_size)
File "/home/rsalasch/miniconda3/envs/tf1_4/lib/python3.6/site-packages/torch/onnx/utils.py", line 351, in _model_to_graph
fixed_batch_size=fixed_batch_size, params_dict=params_dict)
File "/home/rsalasch/miniconda3/envs/tf1_4/lib/python3.6/site-packages/torch/onnx/utils.py", line 154, in _optimize_graph
graph = torch._C._jit_pass_onnx(graph, operator_export_type)
File "/home/rsalasch/miniconda3/envs/tf1_4/lib/python3.6/site-packages/torch/onnx/init.py", line 199, in _run_symbolic_function
return utils._run_symbolic_function(*args, **kwargs)
File "/home/rsalasch/miniconda3/envs/tf1_4/lib/python3.6/site-packages/torch/onnx/utils.py", line 740, in _run_symbolic_function
return op_fn(g, *inputs, **attrs)
File "/home/rsalasch/miniconda3/envs/tf1_4/lib/python3.6/site-packages/torch/onnx/symbolic_helper.py", line 128, in wrapper
args = [_parse_arg(arg, arg_desc) for arg, arg_desc in zip(args, arg_descriptors)]
File "/home/rsalasch/miniconda3/envs/tf1_4/lib/python3.6/site-packages/torch/onnx/symbolic_helper.py", line 128, in
args = [_parse_arg(arg, arg_desc) for arg, arg_desc in zip(args, arg_descriptors)]
File "/home/rsalasch/miniconda3/envs/tf1_4/lib/python3.6/site-packages/torch/onnx/symbolic_helper.py", line 81, in _parse_arg
"', since it's not constant, please try to make "

RuntimeError: Failed to export an ONNX attribute 'onnx::Gather', since it's not constant, please try to make things (e.g., kernel size) static if possible

I think the error is caused by the AveragePool2D

(6)
(d2): S2block(
(down_res): AvgPool2d(kernel_size=4, stride=4, padding=0)
(7)
(d2): S2block(
(down_res): AvgPool2d(kernel_size=2, stride=2, padding=0)

Please let me know if you have converted the model to onnx format before or if this error seems familiar.

Hello. Thank you for sharing your code and paper.
Could you tell me how to use the raw dataset EG1800 to verify the baseline?
Thank you.

Thank for your sharing paper, code and dataset.
But I can't download Nukki.zip.

@HYOJINPARK Thank you

Upload a pretrained model so that we can test

According to Visualize_video.py, i wrote a infer.py for testing pretrained model located in result folder. But i found output result is terriable.

Below is the complete code of infer.py script.

import cv2
import json
import models
import argparse
import torch
import numpy as np

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('-c', '--config', type=str)
    parser.add_argument('-ip', '--img_path', type=str)
    args = parser.parse_args()

    with open(args.config) as fin:
        config = json.load(fin)

    test_config = config['test_config']
    data_config = config['data_config']

    if test_config["loss"] == "Lovasz":
        test_config["num_classes"] = 1
        print("Use Lovasz loss ")
        Lovasz = True
    else:
        print("Use Cross Entropy loss ")
        Lovasz = False

    if test_config['Model'].startswith('Stage2'):
        model = models.__dict__[test_config["Model"]](classes=test_config["num_classes"],
                                                      p=test_config["p"], q=test_config["q"])
    elif test_config["Model"].startswith('Dnc_SINet'):
        model = models.__dict__[test_config["Model"]](
            classes=test_config["num_classes"], p=test_config["p"], q=test_config["q"],
            chnn=test_config["chnn"])

    mean = [107.304565, 115.69884, 132.35703]
    std = [63.97182, 65.1337, 68.29726]
    model.eval()

    image = cv2.imread(args.img_path)
    image = cv2.resize(image, (data_config['w'], data_config['h']))
    img = image.astype(np.float32)

    img = img.astype(np.float32)
    for j in range(3):
        img[:, :, j] -= mean[j]
    for j in range(3):
        img[:, :, j] /= std[j]

    img /= 255.
    img = img.transpose((2, 0, 1))
    img_tensor = torch.from_numpy(img)
    img_tensor = torch.unsqueeze(img_tensor, 0)

    with torch.no_grad():
        img_variable = torch.autograd.Variable(img_tensor)

        if torch.cuda.is_available():
            img_variable = img_variable.cuda()

        out = model(img_variable)

    classMap_numpy = (out[0].data.cpu() > 0).numpy()[0]
    idx_fg = (classMap_numpy == 1)
    seg_img = image * idx_fg[:, :, np.newaxis]
    cv2.imwrite("test.jpg", seg_img.astype(np.uint8))

And run with the following command

python3 infer.py -c setting/Test_ExtremeC3Net.json -ip /Users/edvardzeng/Dataset/PortraitDataset/Nukki/baidu_V1/input/43.png

Output result:

Did i missing something ?

Looking forward to your replay.

Thx.

Hi @HYOJINPARK @yjyoo3312

Thank you for the wonderful library! Wow!
why set ignore_idx==255 in lovasz loss? which is the person label in target .
Can you guide me please.....

============

main.py
if train_config["loss"] == "Lovasz":
from etc.lovasz_losses import lovasz_hinge
criteria = lovasz_hinge(ignore=data_config["ignore_idx"])
else:
from etc.Criteria import CrossEntropyLoss2d
criteria = CrossEntropyLoss2d(weight,ignore=data_config["ignore_idx"]) # weight

SINet.json
"data_config": {
"cash" : "./pickle_file/portrait.p",
"dataset_name": "CVportrait",
"data_dir": "../../Link512DATA/",
"classes" : 2,
"ignore_idx" : 255,
"num_work" : 4,

I have a question about inference time.

I set the test batch size to 1, and the result is shown below.

I use teslaT4 16G GPU , and the result is really not fast.

hi, the SINet.pth you give work normal.
but when i train the sinet using myown data , use my pth, but it occured

the layer name in my pth is not same with yours, some layer have no "encoder."

thank you for the nice work and hope for your reply.

Hello @HYOJINPARK ,

when i try to use the result/SINet/SINet.pth with ./etc/Visualize_video.py i get the following Error:

RuntimeError: Error(s) in loading state_dict for SINet:
	size mismatch for encoder.classifier.conv.weight: copying a param of torch.Size([1, 144, 1, 1]) from checkpoint, where the shape is torch.Size([2, 144, 1, 1]) in current model.
	size mismatch for bn_3.weight: copying a param of torch.Size([1]) from checkpoint, where the shape is torch.Size([2]) in current model.
	size mismatch for bn_3.bias: copying a param of torch.Size([1]) from checkpoint, where the shape is torch.Size([2]) in current model.
	size mismatch for bn_3.running_mean: copying a param of torch.Size([1]) from checkpoint, where the shape is torch.Size([2]) in current model.
	size mismatch for bn_3.running_var: copying a param of torch.Size([1]) from checkpoint, where the shape is torch.Size([2]) in current model.
	size mismatch for level2_C.conv.weight: copying a param of torch.Size([1, 48, 1, 1]) from checkpoint, where the shape is torch.Size([2, 48, 1, 1]) in current model.
	size mismatch for level2_C.bn.weight: copying a param of torch.Size([1]) from checkpoint, where the shape is torch.Size([2]) in current model.
	size mismatch for level2_C.bn.bias: copying a param of torch.Size([1]) from checkpoint, where the shape is torch.Size([2]) in current model.
	size mismatch for level2_C.bn.running_mean: copying a param of torch.Size([1]) from checkpoint, where the shape is torch.Size([2]) in current model.
	size mismatch for level2_C.bn.running_var: copying a param of torch.Size([1]) from checkpoint, where the shape is torch.Size([2]) in current model.
	size mismatch for level2_C.act.weight: copying a param of torch.Size([1]) from checkpoint, where the shape is torch.Size([2]) in current model.
	size mismatch for bn_2.weight: copying a param of torch.Size([1]) from checkpoint, where the shape is torch.Size([2]) in current model.
	size mismatch for bn_2.bias: copying a param of torch.Size([1]) from checkpoint, where the shape is torch.Size([2]) in current model.
	size mismatch for bn_2.running_mean: copying a param of torch.Size([1]) from checkpoint, where the shape is torch.Size([2]) in current model.
	size mismatch for bn_2.running_var: copying a param of torch.Size([1]) from checkpoint, where the shape is torch.Size([2]) in current model.
	size mismatch for classifier.1.weight: copying a param of torch.Size([1, 1, 3, 3]) from checkpoint, where the shape is torch.Size([2, 2, 3, 3]) in current model.

With self trained models i don't get this error. How can i fix this?

Your response would be highly appreciated. Thank you!

when i run python main.py, the error accurs as below:

File "/media/cendelian/新加卷/CV/Project/人像分割/portrait_segmentation/data/DataSet.py", line 42, in getitem
label_bool = 255 * ((label > 200).astype(np.uint8))
TypeError: '>' not supported between instances of 'NoneType' and 'int'

could you help me

Thank you for your excellent project.I have a question about ：
config = [[[3, 1], [5, 1]], [[3, 1], [3, 1]],
[[3, 1], [5, 1]], [[3, 1], [3, 1]], [[5, 1], [3, 2]], [[5, 2], [3, 4]],
[[3, 1], [3, 1]], [[5, 1], [5, 1]], [[3, 2], [3, 4]], [[3, 1], [5, 2]]]
Is there any rule in setting this parameter?

hello!
Nice work! I want to train the model with my own dataset, is there a guide on how to start with?
Thanks for your reply in advance

Hi, i used the SINet.json to train the model. After training ends, there are two folders, named 'Dnc_SINet07-12_2344' and 'Enc_SINet07-10_1303'. Both floders have model.pth. And i have try the model.pth in these two folders, but both are wrong. So i just wonder, how can i test the model after i have finishing the training?

Hello,
I am trying to run "main.py", but i get the following error. Could you please provide a little bit explanation.

python main.py --c ExtremeC3Net.json
no tensorflow found. set use_tensorboard = False
no visdom found. set visdom_port = None
Traceback (most recent call last):
File "main.py", line 40, in
with open(args.config) as fin:
FileNotFoundError: [Errno 2] No such file or directory: 'ExtremeC3Net.json'

hi,
when I run the command
python main.py --c ./setting/SINet.json
there is a error:
ValueError: invalid literal for int() with base 10: '501.png\n'

my dir tree is

.self.data_dir(../Link512DATA/)
├── Nukki
│   ├── baidu_V1
│   │   ├── input
│   │   ├── target
│   │   ├── train.txt
│   │   └── val.txt
│   └── baidu_V2
│   ├── input
│   ├── target
│   ├── train.txt
│   └── val.txt
└── Portrait
└── train.txt

the Portrait's train.txt is same as the baidu_V1's train.txt

could you help me?

when i train or test command, there are the same error:
File "/media/cendelian/7CCAEC82CAEC39D2/CV/Project/人像分割/portrait_segmentation/etc/help_function.py", line 25, in val_edge
rand_pick = np.random.randint(0, len(val_loader))
File "mtrand.pyx", line 747, in numpy.random.mtrand.RandomState.randint
File "_bounded_integers.pyx", line 1270, in numpy.random._bounded_integers._rand_int64
ValueError: low >= high

could you help me

syn_bg.jpg not found

I trained the SINet model on EG1800 and the baidu augmented datasets, using the default settings in the code. I tried both cross entropy and lovasz loss functions for training. Both of them acheived mIOU ~ 94.5; but they do not seem to perform well on real world portrait images(even the provided SINet.pth checkpoint). I tried testing the model using the demo video from portrait-net repo and 'Visualize_video.py' script. In both cases there were artefacts on the images in the background regions and sometimes they appear on foreground regions also.

Here are the results:-

Output Videos:
test_results.zip

HI, I haven't figure out why a bn is used after unsample, is the bn necessary? I would really appreciate for your reply

@HYOJINPARK @yjyoo3312

Thank you for the wonderful library! Wow!

I tried the demo on a sample image, but noticed that the edges on the output are rough. Can you guide me please.....

This is for SINet

• Photo
  

• Mask
  

• Composite
  

Hello. Thank you for sharing your code and paper.
I am checking your code and there are some confuse what I face.
First in the readme file, if I want to use the custom dataset I need to make new pickle file.
Could you explain which part of the code should I change?
In the train and test config, there is pickle file we need to change.
Is it automatically change or should we make this manually.
Also in the config there is dataset address we need to change
"data_dir": "../../Link512DATA/"
is it assume that the structure of folder is below?
Link512DATA/
└── Nukki
└── baidu_V1
├── input
├── target
├── train.txt
└── val.txt
└── baidu_V2
├── input
├── target
├── train.txt
└── val.txt
└── Portrait

syn_bg = cv2.imread(os.path.join(savedir,'syn_bg.jpg'))

Hi,

I tried retrain the model on biadu_V2, i am getting training iou as 0.945, but i evaluated on real life images, I am not able to get good results.

Also I can see the loss function you used in here is lovasz whereas the paper SINET has a completely different loss function. Can you provide some information on why the loss function and what could be the problem with these test images?

Do you have any tip for small object trainning.
I am dealing the small binary segmentation but it doesn';t have good result.
Do you have any tip.

I run the test data by my computer camera with resize image of 224*224. In CPU environment of I5 1.6gGHz， the model predict time is about 350+ms. Is there any problems? I wonder to know why the Flops is 0.128G, and the algorithm speed is so low.

Hello Hyojin Park,

Thanks for the amazing work. I am trying to run the code (ExtPortraitSeg "main.py") on my dataset. However, I have difficulty to do so. Could you please explain the procedure. Your response would be highly appreciated. Thank you