lizhengwei1992 / fast_portrait_segmentation

Fast (aimed to "real time") Portrait Segmentation on mobile phone

Fast_Portrait_Segmentation

Fast (aimed to "real time") Portrait Segmentation at mobile phone

This project is not normal semantic segmentation but focus on real-time protrait segmentation.All the experimentals works with pytorch.

I hope to find a effcient network which can run on mobile phone. Currently, successfull application of person body/protrait segmentation can be find in APP like SNOW&B612, whose technology is proposed by a Korea company Nalbi.

Models

• mobilenet_dilate_unet^[1][2][7][9]

  Encoder : mobilenet_v2(os: 32)

  Decoder : unet(concat low level feature) use dilate convolution at different stage(d = 2, 6, 12, 18)

• Shuffle_Seg_SkipNet^[4][10][18]

  Encoder : shufflenet

  Decoder : skip connection (add low level feature)

• esp_dense_seg^{[20][10][15][19]}

• residualdense_bisenet^[15][23][24]

  Attention model is a potential module in the segmentation task. I use a very light residual-dense net as the backbone of the Context Path. The details about fussion of last features in Contxt Path is not clear in the paper(BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation).

• Segmentation + Matting ^[7][12][15]

  Hard segmentation + Soft matting.(coming soon)

update 2019/04/10: The code and pre_trained model of final version of the portrait_segmentation is released ! ! ! mobile_phone_human_matting

Speed Analysis

⚡ Real-time ! ! ! 🎉🎉🎉

Platform : ncnn.

Mobile phone: Samsung Galaxy S8+(cpu).

	model size (M)	time(ms)
model_seg_matting	3.3	~40

update : 2018/12/27: Demo video on my iphone 6 (baiduyun)

Result Examples

HUAWEI Mate 20 released recently can keep color on human and make the bacgrand gray in real time (click to view ). I test my model using cpu on my MAC, getting some videos here.

References

papers

• [1] MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• [2] MobileNetV2: Inverted Residuals and Linear Bottlenecks
• [3] ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
• [4] ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• [5] CondenseNet: An Efficient DenseNet using Learned Group Convolutions
• [6] Xception: Deep Learning with Depthwise Separable Convolutions
• [7] U-Net: Convolutional Networks for Biomedical Image Segmentation
• [8] Rethinking Atrous Convolution for Semantic Image Segmentation
• [9] Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• [10] Fully Convolutional Networks for Semantic Segmentation
• [11] Automatic Portrait Segmentation for Image Stylization
• [12] Fast Deep Matting for Portrait Animation on Mobile Phone
• [13] DenseASPP for Semantic Segmentation in Street Scenes
• [14] Learning a Discriminative Feature Network for Semantic Segmentation
• [15] ERFNet: Efficient Residual Factorized ConvNet for Real-time Semantic Segmentation
• [16] ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
• [17] SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
• [18] ICNet for Real-Time Semantic Segmentation on High-Resolution Image
• [19] ShuffleSeg: Real-time Semantic Segmentation Network
• [20] ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation
• [21] Efficient Semantic Segmentation using Gradual Grouping
• [22] Analysis of efficient CNN design techniques for semantic segmentation
• [23] Dual Attention Network for Scene Segmentation
• [24] BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation
• [25] CCNet: Criss-Cross Attention for Semantic Segmentation
• [26] ESPNetv2: A Light-weight, Power Efficient, and General Purpose Convolutional Neural Network

fast_portrait_segmentation's Issues

acc,loss

It is cool.
HI,could please provide the image of acc/val_loss?

Model architecture overview

Hi, are you using encoder - decoder architecture like the U-Net or coarse segmentation - fine matting architecture in the real time model ? If possible can you give in depth information regrading the same.

where is BilinearConvTranspose2d

Thanks for your share, in file shuffle_seg_skipnet.py, there is a function named BilinearConvTranspose2d. Is it imported from model.layers? but I didn't find model.layers in your project.

mac上15fps左右，比较慢啊！！华为手机上实时的视频分割效果好像不是很好

dataset

bro, have u uploaded the supervisely dataset to googledrive or baiduyun? The origin link is very slowly. Thank u for sharing it.

May I ask about COCO datasets.

Hi, thanks for the sharing of results. Do you use all of the classes in COCO datasets for training or just use one class for 'person'.

hi，can you give the train code

hi

Are you going to continue this project? Do you have any model trained to be shared?