pandinosaurus / colorization

Automatic line art colorization using various types of hint or without hint

adeleine

Update

• 03/20/2023: Change repository's name from Colorization to adeleine

Overview

This repository is about the deep-learing based automatic line art colorization with or without hints. There are mainly three types of hints.

• Atari: Colorization with hints that include some lines in desired color (ex. PaintsChainer)
• Tag: Colorization with tags (ex. Tag2Pix)
• Reference: Colorization with reference images (ex. style2paints V1)

Line extraction method

There are many kinds of line extraction methods, such as XDoG or SketchKeras. If we train the model with only single type of the line art, trained model tends to overfit and cannot colorize other types of line art properly (ref Tag2Pix). Therefore, three types of line art are mainly used.

• XDoG: Line extraction using two Gaussian distributions difference to standard deviations
• SketchKeras: Line extraction using UNet. Lines obtained by SketchKeras are like pencil drawings.
• Sketch Simplification: Line extraction using Fully-Convolutional Networks. Lines obtained by Sketch Simplification are like digital drawings.

An example obtained by these line extraction methods is as follows.

Moreover, I usually add two types of data augmenation to line arts to avoid overfitting.

• Randomly morphology transformation to take various thicks of lines
• Randomly RGB values of lines to incorporate various depths of lines

Experiments without hints

Methods

• pix2pix
• pix2pixHD
• bicyclegan

Results

Method	Result
pix2pix & pix2pixHD
bicyclegan

Experiment with atari

Methods

• userhint
• userhint v2
• whitebox
• spade

Results

Method	Result
userhint
userhint v2
whitebox
spade

Experiment with reference

Methods

• adain
• scft
• video

Results

Method	Result
adain
scft
video

GUI application

LICENSE

MIT License on only Adeleine. The license is not effective on other implementations.

References

• XDoG: An eXtended difference-of-Gaussians compendium including advanced image stylization
• sketchKeras
• Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup
• Image-to-Image Translation with Conditional Adversarial Networks
• High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
• Toward Multimodal Image-to-Image Translation
• Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization
• Mode Seeking Generative Adversarial Networks for Diverse Image Synthesis
• Semantic Image Synthesis with Spatially-Adaptive Normalization
• Real-Time User-Guided Image Colorization with Learned Deep Priors
• Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN
• Two-stage Sketch Colorization
• Tag2Pix: Line Art Colorization Using Text Tag With SECat and Changing Loss
• Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence
• Learning to Cartoonize Using White-box Cartoon Representations
• Deep Line Art Video Colorization with a Few References
• DanbooRegion: An Illustration Region Dataset