richzhang / colorization

Automatic colorization using deep neural networks. "Colorful Image Colorization." In ECCV, 2016.

Colorful Image Colorization [Project Page]

Richard Zhang, Phillip Isola, Alexei A. Efros. In ECCV, 2016.

+ automatic colorization functionality for Real-Time User-Guided Image Colorization with Learned Deep Priors, SIGGRAPH 2017!

[Sept20 Update] Since it has been 3-4 years, I converted this repo to support minimal test-time usage in PyTorch. I also added our SIGGRAPH 2017 (it's an interactive method but can also do automatic). See the Caffe branch for the original release.

Clone the repository; install dependencies

git clone https://github.com/richzhang/colorization.git
pip install requirements.txt

Colorize! This script will colorize an image. The results should match the images in the imgs_out folder.

python demo_release.py -i imgs/ansel_adams3.jpg

Model loading in Python The following loads pretrained colorizers. See demo_release.py for some details on how to run the model. There are some pre and post-processing steps: convert to Lab space, resize to 256x256, colorize, and concatenate to the original full resolution, and convert to RGB.

import colorizers
colorizer_eccv16 = colorizers.eccv16().eval()
colorizer_siggraph17 = colorizers.siggraph17().eval()

Original implementation (Caffe branch)

The original implementation contained train and testing, our network and AlexNet (for representation learning tests), as well as representation learning tests. It is in Caffe and is no longer supported. Please see the caffe branch for it.

Citation

If you find these models useful for your resesarch, please cite with these bibtexs.

@inproceedings{zhang2016colorful,
  title={Colorful Image Colorization},
  author={Zhang, Richard and Isola, Phillip and Efros, Alexei A},
  booktitle={ECCV},
  year={2016}
}

@article{zhang2017real,
  title={Real-Time User-Guided Image Colorization with Learned Deep Priors},
  author={Zhang, Richard and Zhu, Jun-Yan and Isola, Phillip and Geng, Xinyang and Lin, Angela S and Yu, Tianhe and Efros, Alexei A},
  journal={ACM Transactions on Graphics (TOG)},
  volume={9},
  number={4},
  year={2017},
  publisher={ACM}
}

Misc

Contact Richard Zhang at rich.zhang at eecs.berkeley.edu for any questions or comments.

colorization's Issues

Model is inaccessible (403)

$ wget http://eecs.berkeley.edu/~rich.zhang/projects/2016_colorization/files/demo_v0/colorization_release_v0.caffemodel
--2016-04-01 11:10:19--  http://eecs.berkeley.edu/~rich.zhang/projects/2016_colorization/files/demo_v0/colorization_release_v0.caffemodel
Resolving eecs.berkeley.edu (eecs.berkeley.edu)... 128.32.244.183
Connecting to eecs.berkeley.edu (eecs.berkeley.edu)|128.32.244.183|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2016-04-01 11:10:19 ERROR 403: Forbidden.

Are the probabilities in prior_probs.npy smoothed with a Gaussian kernel?

Hi,

I didn't find the code to generate the prior_probs.npy file, so I decided to ask here. Are the probabilities in the prior_probs.npy file raw empirical probabilities? Or are they smoothed probabilities?

Many thanks!

./train/fetch_caffe.sh gets different version of Caffe

It seems that ./train/fetch_caffe.sh doesn't get the modified version of Caffe that it is supposed get. The downloaded tar file opens to to a directory called caffe_private_pascal/ (instead of caffe-colorization/, which is what is called by ./train/train_model.sh), and the presets for Makefile.config.example does not match with what the model asks for. I had to change a bunch of things after Googling really hard and comment/uncomment a bunch of other things to get the model to to work. Did you remove the supported version of Caffe?

can the representation learning parts from the colorization model be trained independently?

See in the classification or segmentation task, the final test model is just the parts before the layer "conv8", and the structure is also a little different from the original parts in colorization model. Since these parts can be trained directly from imagent or pascal dataset for classification or segmentation, it is a little confusion to say that the colorization model holds the cross-channel encoding ability.

Is it right when direct using bgr order in testing?

Hi, I notice that the color channel order of the image data is different between training and testing. In training, there is explicit BGR->RGB conversion, while in testing the image is inputed and directly used for rgb2lab. This is different, why not also do BGR->RGB? is that right?

A very simple net to colorize B&W photos designed for deep learning beginners

We use a very simple model for colorizing B&W photos:

Following is the result using only 179 train samples, no transfer learning, no data augmentation

input	output

To get a better result, simply increase the number and size of train samples, but which will take much more time to train
Following is the result using 645 train samples:

input	output

https://github.com/microic/niy/blob/master/examples/colorizing_photos

Most colorizations (with people) are brownish

Quite a lot of grayscale pictures (taken from black & white movies) are colorized into brownish color.
How that can be improved ?

Prior_probs issue

Question regarding capability

Hey guys! I am really impressed by this library.

So I have a video fingerprinting technology (no, not histogram, perceptual identification) which is pretty freaking accurate to a high resolution of precision (10 seconds from the source video is enough to match). The only "weak point" I have is when color saturation is changed from the source video.

Something I've always really wanted was a way to "normalize" the color. In fact, since my system works way better with color than black-and-white video, colorizing a black and white video would be awesome! This is something I absolutely must play with.

The question I have that you may be able to answer is whether this would produce reasonably consistent results:

1. Take source image, copy it, make it black and white
2. Take source image, copy it, saturate the color a bit, make it black and white
3. Colorize the result of #1
4. Colorize the result of #2

When you compare those two, how similar are they? Assume reasonable saturation change so it does not harm the user perception of the original image too much.

If you know this off-hand, that'd be amazing! (I can't test right now)

--Collin

K-Means Init Error

I would like to add more layers to the Network and init it using the magic_init_mod.py. When i tried to run the following command:

python resources/magic_init/magic_init_mod.py 'path/models/colorization_train_val_v2_custome_30.prototxt' 30_layers.caffemodel -d 'img/ILSVRC2012_test_00005167.JPEG' -q -nit 10 -cs

Everything went right until the last line showed an error:

Traceback (most recent call last):
File "resources/magic_init/magic_init_mod.py", line 511, in
main()
File "resources/magic_init/magic_init_mod.py", line 507, in main
calibrateGradientRatio(n)
File "resources/magic_init/magic_init_mod.py", line 312, in calibrateGradientRatio
output_std = np.mean(np.std(flattenData(active_data[last_tops[0]]), axis=0))
File "resources/magic_init/magic_init_mod.py", line 29, in flattenData
return np.concatenate([d.swapaxes(0, 1).reshape((d.shape[1],-1)) for d in data], axis=1).T
ValueError: bad axis1 argument to swapaxes

I've also tried the default model which is "colorization_train_val_v2.prototxt'" but still had the same issue

what are the meaning of such values in file "pts_in_hull.py"

Hi, see the datas in pts_in_hull.py:

934e 554d 5059 0100 4600 7b27 6465 7363
7227 3a20 273c 6938 272c 2027 666f 7274
7261 6e5f 6f72 6465 7227 3a20 4661 6c73
652c 2027 7368 6170 6527 3a20 2833 3133
2c20 3229 2c20 7d20 2020 2020 2020 200a
a6ff ffff ffff ffff 3200 0000 0000 0000
a6ff ffff ffff ffff 3c00 0000 0000 0000
................................................................
Does these hex numbers are the values of the "ab" output space？But why they are discontinuous? And why the last line in this file is 318th line?

puzzled on color conversion

In some prototxt files, the layer "ColorConv" is used to deal with rgb->lab. But in other prototxt files, the python code "color.rgb2lab" take the place. why do not use the same way? In addition, the layer ColorConv" can not be found in normal Caffe version. what the data type (0-1,or 0-255) should be used when adopting ColorConv?

why the colorized image take the same colour

Hi, we take our own color image data set to train the model. yes, it works. but after training when testing, we found that the output colorised image take the same ab value. it is very strange. Have you met the same problem previously?

Protobuf version incompatibility

Hello. I've cloned this repo and compiled it according to instructions. But there is raised error with following text when I try to run a training.

[libprotobuf FATAL google/protobuf/stubs/common.cc:61] This program requires version 3.2.0 of the Protocol Buffer runtime library, but the installed version is 2.6.1. Please update your library. If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library. (Version verification failed in "google/protobuf/descriptor.pb.cc".) terminate called after throwing an instance of 'google::protobuf::FatalException' what(): This program requires version 3.2.0 of the Protocol Buffer runtime library, but the installed version is 2.6.1. Please update your library. If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library. (Version verification failed in "google/protobuf/descriptor.pb.cc".) *** Aborted at 1489510280 (unix time) try "date -d @1489510280" if you are using GNU date *** PC: @ 0x7f06e52e5428 gsignal *** SIGABRT (@0x3e8000032fd) received by PID 13053 (TID 0x7f06e7ff4740) from PID 13053; stack trace: *** @ 0x7f06e52e54b0 (unknown) @ 0x7f06e52e5428 gsignal @ 0x7f06e52e702a abort @ 0x7f06e60f884d __gnu_cxx::__verbose_terminate_handler() @ 0x7f06e60f66b6 (unknown) @ 0x7f06e60f6701 std::terminate() @ 0x7f06e60f6919 __cxa_throw @ 0x7f06de5d1647 google::protobuf::internal::LogMessage::Finish() @ 0x7f06de5d187d google::protobuf::internal::VerifyVersion() @ 0x7f06d5a1f0d4 google::protobuf::protobuf_google_2fprotobuf_2fdescriptor_2eproto::TableStruct::InitDefaultsImpl() @ 0x7f06de5d1f75 google::protobuf::GoogleOnceInitImpl() @ 0x7f06d5a19d85 google::protobuf::protobuf_google_2fprotobuf_2fdescriptor_2eproto::InitDefaults() @ 0x7f06d5a19db9 google::protobuf::protobuf_google_2fprotobuf_2fdescriptor_2eproto::AddDescriptorsImpl() @ 0x7f06de5d1f75 google::protobuf::GoogleOnceInitImpl() @ 0x7f06d5a19e35 google::protobuf::protobuf_google_2fprotobuf_2fdescriptor_2eproto::AddDescriptors() @ 0x7f06e7e0d4ea (unknown) @ 0x7f06e7e0d5fb (unknown) @ 0x7f06e7e12712 (unknown) @ 0x7f06e7e0d394 (unknown) @ 0x7f06e7e11bd9 (unknown) @ 0x7f06d750ef09 (unknown) @ 0x7f06e7e0d394 (unknown) @ 0x7f06d750f571 (unknown) @ 0x7f06d750efa1 dlopen @ 0x7f06e596c88d _PyImport_GetDynLoadFunc @ 0x7f06e59db4be _PyImport_LoadDynamicModule @ 0x7f06e59dc300 (unknown) @ 0x7f06e59dc5c8 (unknown) @ 0x7f06e59dd6db PyImport_ImportModuleLevel @ 0x7f06e5954698 (unknown) @ 0x7f06e59a01e3 PyObject_Call @ 0x7f06e5a76447 PyEval_CallObjectWithKeywords Aborted (core dumped)

https://github.com/richzhang/colorization/pullsFull log of training run is here. I use Ubuntu 16.04 LTS system.
I tried to install Protobuf 3.2.0 from Ubuntu 16.10 ppa (https://launchpad.net/~maarten-fonville/), but error persists.

Is this bug? Or maybe I am doing something wrong? Where can find any information?
Thanks

error trainning print 'Initial iteration: %i'%(self.iter+1) SyntaxError: invalid syntax

I am trainning colorization but it apperence a bug, can you tell me it come from you project or my lmdb dataset? I am newbie and hear is log:

I0816 11:17:49.565140 12297 layer_factory.hpp:77] Creating layer data
I0816 11:17:49.574599 12297 db_lmdb.cpp:35] Opened lmdb /home/ID/caffe-rc5/examples/draw/draw_train_lmdb
I0816 11:17:49.574626 12297 net.cpp:84] Creating Layer data
I0816 11:17:49.574637 12297 net.cpp:380] data -> data
I0816 11:17:49.574985 12297 data_layer.cpp:45] output data size: 40,3,176,176
I0816 11:17:49.575086 12297 net.cpp:122] Setting up data
I0816 11:17:49.575095 12297 net.cpp:129] Top shape: 40 3 176 176 (3717120)
I0816 11:17:49.575106 12297 net.cpp:137] Memory required for data: 14868480
I0816 11:17:49.575116 12297 layer_factory.hpp:77] Creating layer img_lab
I0816 11:17:49.577070 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.578773 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.580446 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.582123 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.583827 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.585499 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.587162 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.589007 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.590741 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.592401 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.594074 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.595671 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.597316 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.599025 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.600615 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.602229 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.603808 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.605463 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.607041 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.608664 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.610369 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.611999 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.613672 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.615299 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.616926 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.618705 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.620426 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.622099 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.623750 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.625425 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.627096 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.628756 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.630365 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.631958 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.633611 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.635215 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.636791 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.638662 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.640295 12298 data_layer.cpp:73] Restarting data prefetching from start.
I0816 11:17:49.641973 12298 data_layer.cpp:73] Restarting data prefetching from start.
File "/home/ID/colorization/resources/caffe_traininglayers.py", line 213
print 'Initial iteration: %i'%(self.iter+1)
^
SyntaxError: invalid syntax

Thank you for help!

Unknown layer type: SoftmaxCrossEntropyLoss

Check failed: registry.count(type) == 1 (0 vs. 1) Unknown layer type: SoftmaxCrossEntropyLoss (known types: AbsVal, Accuracy, ArgMax, BNLL, BatchNorm, BatchReindex, Bias, Concat, ContrastiveLoss, Convolution, Crop, Data, Deconvolution, Dropout, DummyData, ELU, Eltwise, Embed, EuclideanLoss, Exp, Filter, Flatten, HDF5Data, HDF5Output, HingeLoss, Im2col, ImageData, InfogainLoss, InnerProduct, Input, LRN, LSTM, LSTMUnit, Log, MVN, MemoryData, MultinomialLogisticLoss, PReLU, Parameter, Pooling, Power, Python, RNN, ReLU, Reduction, Reshape, SPP, Scale, Sigmoid, SigmoidCrossEntropyLoss, Silence, Slice, Softmax, SoftmaxWithLoss, Split, TanH, Threshold, Tile, WindowData)

I have already write the below code in ~/.bashrc, also reboot
export PYTHONPATH=/home/my/Documents/my/colorization/resources/:$PYTHONPATH

Inconsistency between code and paper

In paper the loss is defined as :
L = - sum( v * sum(Z * log(Z_cap) ) ),
where we can take advantage of Python broadcasting and can write it as
L = -sum( v * Z * log(Z_cap) )

where as in code conv8_313_boost layer, which is ClassRebalanceMultLayer layer, bottom blob is passed as it is. Which makes the loss:
L = -sum( Z * log(Z_cap) )

Considering even commented code which is
top[0].data[...] = bottom[0].data[...]*bottom[1].data[...]
where bottom[0] can be considered as Z_cap, and bottom[1] can be considered as v.
Due to this, the loss will be computed as :
L = -sum( Z * log( v * Z_cap) )
Is it intentional ?

Also in ClassRebalanceMultLayer, forward pass is not doing anything, but backward pass is boosting gradients according to weighting terms. Can you explain the logic behind this ?

Training on small images

Hello. In present model, images are cropped to 176x176. If I want to train this on small images(from 32x32) what should I do? Ho do I modify model? Thanks

typo in solver file

The solver file ./train/solver.prototxt has a typo that causes it to fail:

Error parsing text-format caffe.SolverParameter: 5:1: Invalid value for boolean field "test_initialization". Value: "False".

The problem is a capitalised "false":

test_initialization: False # don't run validation when loading

This should be false not False.

crash on the final step

after set up Caffe ( all test passed) and other requirement,
when I running to the final step( (3) Colorization time part ) in colorization_demo_v0.ipynb

python always crash. ("Python quit unexpectedly.")
then I try to run the script without ipython notebook, got the following error:
......
I0406 20:55:53.224961 1960824832 net.cpp:219] conv3_2 does not need backward computation.
I0406 20:55:53.225069 1960824832 net.cpp:219] relu3_1 does not need backward computation.
I0406 20:55:53.225086 1960824832 net.cpp:219] conv3_1 does not need backward computation.
I0406 20:55:53.225095 1960824832 net.cpp:219] conv2_2norm does not need backward computation.
I0406 20:55:53.225101 1960824832 net.cpp:219] relu2_2 does not need backward computation.
I0406 20:55:53.225108 1960824832 net.cpp:219] conv2_2 does not need backward computation.
I0406 20:55:53.225116 1960824832 net.cpp:219] relu2_1 does not need backward computation.
I0406 20:55:53.225123 1960824832 net.cpp:219] conv2_1 does not need backward computation.
I0406 20:55:53.225131 1960824832 net.cpp:219] conv1_2norm does not need backward computation.
I0406 20:55:53.225138 1960824832 net.cpp:219] relu1_2 does not need backward computation.
I0406 20:55:53.225145 1960824832 net.cpp:219] conv1_2 does not need backward computation.
I0406 20:55:53.225152 1960824832 net.cpp:219] relu1_1 does not need backward computation.
I0406 20:55:53.225358 1960824832 net.cpp:219] bw_conv1_1 does not need backward computation.
I0406 20:55:53.225379 1960824832 net.cpp:219] input does not need backward computation.
I0406 20:55:53.225431 1960824832 net.cpp:274] Network initialization done.
I0406 20:55:53.444080 1960824832 upgrade_proto.cpp:66] Attempting to upgrade input file specified using deprecated input fields: colorization_release_v0.caffemodel
I0406 20:55:53.444106 1960824832 upgrade_proto.cpp:69] Successfully upgraded file specified using deprecated input fields.
W0406 20:55:53.444110 1960824832 upgrade_proto.cpp:71] Note that future Caffe releases will only support input layers and not input fields.
F0406 20:56:02.908959 1960824832 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0) out of memory

Test on macbook pro with 2GB GPU memory, maybe it need more GPU memory?

the question about data labels

Hello，i want to ask you the trainning and test datas need labels？Thank your response very much.

Training crash

Hi,

I used my own data to train the net, but got the below error message. Could you please show some hints? Thanks.

I1203 01:20:34.406128 11072 net.cpp:165] Memory required for data: 7864320
I1203 01:20:34.406141 11072 layer_factory.hpp:77] Creating layer nongray_mask
Traceback (most recent call last):
File "/tmp/colorization/resources/caffe_traininglayers.py", line 4, in
import numpy as np
File "/usr/local/lib/python2.7/dist-packages/numpy/init.py", line 142, in
from . import add_newdocs
File "/usr/local/lib/python2.7/dist-packages/numpy/add_newdocs.py", line 13, in
from numpy.lib import add_newdoc
File "/usr/local/lib/python2.7/dist-packages/numpy/lib/init.py", line 8, in
from .type_check import *
File "/usr/local/lib/python2.7/dist-packages/numpy/lib/type_check.py", line 11, in
import numpy.core.numeric as _nx
File "/usr/local/lib/python2.7/dist-packages/numpy/core/init.py", line 58, in
from numpy.testing.nosetester import numpy_tester
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/init.py", line 12, in
from . import decorators as dec
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/decorators.py", line 21, in
from .utils import SkipTest
File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py", line 15, in
from tempfile import mkdtemp, mkstemp
File "/usr/lib/python2.7/tempfile.py", line 32, in
import io as io
File "/tmp/colorization/caffe-colorization/python/caffe/io.py", line 2, in
import skimage.io
File "/usr/lib/python2.7/dist-packages/skimage/init.py", line 166, in
from .util.dtype import *
File "/usr/lib/python2.7/dist-packages/skimage/util/init.py", line 1, in
from .dtype import (img_as_float, img_as_int, img_as_uint, img_as_ubyte,
File "/usr/lib/python2.7/dist-packages/skimage/util/dtype.py", line 8, in
dtype_range = {np.bool: (False, True),
AttributeError: 'module' object has no attribute 'bool'

Segmentation fault when running ./train/train_model.sh

When running ./train/train_model.sh, I got the following error:

I0418 09:54:54.054507 20872 layer_factory.hpp:77] Creating layer data
I0418 09:54:54.054597 20872 db_lmdb.cpp:35] Opened lmdb ./caffe-colorization/examples/imagenet/ilsvrc12_train_lmdb
I0418 09:54:54.054622 20872 net.cpp:84] Creating Layer data
I0418 09:54:54.054630 20872 net.cpp:380] data -> data
I0418 09:54:54.055681 20872 data_layer.cpp:45] output data size: 40,3,176,176
I0418 09:54:54.076776 20872 net.cpp:122] Setting up data
I0418 09:54:54.076802 20872 net.cpp:129] Top shape: 40 3 176 176 (3717120)
I0418 09:54:54.076805 20872 net.cpp:137] Memory required for data: 14868480
I0418 09:54:54.076812 20872 layer_factory.hpp:77] Creating layer img_lab
*** Aborted at 1492480494 (unix time) try "date -d @1492480494" if you are using GNU date ***
PC: @ 0x7f3516e6e873 std::_Hashtable<>::clear()
*** SIGSEGV (@0x9) received by PID 20872 (TID 0x7f352eb42740) from PID 9; stack trace: ***
@ 0x7f352bd584b0 (unknown)
@ 0x7f3516e6e873 std::_Hashtable<>::clear()
@ 0x7f3516e60346 google::protobuf::DescriptorPool::FindFileByName()
@ 0x7f3516e3eac8 google::protobuf::python::cdescriptor_pool::AddSerializedFile()
@ 0x7f352c3c17d0 PyEval_EvalFrameEx
@ 0x7f352c4ea01c PyEval_EvalCodeEx
@ 0x7f352c4403dd (unknown)
@ 0x7f352c4131e3 PyObject_Call
@ 0x7f352c433ae5 (unknown)
@ 0x7f352c3ca123 (unknown)
@ 0x7f352c4131e3 PyObject_Call
@ 0x7f352c3be13c PyEval_EvalFrameEx
@ 0x7f352c4ea01c PyEval_EvalCodeEx
@ 0x7f352c3b8b89 PyEval_EvalCode
@ 0x7f352c44d1b4 PyImport_ExecCodeModuleEx
@ 0x7f352c44db8f (unknown)
@ 0x7f352c44f300 (unknown)
@ 0x7f352c44f5c8 (unknown)
@ 0x7f352c4506db PyImport_ImportModuleLevel
@ 0x7f352c3c7698 (unknown)
@ 0x7f352c4131e3 PyObject_Call
@ 0x7f352c4e9447 PyEval_CallObjectWithKeywords
@ 0x7f352c3bc5c6 PyEval_EvalFrameEx
@ 0x7f352c4ea01c PyEval_EvalCodeEx
@ 0x7f352c3b8b89 PyEval_EvalCode
@ 0x7f352c44d1b4 PyImport_ExecCodeModuleEx
@ 0x7f352c44db8f (unknown)
@ 0x7f352c44f300 (unknown)
@ 0x7f352c44f5c8 (unknown)
@ 0x7f352c4506db PyImport_ImportModuleLevel
@ 0x7f352c3c7698 (unknown)
@ 0x7f352c4131e3 PyObject_Call
./train/train_model.sh: line 2: 20872 Segmentation fault ./caffe-colorization/build/tools/caffe train -solver ./train/solver.prototxt -weights ./models/init_v2.caffemodel -gpu $1

My GPU is GTX1070-8G, memory is 32G and system is Ubuntu16.04.
The ImageNet-lmdb-file was created with ./caffe-colorization/examples/imagenet/create_imagenet.sh without resizing, resized to 256x256 and resized to 176x176, but all three cases came to the same error.
I think maybe it relates to protobuf, so I tried protobuf-v3.2.0, protobuf-v3.2.0-rc.1, protobuf-v3.2.0rc2 and protobuf-v3.2.1, but again all cases came to the same error.
With the lastest caffe-1.0, the error is same.
Can anyone help me about this "Segmentation fault" issue?
Thanks appreciately!

This division returns an array of 1s in my case. Is this to be expected?

colorization/interactive-deep-colorization/caffe_files/color_quantization.py

Line 30 in ab39729

wts = wts/np.sum(wts,axis=1)[:,util.na()]

8888

training

hi did you meet this problem?
I1015 14:42:04.869479 6838 net.cpp:165] Memory required for data: 743424
I1015 14:42:04.869496 6838 layer_factory.hpp:77] Creating layer img_lab
*** Aborted at 1508049725 (unix time) try "date -d @1508049725" if you are using GNU date ***
PC: @ 0x7f016bf717a3 std::_Hashtable<>::clear()
*** SIGSEGV (@0x9) received by PID 6838 (TID 0x7f01ca36fa80) from PID 9; stack trace: ***
@ 0x7f01c75fa4b0 (unknown)
@ 0x7f016bf717a3 std::_Hashtable<>::clear()
@ 0x7f016bf6a116 google::protobuf::DescriptorPool::FindFileByName()
@ 0x7f016bf23f98 google::protobuf::python::cdescriptor_pool::AddSerializedFile()
@ 0x7f01c7c657d0 (unknown)
@ 0x7f01c7d8e01c (unknown)
@ 0x7f01c7ce43dd (unknown)
@ 0x7f01c7cb71e3 (unknown)
@ 0x7f01c7cd7ae5 (unknown)
@ 0x7f01c7c6e123 (unknown)
@ 0x7f01c7cb71e3 (unknown)
@ 0x7f01c7c6213c (unknown)
@ 0x7f01c7d8e01c (unknown)
@ 0x7f01c7c5cb89 (unknown)
@ 0x7f01c7cf11b4 (unknown)
@ 0x7f01c7cf1b8f (unknown)
@ 0x7f01c7cf3300 (unknown)
@ 0x7f01c7cf35c8 (unknown)
@ 0x7f01c7cf46db (unknown)
@ 0x7f01c7c6b698 (unknown)
@ 0x7f01c7cb71e3 (unknown)
@ 0x7f01c7d8d447 (unknown)
@ 0x7f01c7c605c6 (unknown)
@ 0x7f01c7d8e01c (unknown)
@ 0x7f01c7c5cb89 (unknown)
@ 0x7f01c7cf11b4 (unknown)
@ 0x7f01c7cf1b8f (unknown)
@ 0x7f01c7cf3300 (unknown)
@ 0x7f01c7cf35c8 (unknown)
@ 0x7f01c7cf46db (unknown)
@ 0x7f01c7c6b698 (unknown)
@ 0x7f01c7cb71e3 (unknown)
./train/train_model.sh: line 2: 6838 Segmentation fault (core dumped) /home/weiwei/work/cccv2017/caffe/build/tools/caffe train -solver ./train/solver.prototxt -weights ./models/init_v2.caffemodel -gpu $1

Demo do not show a colored image

HI @richzhang .
I followed the instructions as the same as the notebook says.But I did NOT get a colored image .Instead I got a strange image as below:

This is my test code(I just copy them from the notebook) :

import numpy as np
import matplotlib.pyplot as plt
import os
import skimage.color as color
import scipy.ndimage.interpolation as sni
import caffe

plt.rcParams['figure.figsize'] = (12, 6)

gpu_id = 0
caffe.set_mode_gpu()
caffe.set_device(gpu_id)

Select desired model

net = caffe.Net('./models/colorization_deploy_v1.prototxt', './models/colorization_release_v2.caffemodel', caffe.TEST)

net = caffe.Net('../models/colorization_deploy_v2.prototxt', '../models/colorization_release_v2_norebal.caffemodel', caffe.TEST)

net = caffe.Net('../models/colorization_deploy_v1.prototxt', '../models/colorization_release_v1.caffemodel', caffe.TEST)

If you are training your own network, you may replace the *.caffemodel path with your trained network.

(H_in,W_in) = net.blobs['data_l'].data.shape[2:] # get input shape
(H_out,W_out) = net.blobs['class8_ab'].data.shape[2:] # get output shape
print 'Input dimensions: (%i,%i)'%(H_in,W_in)
print 'Output dimensions: (%i,%i)'%(H_out,W_out)

pts_in_hull = np.load('./resources/pts_in_hull.npy') # load cluster centers
net.params['class8_ab'][0].data[:,:,0,0] = pts_in_hull.transpose((1,0)) # populate cluster centers as 1x1 convolution kernel
print 'Annealed-Mean Parameters populated'

load the original image

print 'load the original image'
img_rgb = caffe.io.load_image('./imgs/hbfj1.jpg')
img_lab = color.rgb2lab(img_rgb) # convert image to lab color space
img_l = img_lab[:,:,0] # pull out L channel
(H_orig,W_orig) = img_rgb.shape[:2] # original image size

create grayscale version of image (just for displaying)

print 'create grayscale version of image (just for displaying)'
img_lab_bw = img_lab.copy()
img_lab_bw[:,:,1:] = 0
img_rgb_bw = color.lab2rgb(img_lab_bw)

resize image to network input size

print 'resize image to network input size'
img_rs = caffe.io.resize_image(img_rgb,(H_in,W_in)) # resize image to network input size
img_lab_rs = color.rgb2lab(img_rs)
img_l_rs = img_lab_rs[:,:,0]

show original image, along with grayscale input to the network

print 'show original image, along with grayscale input to the network'
img_pad = np.ones((H_orig,W_orig/10,3))
plt.imshow(np.hstack((img_rgb, img_pad, img_rgb_bw)))
plt.title('(Left) Loaded image / (Right) Grayscale input to network')
plt.axis('off')

net.blobs['data_l'].data[0,0,:,:] = img_l_rs-50 # subtract 50 for mean-centering
net.forward() # run network

ab_dec = net.blobs['class8_ab'].data[0,:,:,:].transpose((1,2,0)) # this is our result
ab_dec_us = sni.zoom(ab_dec,(1._H_orig/H_out,1._W_orig/W_out,1)) # upsample to match size of original image L
img_lab_out = np.concatenate((img_l[:,:,np.newaxis],ab_dec_us),axis=2) # concatenate with

concatenate with original image L

img_rgb_out = np.clip(color.lab2rgb(img_lab_out),0,1) # convert back to rgb

plt.imshow(img_rgb_out);
plt.axis('off')
plt.show()

Am I missing something?
Thank you very much!

Which prototxt and class labels go with which model?

feature_learning_testing

Hi, Richard,

For the feature-learning-testing segmentation part, you firstly pre-train an AlexNet networkon the colourization task, and then fine-tune a FCN-AlexNet using the pre-trianed AlexNet model for segmentation on PASCAL VOC 2012, is it right?

If yes, however, I find the pre-trained model in the file solve.py for segmentation
"default='./models/alexnet_release_450000_nobn_rs.caffemodel')".
I am not sure the model "alexnet_release_450000_nobn_rs.caffemodel" is trained for colourization because the file 'train_val_nobn_rs.prototxt' which looks related to the model from their names is a classification task. Anything I understand wrongly?

Thank you very much.

Help me

[libprotobuf ERROR google/protobuf/text_format.cc:245] Error parsing text-format caffe.NetParameter: 213:13: Message type "caffe.ConvolutionParameter" has no field named "dilation".
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0402 13:04:55.491230 21321 upgrade_proto.cpp:928] Check failed: ReadProtoFromTextFile(param_file, param) Failed to parse NetParameter file: colorization_deploy_v0.prototxt
*** Check failure stack trace: ***

Ubuntu14.04 GTX9602G
I installed NVIDIA's caffe following the steps below
https://github.com/NVIDIA/DIGITS/blob/master/docs/UbuntuInstall.md

Thank you richzhang

Solved

the problem of caffe_traininglayers.py

Hi, we take our own color image data set to train the model.but we encounter a question,which is your caffe_traininglayers.py. When running ./train/train_model.sh, I got the following error:TypeError: 'float' object cannot be interpreted as an index .As follows:
I0612 09:26:00.048768 1935 caffe.cpp:221] Starting Optimization
I0612 09:26:00.048841 1935 solver.cpp:279] Solving LtoAB
I0612 09:26:00.048853 1935 solver.cpp:280] Learning Rate Policy: step
Traceback (most recent call last):
File "/home/joy/colorization-master/resources/caffe_traininglayers.py", line 72, in forward
top[0].data[...] = self.nnenc.encode_points_mtx_nd(bottom[0].data[...],axis=1)
File "/home/joy/colorization-master/resources/caffe_traininglayers.py", line 329, in encode_points_mtx_nd
(dists,inds) = self.nbrs.kneighbors(pts_flt)
File "/home/dpa/anaconda2/lib/python2.7/site-packages/sklearn/neighbors/base.py", line 399, in kneighbors
for s in gen_even_slices(X.shape[0], n_jobs)
File "/home/dpa/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 800, in call
while self.dispatch_one_batch(iterator):
File "/home/dpa/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 658, in dispatch_one_batch
self._dispatch(tasks)
File "/home/dpa/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 566, in _dispatch
job = ImmediateComputeBatch(batch)
File "/home/dpa/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 180, in init
self.results = batch()
File "/home/dpa/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 72, in call
return [func(*args, **kwargs) for func, args, kwargs in self.items]
File "sklearn/neighbors/binary_tree.pxi", line 1310, in sklearn.neighbors.ball_tree.BinaryTree.query (sklearn/neighbors/ball_tree.c:10592)
File "sklearn/neighbors/binary_tree.pxi", line 588, in sklearn.neighbors.ball_tree.NeighborsHeap.init (sklearn/neighbors/ball_tree.c:4931)
TypeError: 'float' object cannot be interpreted as an index
Can anyone help me about this issue?
Thanks appreciately!

incorrect caffe paths in ./train scripts

As per the Training Usage section, running ./train/fetch_caffe.sh creates a ./caffe_private_pascal folder, not a ./caffe-colorization folder, for caffe installation.
So the paths in your training scripts in ./train should also be changed accordingly.

thanks,
Aruni

Training not saving snapshots

I hit around 10,000 iterations so far on a training project, but my snapshots folder train/colornet is empty. I have stopped training for the time being, Any idea why this might be happening? At first the script would not run unless i created and properly set permissions on those folders, and once I did it ran. So I know it looked for them. But it's not saving anything to them.

Demo does not run with caffe-cpu

Hi Richzhang! I run colorization with demo by python3 and caffe cpu-only. When I run 'Test-Time Demo in IPython Notebook':

python3 ./colorize.py -img_in ./demo/imgs/ILSVRC2012_val_00041580.JPEG -img_out ./out.png --gpu 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0811 16:05:14.388861 419 common.cpp:66] Cannot use GPU in CPU-only Caffe: check mode.
*** Check failure stack trace: ***
Aborted (core dumped)

Can you support me?

Anaconda Windows

Hello.
I'm trying to start "colorization" on windows.
Since the beginning I have successfully launched a "Deep Dream". ( Using this manual ). But when you try to use the same method "colorization" I can't run it successfully.
On a piece of code (probably)
net = caffe.Net('../models/colorization_deploy_v2.prototxt', '../models/colorization_release_v2.caffemodel', caffe.TEST)
Python just crash...
I tried everything I could and I do not know where there may be a problem.
Please, I need help. And I am sorry that I ask it in here. I do not know where else I can ask for help.

--Update--

Error: imgur

possible SciKit Learn version issue

When training the network from scratch using ./train/train_model.sh 0, the following error happens at the "Solving LtoAB" step of the training:

caffe_traininglayers.py", line 331, in encode_points_mtx_nd
    (dists,inds) = self.nbrs.kneighbors(pts_flt)

....

  File "sklearn/neighbors/binary_tree.pxi", line 1309, in sklearn.neighbors.ball_tree.BinaryTree.query (sklearn/neighbors/ball_tree.c:11514)
  File "sklearn/neighbors/binary_tree.pxi", line 587, in sklearn.neighbors.ball_tree.NeighborsHeap.__init__ (sklearn/neighbors/ball_tree.c:5582)
TypeError: 'float' object cannot be interpreted as an index

The data pts_flt is a float32 numpy ndarray. Could this be due to a version problem in sklearn itself (I am using 0.18.1)?
Please let me know what version of scikit is used for this codebase and I'll match that and try training again.

thanks,
Aruni

I still get the same error even using the latest caffe, please give some advice

HI,Rich zhang:
I cloned the latest version of caffe and still get the same result as "[libprotobuf ERROR google/protobuf/text_format.cc:245] Error parsing text-format caffe.NetParameter: 213:13: Message type "caffe.ConvolutionParameter" has no field named "dilation".".
I am using jupyter notebook on the server and chrome browser on the client. Could you give me any suggestion?Thanks.

Need a clarification why exactly the number 313

Hi,

I know that this is not the place for asking questions about the theory behind your research, but just I wanted to clarify the derivation of the number 313. I am trying to do a similar experiment on my own (https://github.com/preslavrachev/nn-photo-colorization), but I am approaching it from the ground up, basically setting up my own model and everything. I ended up reading your paper, and I got really interested in integrating some of your findings into my experiment. I understand the logic behind the ab space classification, but I can't exactly figure out why 313. Is this a heuristic you came up with, or is it something trivial that I'm just missing on?

I have to say, I am very interested to ML, but pretty new to the field, so please, excuse my ignorance, in case I have missed something extremely trivial.

Me

Data Load Error

Hello, I am trying to run the demo but the code fails on the model load, here is the last lines of the log:

I0422 05:05:37.127264 11607 net.cpp:274] Network initialization done.
I0422 05:05:37.197033 11607 upgrade_proto.cpp:66] Attempting to upgrade input file specified using deprecated input fields: colorization_release_v0.caffemodel
I0422 05:05:37.197067 11607 upgrade_proto.cpp:69] Successfully upgraded file specified using deprecated input fields.
W0422 05:05:37.197072 11607 upgrade_proto.cpp:71] Note that future Caffe releases will only support input layers and not input fields.
F0422 05:05:37.197129 11607 net.cpp:765] Check failed: target_blobs.size() == source_layer.blobs_size() (5 vs. 3) Incompatible number of blobs for layer conv1_2norm

Here is the full log output if you need it. Gist

I am using the latest Caffe with cuDNNv5 on Ubuntu 16.04. Also, my Caffe is compiled from this PR for the cuDNNv5 support.

Update: I suppose that PR adds some breaking changes in the batch normalization (now, cuDNN do that) that raise this issue.

Update #2: I think, the issue in this change.

can the model handle any size test images?

Hi, thank you for your excellent research , your paper uses the 224 x 244 image for training, I would like to ask can the training model handle any size image?

How much time it will take to train to get the final result

by using GTX TITAN x,HOW MUCH TIME IT WILL TAKES? THANK YOU !

Details of detection and segmentation

Hi, I want to know some details of detection and segmentation experiments. What's the initial learning rate used for detection and segmentation? Were all of the methods presented in Table 2 trained using the same learning rates for detection and segmentation or different ones for distinct methods? Thanks!

Colorization on 1929 movie

Hi

Thanks for this script ! I tried your colorization (v2 with class rebalancing) on some frames from Man With a Movie Camera (Vertov, 1929) here are the results :
https://drive.google.com/open?id=0B5-1OeNPsecwdTZ4dS0wUUhIaUU

• The results are all oranges :( what do you think I can do to have a better quality ?
• Should I train my own network with frames from documentaries ?
• Do you think I should start training from scratch or start from your network v2 ?
• Could we introduce some kind of temporal consistency across a bunch of frames for movie colorization ?

Thanks in advance !
Antoine.

wget command not found

When running fetch_release_models.sh in models directory I get the above error.
I also went to the referenced urls in the file and they are not present on the website.

These are the links that file is referencing:
wget http://eecs.berkeley.edu/~rich.zhang/projects/2016_colorization/files/demo_v2/colorization_release_v2.caffemodel -O ./models/colorization_$
wget http://eecs.berkeley.edu/~rich.zhang/projects/2016_colorization/files/demo_v2/colorization_release_v2_norebal.caffemodel -O ./models/color$
wget http://eecs.berkeley.edu/~rich.zhang/projects/2016_colorization/files/demo_v1/colorization_release_v1.caffemodel -O ./models/colorization_$

Caffe-cpu in trainning and create lmdb

Hi Rich Zhang! I am trying to train colorization with caffe-cpu on ubuntu 18.04. After I run ./train/fetch_caffe.sh the ./caffe-colorization does not exist, instead ./caffe_private_pascal appear and how to import image by mysefl not use image from imagenet.org.
Can you support me? I am a newbie

Can't find problem

at start I see only alert:
/home/yaroslav/caffe/python/caffe/pycaffe.py:13: RuntimeWarning: to-Python converter for boost::shared_ptrcaffe::Net already registered; second conversion method ignored.
from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver,
/home/yaroslav/caffe/python/caffe/pycaffe.py:13: RuntimeWarning: to-Python converter for boost::shared_ptrcaffe::Blob already registered; second conversion method ignored.
from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver,
/home/yaroslav/caffe/python/caffe/pycaffe.py:13: RuntimeWarning: to-Python converter for boost::shared_ptrcaffe::Solver already registered; second conversion method ignored.
from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \

but later kernel crashes with that error in console:
F0407 23:49:31.456502 20193 upgrade_proto.cpp:86] Check failed: ReadProtoFromBinaryFile(param_file, param) Failed to parse NetParameter file: colorization_release_v0.caffemodel
*** Check failure stack trace: ***
2016-04-07 23:49:32.242 [NotebookApp] KernelRestarter: restarting kernel (1/5)

using cuda 7.5 and latest caffe from https://github.com/BVLC/caffe/

matlab

i want to compile the included caffe for use in training. looks like matlab is required? Can i comment out the matlab line in the makefile?

Prokudin-Gorskii Collection showing limits of the Algorithm

so I found the Prokudin-Gorskii photo collection of early color prints, and to a benefit, the collection includes the original B+W negatives, which allowed for easy comparison. some images such as this (the original color print for comparison here ) turn out amazingly well. However, to contrast that, some of them turn out quite bad such as this one, especially compared to the digitally retouched version of the original. is this perhaps just an issue of needing more training data?