mlberkeley / oreilly-captions

Caption This!

This repository contains source code corresponding to our article "Caption this, with TensorFlow!"

Git Basics

1. Go to your home directory by opening your terminal and entering cd ~

2. Clone the repository by entering

   git clone https://github.com/mlberkeley/oreilly-captions.git
   

Docker (highly recommended)

Install Docker using the platform-specific installation instructions for Docker here. Our iPython notebooks are compatible with TensorFlow 1.0.

Option A: Use our pre-built Docker image from Docker Hub

3. After installing Docker, pull a prebuilt image from our Docker Hub by entering:

   docker pull mlatberkeley/showandtell
   

   You will need a Docker Hub account in order to pull the image (get one here). If it's your first time pulling a Docker image from Docker Hub you will need to login to your Docker Hub account from your terminal with docker login, and follow the username and password prompt.

4. To run the pulled image (after cloning and downloading the repository) enter

   docker run -it -p 8888:8888 -v <path to repo>:/root mlatberkeley/showandtell
   

   where <path to repo> should be the absolute path to your cloned repository. If you followed our Git Basics section the path should be <path to your home directory>/oreilly-captions.

5. After building, starting, and attaching to the appropriate Docker container, run the provided Jupyter notebooks by entering

   jupyter notebook --ip 0.0.0.0
   

   and navigate to http://0.0.0.0:8888 in your browser.

Option B: Download and build your own Docker image from our GitHub repo

If you want to build a GPU or CPU-based Docker image of your own, you can use the Dockerfiles provided in the /dockerfiles/ subdirectory of our GitHub repo.

3. After cloning the repo to your machine, enter

   docker build -t showandtell_<image_type> -f ./dockerfiles/Dockerfile.<image_type> ./dockerfiles/
   

   where <image_type> is either gpu or cpu. (Note that, in order to run these files on your GPU, you'll need to have a compatible GPU, with drivers installed and configured properly as described in TensorFlow's documentation.)

4. Run the Docker image by entering

   docker run -it -p 8888:8888 -v <path to repo>:/root showandtell_<image_type>
   

   where <image_type> is either gpu or cpu, depending on the image you built in the last step.

5. After building, starting, and attaching to the appropriate Docker container, run the provided Jupyter notebooks by entering

   jupyter notebook --ip 0.0.0.0
   

   and navigate to http://0.0.0.0:8888 in your browser.

Note If you are using Docker Toolbox as opposed to native Docker you will have to navigate to the Daemon IP adress (instead of 0.0.0.0) provided right after starting the Docker Quickstart Terminal (for us this was 192.168.99.100) in order to use Jupyter.

Debugging docker

If you receive an error of the form:

WARNING: Error loading config file:/home/rp/.docker/config.json - stat /home/rp/.docker/config.json: permission denied
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.26/images/json: dial unix /var/run/docker.sock: connect: permission denied

It's most likely because you installed Docker using sudo permissions with a packet manager such as brew or apt-get. To solve this permission denied simply run docker with sudo (ie. run docker commands with sudo docker <command and options> instead of just docker <command and options>).

The Notebooks

There are three notebooks:

• 1. O'Reilly Training.ipynb - Contains code to train a TensorFlow caption generator from a VGG16 word embedding as described in our article. Note: you must run this notebook's train method before running any of the other notebooks in order to generate a mapping between integers and our vocabulary's words that will be reused in the other notebooks.
• 2. O'Reilly Generate.ipynb - Contains the same code as 1. O'Reilly Training.ipynb except it introduces functionality to generate captions from an image embedding (as opposed to just being able to train on captions). Functions as a sanity check for the quality of captions we are generating.
• 3. O'Reilly Generate from image.ipynb - Builds on the previous notebook, except instead of feeding an image embedding to our caption generation model, it first feeds an image to the VGG-16 Convolutional Neural Network to generate an image feature embedding. This gives us an end-to-end pipeline for going from an image to a caption.
• In order to run the test notebook edit the image path in the ipynb (more details in the .ipynb itself).

Additional Downloads:

In order to run the first two notebooks, you will need VGG-16 image embeddings for the Flickr-30K dataset. These image embeddings are available from our Google Drive.

Additionally, you will need the corresponding captions for these images (results_20130124.token), which can also be downloaded from our Google Drive.

In order to run the 3. O'Reilly Generate from image.ipynb notebook you will need to download a pretrained TensorFlow model for VGG-16 generated from the original Caffe model from the VGG-16 paper.

Place all of these downloads in the ./data/ directory.

Pretrained Weights:

We've trained the caption generator (w/o training VGG-16 End2End) to 500 epochs, and we've placed the resulting checkpoint files in ./models/tensorflow. You should experience an average reconstruction loss of ~1.75-1.85.

oreilly-captions's Issues

Notify someone on an issue with a mention

Notify someone on an issue with a mention

TypeError: unorderable types: str() < NoneType()

After training the part 1 file for 50 epochs...

In 2. O'Reilly Generate.ipynb, I get this error

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/mnt/a/sw/xe_xk_cle5.2UP02_pe2.3.0/bwpy/0.3.0/python-single/usr/lib/python3.4/site-packages/sitecustomize.d/syslog-imports.py", line 101, in master_log_imports
log_imports()
File "/mnt/a/sw/xe_xk_cle5.2UP02_pe2.3.0/bwpy/0.3.0/python-single/usr/lib/python3.4/site-packages/sitecustomize.d/syslog-imports.py", line 40, in wrapper
function(*args, **kwargs)
File "/mnt/a/sw/xe_xk_cle5.2UP02_pe2.3.0/bwpy/0.3.0/python-single/usr/lib/python3.4/site-packages/sitecustomize.d/syslog-imports.py", line 84, in log_imports
modulelist = ', '.join(list(impts))
File "/mnt/a/sw/xe_xk_cle5.2UP02_pe2.3.0/bwpy/0.3.0/python-single/usr/lib/python3.4/site-packages/sitecustomize.d/syslog-imports.py", line 60, in imports
if in_prefix(val.file):
File "/mnt/a/sw/xe_xk_cle5.2UP02_pe2.3.0/bwpy/0.3.0/python-single/usr/lib/python3.4/site-packages/sitecustomize.d/syslog-imports.py", line 51, in in_prefix
if len(os.path.commonprefix([file, s])) > 1:
File "/sw/bw/bwpy/0.3.0/python-single//usr/lib/python-exec/python3.4/../../../lib/python3.4/genericpath.py", line 72, in commonprefix
s1 = min(m)
TypeError: unorderable types: str() < NoneType()

KeyError: "The name 'import/Relu_1:0' refers to a Tensor which does not exist. The operation, 'import/Relu_1', does not exist in the graph."

Using Docker with Windows 10. All steps run in jupyter notebook except the last command on 3. O'Reilly Generate from image.ipynb

test(sess,image,generated_words,ixtoword, image_path)

KeyError Traceback (most recent call last)
in ()
----> 1 test(sess,image,generated_words,ixtoword, image_path)

in test(sess, image, generated_words, ixtoword, test_image_path)
4
5 feat = read_image(test_image_path)
----> 6 fc7 = sess.run(graph.get_tensor_by_name("import/Relu_1:0"), feed_dict={images:feat})
7
8 saver = tf.train.Saver()

/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/ops.py in get_tensor_by_name(self, name)
2620 raise TypeError("Tensor names are strings (or similar), not %s."
2621 % type(name).name)
-> 2622 return self.as_graph_element(name, allow_tensor=True, allow_operation=False)
2623
2624 def _next_id(self):

/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/ops.py in as_graph_element(self, obj, allow_tensor, allow_operation)
2471
2472 with self._lock:
-> 2473 return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
2474
2475 def _as_graph_element_locked(self, obj, allow_tensor, allow_operation):

/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/ops.py in _as_graph_element_locked(self, obj, allow_tensor, allow_operation)
2513 raise KeyError("The name %s refers to a Tensor which does not "
2514 "exist. The operation, %s, does not exist in the "
-> 2515 "graph." % (repr(name), repr(op_name)))
2516 try:
2517 return op.outputs[out_n]

KeyError: "The name 'import/Relu_1:0' refers to a Tensor which does not exist. The operation, 'import/Relu_1', does not exist in the graph."

Where is the pre-trained model?

Hello, I saw you placed the resulting checkpoint files in ./models/tensorflow. But I cannot find it, there is only one .keep file.

Trying to run on GPU

This is the error that I get when I try to build the gpu-based docker. I am unsure what I am doing wrong. I appreciate any advice/help. Thank you :)

AttributeError

When I ran the first Training code ,it reminded me that " AttributeError : module 'pandas' has no attribute 'computation' ". I have tried some methods to solve this problem , but failed . I don't know why this problem happended. Could someone help me solve this problem? I will appreciate that!

kernel not found in checkpoint

Using TensorFlow backend.
(158915, 4096)
(158915,)
Two young guys with shaggy hair look at their hands while hanging out in the yard .
preprocessing 30 word vocab
preprocessed words 20326 -> 2942
Traceback (most recent call last):
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1327, in _do_call
return fn(*args)
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1306, in _run_fn
status, run_metadata)
File "C:\Program Files\Python36\lib\contextlib.py", line 88, in exit
next(self.gen)
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: Key RNN/basic_lstm_cell/kernel not found in checkpoint
[[Node: save/RestoreV2_1 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_1/tensor_names, save/RestoreV2_1/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:/Users/SONY/Desktop/img_cap/training.py", line 206, in
train(.001,True,True) #continue training from pretrained weights @epoch500
File "C:/Users/SONY/Desktop/img_cap/training.py", line 176, in train
saver.restore(sess,tf.train.latest_checkpoint(model_path_transfer))
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 1560, in restore
{self.saver_def.filename_tensor_name: save_path})
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\client\session.py", line 895, in run
run_metadata_ptr)
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1321, in _do_run
options, run_metadata)
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key RNN/basic_lstm_cell/kernel not found in checkpoint
[[Node: save/RestoreV2_1 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_1/tensor_names, save/RestoreV2_1/shape_and_slices)]]

Caused by op 'save/RestoreV2_1', defined at:
File "", line 1, in
File "C:\Program Files\Python36\lib\idlelib\run.py", line 144, in main
ret = method(*args, **kwargs)
File "C:\Program Files\Python36\lib\idlelib\run.py", line 474, in runcode
exec(code, self.locals)
File "C:/Users/SONY/Desktop/img_cap/training.py", line 206, in
train(.001,True,True) #continue training from pretrained weights @epoch500
File "C:/Users/SONY/Desktop/img_cap/training.py", line 165, in train
saver = tf.train.Saver(max_to_keep=100)
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 1140, in init
self.build()
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 1172, in build
filename=self._filename)
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 688, in build
restore_sequentially, reshape)
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 407, in _AddRestoreOps
tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 247, in restore_op
[spec.tensor.dtype])[0])
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 663, in restore_v2
dtypes=dtypes, name=name)
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 767, in apply_op
op_def=op_def)
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1204, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

NotFoundError (see above for traceback): Key RNN/basic_lstm_cell/kernel not found in checkpoint
[[Node: save/RestoreV2_1 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_1/tensor_names, save/RestoreV2_1/shape_and_slices)]]

# please resolve this!

Do we have to use docker?

I have problems to use docker toolbox on windows which i can not sovle by network resource.
Acturlly, i have confused for several days, so is docker just a recommendation?
Thanks.

Running 3. on two images in a row is disallowed

Have to restart Kernel each time 3. is run or the following error generates:

ValueError: Variable RNN/basic_lstm_cell/weights already exists, disallowed. Did you mean to set reuse=True in VarScope? Originally defined at:

File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/rnn/python/ops/core_rnn_cell_impl.py", line 747, in _linear
"weights", [total_arg_size, output_size], dtype=dtype)
File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/rnn/python/ops/core_rnn_cell_impl.py", line 179, in call
concat = _linear([inputs, h], 4 * self._num_units, True, scope=scope)
File "", line 97, in build_generator
output, state = self.lstm(image_embedding, state)

The kernel appears to have died. It will restart automatically.

hey any idea why this happens when i start to run the 3. ?

got the same problem both on
Anaconda python 3.6 + windows 10 x64
and
Anaconda python 3.6 + mac 10.12
thanks !

Pretrained weights for oreilly-captions model?

Can someone release a pre-trained model for the oreilly-captions model trained on Flickr30k? Would be great for someone here who doesn't have the computational resource yet to do a full training run. Thanks!

ValueError: Variable RNN/basic_lstm_cell/weights already exists, disallowed. Did you mean to set reuse=True in VarScope?

if not os.path.exists('data/ixtoword.npy'):

print ('You must run 1. O\'reilly Training.ipynb first.')

else:

with open(vgg_path,'rb') as f:

    fileContent = f.read()

    graph_def = tf.GraphDef()

    graph_def.ParseFromString(fileContent)

​

images = tf.placeholder("float32", [1, 224, 224, 3])

tf.import_graph_def(graph_def, input_map={"images":images})

​

ixtoword = np.load('data/ixtoword.npy').tolist()

n_words = len(ixtoword)

maxlen=15

graph = tf.get_default_graph()

sess = tf.InteractiveSession(graph=graph)

caption_generator = Caption_Generator(dim_in, dim_hidden, dim_embed, batch_size, maxlen+2, n_words)

graph = tf.get_default_graph()

​

image, generated_words = caption_generator.build_generator(maxlen=maxlen)

---

ValueError Traceback (most recent call last)
in ()
18 graph = tf.get_default_graph()
19
---> 20 image, generated_words = caption_generator.build_generator(maxlen=maxlen)

in build_generator(self, maxlen, batchsize)
95 # in the first iteration we have no previous word, so we directly pass in the image embedding
96 # and set the previous_word to the embedding of the start token ([0]) for the future iterations
---> 97 output, state = self.lstm(image_embedding, state)
98 previous_word = tf.nn.embedding_lookup(self.word_embedding, [0]) + self.embedding_bias
99

/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/rnn/python/ops/core_rnn_cell_impl.py in call(self, inputs, state, scope)
177 else:
178 c, h = array_ops.split(value=state, num_or_size_splits=2, axis=1)
--> 179 concat = _linear([inputs, h], 4 * self._num_units, True, scope=scope)
180
181 # i = input_gate, j = new_input, f = forget_gate, o = output_gate

/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/rnn/python/ops/core_rnn_cell_impl.py in _linear(args, output_size, bias, bias_start, scope)
745 with vs.variable_scope(scope) as outer_scope:
746 weights = vs.get_variable(
--> 747 "weights", [total_arg_size, output_size], dtype=dtype)
748 if len(args) == 1:
749 res = math_ops.matmul(args[0], weights)

/usr/local/lib/python3.4/dist-packages/tensorflow/python/ops/variable_scope.py in get_variable(name, shape, dtype, initializer, regularizer, trainable, collections, caching_device, partitioner, validate_shape, custom_getter)
986 collections=collections, caching_device=caching_device,
987 partitioner=partitioner, validate_shape=validate_shape,
--> 988 custom_getter=custom_getter)
989 get_variable_or_local_docstring = (
990 """%s

/usr/local/lib/python3.4/dist-packages/tensorflow/python/ops/variable_scope.py in get_variable(self, var_store, name, shape, dtype, initializer, regularizer, trainable, collections, caching_device, partitioner, validate_shape, custom_getter)
888 collections=collections, caching_device=caching_device,
889 partitioner=partitioner, validate_shape=validate_shape,
--> 890 custom_getter=custom_getter)
891
892 def _get_partitioned_variable(self,

/usr/local/lib/python3.4/dist-packages/tensorflow/python/ops/variable_scope.py in get_variable(self, name, shape, dtype, initializer, regularizer, reuse, trainable, collections, caching_device, partitioner, validate_shape, custom_getter)
346 reuse=reuse, trainable=trainable, collections=collections,
347 caching_device=caching_device, partitioner=partitioner,
--> 348 validate_shape=validate_shape)
349
350 def _get_partitioned_variable(

/usr/local/lib/python3.4/dist-packages/tensorflow/python/ops/variable_scope.py in _true_getter(name, shape, dtype, initializer, regularizer, reuse, trainable, collections, caching_device, partitioner, validate_shape)
331 initializer=initializer, regularizer=regularizer, reuse=reuse,
332 trainable=trainable, collections=collections,
--> 333 caching_device=caching_device, validate_shape=validate_shape)
334
335 if custom_getter is not None:

/usr/local/lib/python3.4/dist-packages/tensorflow/python/ops/variable_scope.py in _get_single_variable(self, name, shape, dtype, initializer, regularizer, partition_info, reuse, trainable, collections, caching_device, validate_shape)
637 " Did you mean to set reuse=True in VarScope? "
638 "Originally defined at:\n\n%s" % (
--> 639 name, "".join(traceback.format_list(tb))))
640 found_var = self._vars[name]
641 if not shape.is_compatible_with(found_var.get_shape()):

ValueError: Variable RNN/basic_lstm_cell/weights already exists, disallowed. Did you mean to set reuse=True in VarScope? Originally defined at:

File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/rnn/python/ops/core_rnn_cell_impl.py", line 747, in _linear
"weights", [total_arg_size, output_size], dtype=dtype)
File "/usr/local/lib/python3.4/dist-packages/tensorflow/contrib/rnn/python/ops/core_rnn_cell_impl.py", line 179, in call
concat = _linear([inputs, h], 4 * self._num_units, True, scope=scope)
File "", line 97, in build_generator
output, state = self.lstm(image_embedding, state)

Any idea why I am getting this error and what I can do to fix it ? Thank you for your time

NotFoundError: Key RNN/basic_lstm_cell/bias not found

NotFoundError: Key RNN/basic_lstm_cell/bias not found in checkpoint
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

hi, could you help out with this issue please ?

please how to load the checkpoint on Jupyter?

How can i use the trained model? thanks a lot !

AttributeError: 'NoneType' object has no attribute 'astype'

I tried the new 3. O'Reilly Generate from image and there is a new error on the last line:
test(sess,image,generated_words,ixtoword, image_path)

---

AttributeError Traceback (most recent call last)
in ()
----> 1 test(sess,image,generated_words,ixtoword, image_path)

in test(sess, image, generated_words, ixtoword, test_image_path)
3
4
----> 5 feat = read_image(test_image_path)
6 fc7 = sess.run(graph.get_tensor_by_name("import/Relu_1:0"), feed_dict={images:feat})
7

in read_image(path)
1 def read_image(path):
2
----> 3 img = crop_image(path, target_height=224, target_width=224)
4 if img.shape[2] == 4:
5 img = img[:,:,:3]

in crop_image(x, target_height, target_width, as_float)
2 image = cv2.imread(x)
3 if as_float:
----> 4 image = image.astype(np.float32)
5
6 if len(image.shape) == 2:

AttributeError: 'NoneType' object has no attribute 'astype'

Getting just "a" as output for many input images of Flickr30k

@raulpuric
I've been working on image captioning for quite a while now, and also watched your youtube video for this code. I have taken your code as a base starting point. For convenience, I merged code from all 3 files(Training, Generate, Generate from Image) into one, and made some small changes to variable names for personal understanding. The core logic, however, is the same. Now, due to these graph changes, I can't simply restore your 500 epoch checkpoint, so I have trained it from scratch and reached a cost of ~1.8. But there's a serious problem I am encountering since a long time: For some images, I at least get some vague sentences, but for some, I am simply getting a single word "a" as output. This is driving me crazy and I really need some help. Below link has the complete code, not very different from yours:

https://github.com/aayushARM/imagecaptioning/blob/master/ImageCaption.py

For example, for below Image:

I get this as output:

Weird! This happens with many other images as well.

I am doing this as my academic project. I'd be glad if you can provide some inputs on this.

get_data() in O'Reilly Training.ipynb make kernel issues

I use docker to run Caption this, with TensorFlow. I followed everything in readme and when I run feats, captions = get_data(annotation_path, feature_path) Jupyter show alart that "The kernel appears to have died. It will restart automatically" everytime.

Do I miss something? How could I fix this issue?

Thank you!