I'd like to use this code in my project, however I'm getting an error when running eval_model.py:

ValueError: Input 0 of layer dense_1 is incompatible with the layer: expected axis -1 of input shape to have value 4096 but received input with shape [None, 1000]

I'm thinking that the structure of the network might have changed since 2018, which is when the pretrained weights are from. Could you possibly upload the latest version of the weights if you still have them lying around?

Thanks a bunch!

I am studying Image Caption and very interested in your project. Do you have a model that incorporates attention mechanisms? Thank you very much！

Is it possible for you to share the h5 file so that we can try it in Python?

When I am running command python prepare_data.py getting below error

Using TensorFlow backend.
Traceback (most recent call last):
File "prepare_data.py", line 132, in
features = extract_features(directory)
File "prepare_data.py", line 38, in extract_features
model = Model(inputs=model.inputs, outputs=model.layers[-1].output)
File "C:\Users\XX\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "C:\Users\XX\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\engine\network.py", line 91, in init
self._init_graph_network(*args, **kwargs)
File "C:\Users\XX\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\engine\network.py", line 251, in _init_graph_network
input_shapes=[x._keras_shape for x in self.inputs],
File "C:\Users\XX\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\engine\network.py", line 251, in
input_shapes=[x._keras_shape for x in self.inputs],
AttributeError: 'Tensor' object has no attribute '_keras_shape'

vocab_size = len(tokenizer.word_index) + 1
请问这个地方为什么要+1呢？

After excuting: prepare_data.py and train_model.py while I was trying to run eval_model.py on few images to find caption, I received the below Error:

Traceback (most recent call last):
  File "eval_model.py", line 127, in <module>
    model = load_model(filename)
  File "./.env/lib/python3.8/site-packages/tensorflow/python/keras/saving/save.py", line 206, in load_model
    return hdf5_format.load_model_from_hdf5(filepath, custom_objects,
  File "./.env/lib/python3.8/site-packages/tensorflow/python/keras/saving/hdf5_format.py", line 182, in load_model_from_hdf5
    model_config = json_utils.decode(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'

After model creation, when the first epoch starts, I get this error:

tensorflow.python.framework.errors_impl.InvalidArgumentError: Matrix size-incompatible: In[0]: [47,1000], In[1]: [4096,256]
[[node model/dense/Relu

Can you help me with this please? Seems like some size issue of tensor.

Error Traceback:

Layer (type) Output Shape Param # Connected to

input_1 (InputLayer) [(None, 4096)] 0 []

dropout (Dropout) (None, 4096) 0 ['input_1[0][0]']

dense (Dense) (None, 256) 1048832 ['dropout[0][0]']

input_2 (InputLayer) [(None, 34)] 0 []

repeat_vector (RepeatVector) (None, 34, 256) 0 ['dense[0][0]']

embedding (Embedding) (None, 34, 256) 1940224 ['input_2[0][0]']

concatenate (Concatenate) (None, 34, 512) 0 ['repeat_vector[0][0]',
'embedding[0][0]']

lstm (LSTM) (None, 500) 2026000 ['concatenate[0][0]']

dense_1 (Dense) (None, 7579) 3797079 ['lstm[0][0]']

==================================================================================================
Total params: 8,812,135
Trainable params: 8,812,135
Non-trainable params: 0

None
train_model.py:48: UserWarning: Model.fit_generator is deprecated and will be removed in a future version. Please use Model.fit, which supports generators.
callbacks=[checkpoint], validation_data=val_generator, validation_steps=val_steps)
Epoch 1/20
Traceback (most recent call last):
File "train_model.py", line 58, in
train_model(epochs=20)
File "train_model.py", line 48, in train_model
callbacks=[checkpoint], validation_data=val_generator, validation_steps=val_steps)
File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\engine\training.py", line 2030, in fit_generator
initial_epoch=initial_epoch)
File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\utils\traceback_utils.py", line 67, in error_handler
raise e.with_traceback(filtered_tb) from None
File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Matrix size-incompatible: In[0]: [47,1000], In[1]: [4096,256]
[[node model/dense/Relu
(defined at C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\backend.py:4867)
]] [Op:__inference_train_function_7367]

Errors may have originated from an input operation.
Input Source operations connected to node model/dense/Relu:
In[0] model/dense/BiasAdd (defined at C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\layers\core\dense.py:210)

Operation defined at: (most recent call last)

File "train_model.py", line 58, in
train_model(epochs=20)

File "train_model.py", line 48, in train_model
callbacks=[checkpoint], validation_data=val_generator, validation_steps=val_steps)

File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\engine\training.py", line 2030, in fit_generator
initial_epoch=initial_epoch)

File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)

File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\engine\training.py", line 1216, in fit
tmp_logs = self.train_function(iterator)

File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\engine\training.py", line 878, in train_function
return step_function(self, iterator)

File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\engine\training.py", line 867, in step_function
outputs = model.distribute_strategy.run(run_step, args=(data,))

File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\engine\training.py", line 860, in run_step
outputs = model.train_step(data)

File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\engine\training.py", line 808, in train_step
y_pred = self(x, training=True)

File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)

File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\engine\base_layer.py", line 1083, in call
outputs = call_fn(inputs, *args, **kwargs)

File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler
return fn(*args, **kwargs)

File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\engine\functional.py", line 452, in call
inputs, training=training, mask=mask)

File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\engine\functional.py", line 589, in _run_internal_graph
outputs = node.layer(*args, **kwargs)

File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)

File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\engine\base_layer.py", line 1083, in call
outputs = call_fn(inputs, *args, **kwargs)

File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler
return fn(*args, **kwargs)

File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\layers\core\dense.py", line 213, in call
outputs = self.activation(outputs)

File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\activations.py", line 311, in relu
return backend.relu(x, alpha=alpha, max_value=max_value, threshold=threshold)

File "C:\Users\kingo\miniconda3\envs\icg\lib\site-packages\keras\backend.py", line 4867, in relu
x = tf.nn.relu(x)

2021-11-27 03:22:33.545141: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: FAILED_PRECONDITION: Python interpreter state is not initialized. The process may be terminated.
[[{{node PyFunc}}]]

FileNotFoundError: [Errno 2] No such file or directory: 'models/tokenizer.pkl'

Any idea why test samples are not read?

Where can I get those .pkl files ???? Thank you

Do you have Documentation of this project?

Can you provide me the documentation of this project please?

Received the below error while running eval_model.py

raise ValueError('Input ' + str(input_index) +

    ValueError: Input 0 is incompatible with layer model_1: expected shape=(None, 4096), found shape=(None, 1000)

The error is on:

File "eval_model.py", line 134, in <module>
    captions = generate_desc(model, tokenizer, photo, index_word, max_length)
  File "eval_model.py", line 56, in generate_desc
    y_pred = model.predict([photo,sequence], verbose=0)[0]

May I ask how much loss and val_loss have you got? I have got a val_loss of 3.736 which I thought is not good enough.