๐ผ๏ธ Attend to You: Personalized Image Captioning with Context Sequence Memory Networks. In CVPR, 2017. Expanded : Towards Personalized Image Captioning via Multimodal Memory Networks. In IEEE TPAMI, 2018.
License: MIT License
It's default mode is caption generation.
I want to know how to run hashtag prediction
Thanks for the greatwork! It is creative and the shown results are promising.
I saw from paper you have several baselines to be compared against your proposed CSMN (e.g. 1-nearest neighbor to user contents, RNN seq2seq with active vocabulary)
Would you release the implementations of those baselines?
Besides that, given the recent advancement on NLP (transformer, GPT-2 ... etc) , would you (and how would you) propose your CSMN differently under modern context (as of 2020)?
In your paper๏ผIโve read the CSMN model.I have trouble with the understanding of superscript a and c๏ผwould you mind telling me what the porpose of add superscript a and c to the image memory vector and the user context memory vector?Thanks a lot!
Do we need to train for 5 lac. steps as specified in code ? or for just 20 epochs as per paper ?
Also, earlier I downloaded InstaPic data set, which only had 1.03M samples instead of 1.1M. why ?
Hoping to get a reply soon. Thanks :)
i keep getting errors saying that unexpected keyword arguments for is_training in line 111
Can anyone help me to resolve it?
ํน์ ๋๋ฆฌ์ จ๋ ํ๊ฒฝ์ ์ข ๊ตฌํ ์ ์์๊น์ ??
Hi @cesc-park
If I am correct, the only difference between CVPR and TPAMI papers is use of data sets. Both use the same CSMN architecture.
Can you or someone validate this?
When will the InstaPIC-1.1M Dataset be made available again?
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/lalit/notebooks/Lalit/image_caption/attend2u/train.py", line 211, in
tf.app.run()
File "/home/lalit/.local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/home/lalit/notebooks/Lalit/image_caption/attend2u/train.py", line 208, in main
train()
File "/home/lalit/notebooks/Lalit/image_caption/attend2u/train.py", line 134, in train
loss = _tower_loss(inputs, scope)
File "/home/lalit/notebooks/Lalit/image_caption/attend2u/train.py", line 33, in _tower_loss
net = CSMN(inputs, ModelConfig(FLAGS))
File "utils/configuration.py", line 33, in init
super(ModelConfig, self).init(FLAGS)
File "utils/configuration.py", line 12, in init
attrs = FLAGS.dict['__flags']
KeyError: '__flags'
In papar's EXperiments part, said "we randomly split the dataset into 90% for training, 5k posts for test and the rest for validation" but I read the code and found the dataset is parted into train.txt test1.txt and test2.txt, but the text2.txt is not used in code, Did I miss something?
looking forward to your reply.
ํน์ Pretrained model ์์ผ๋ฉด ํ๋ฒ ํ์ธ์ด ๊ฐ๋ฅํ ์ง์??
๊ฐ์ธ์ ์ผ๋ก ๋๋ ค๋ณด๋ ์๊ฐ๋ณด๋ค.. GPU ์ฑ๋ฅ์ด ๋ถ์กฑํ๋ค์.
๋ถํ๋๋ฆฝ๋๋ค.
The YFCC100M data set cannot be downloaded via Google Cloud Disk. Can you provide other download methods? And I hope you can provide the InstaPIC-1.1M Dataset again.
Hey, I'm getting error while training the data after extraction, Can you help me with it?
I cloned the repository and followed the instructions as given by you guys on this github repo, but there are too many issues in the code. First, even though I installed the tensorflow as per the version given in requirements.txt but there is no module named preprocessing in slim. Anyways I managed that but then I ran into the following errrors:
Traceback (most recent call last):
File "cnn_feature_extractor.py", line 148, in <module>
tf.app.run()
File "/home/pdguest/Lalit/Selfie/HashtagPred/hashtag/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "cnn_feature_extractor.py", line 112, in main
processed_images, num_classes=1000, is_training=False
TypeError: resnet_v1_101() got an unexpected keyword argument 'is_training'
Traceback (most recent call last):
File "cnn_feature_extractor.py", line 148, in <module>
tf.app.run()
File "/home/pdguest/Lalit/Selfie/HashtagPred/hashtag/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "cnn_feature_extractor.py", line 112, in main
processed_images, num_classes=1000, is_training=False
TypeError: resnet_v1_101() got an unexpected keyword argument 'is_training'
Traceback (most recent call last):
File "cnn_feature_extractor.py", line 148, in <module>
tf.app.run()
File "/home/pdguest/Lalit/Selfie/HashtagPred/hashtag/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "cnn_feature_extractor.py", line 112, in main
processed_images, num_classes=1000, is_training=False
TypeError: resnet_v1_101() got an unexpected keyword argument 'is_training'
Traceback (most recent call last):
File "cnn_feature_extractor.py", line 148, in <module>
tf.app.run()
File "/home/pdguest/Lalit/Selfie/HashtagPred/hashtag/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "cnn_feature_extractor.py", line 112, in main
processed_images, num_classes=1000, is_training=False
TypeError: resnet_v1_101() got an unexpected keyword argument 'is_training'
Traceback (most recent call last):
File "cnn_feature_extractor.py", line 148, in <module>
tf.app.run()
File "/home/pdguest/Lalit/Selfie/HashtagPred/hashtag/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "cnn_feature_extractor.py", line 112, in main
processed_images, num_classes=1000, is_training=False
TypeError: resnet_v1_101() got an unexpected keyword argument 'is_training'
Traceback (most recent call last):
File "cnn_feature_extractor.py", line 148, in <module>
tf.app.run()
File "/home/pdguest/Lalit/Selfie/HashtagPred/hashtag/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "cnn_feature_extractor.py", line 112, in main
processed_images, num_classes=1000, is_training=False
TypeError: resnet_v1_101() got an unexpected keyword argument 'is_training'Can you please check your code once again and update it ?
It seems that for one image to be predicted, the context information is from all tags of one user, not from the previous tags before the image is uploaded. Did I miss something?
Model with LSTM is too difficult to reach the convergence, can you provide your pretrained model? Thanks a lot.
When I ran the train.py file using the options as num_gpus 1 and batch_size 10, it throws the following error :
[ERROR:2017-05-08 12:42:08,943] Exception in QueueRunner: 0-th value returned by pyfunc_0 is double, but expects float
[[Node: PyFunc = PyFunc[Tin=[DT_STRING], Tout=[DT_FLOAT], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](DecodeCSV)]]
Caused by op u'PyFunc', defined at:
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/HashtagPred/train.py", line 211, in <module>
tf.app.run()
File "/home/HashtagPred/hashtag/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/home/HashtagPred/train.py", line 208, in main
train()
File "/home/HashtagPred/train.py", line 98, in train
tower_caption_mask = enqueue(False)
File "utils/data_utils.py", line 240, in enqueue
answer_id, context_mask, caption_mask = read_numpy_format_and_label( filename_queue)
File "utils/data_utils.py", line 160, in read_numpy_format_and_label
tf.float32
File "/home/HashtagPred/hashtag/local/lib/python2.7/site-packages/tensorflow/python/ops/script_ops.py", line 189, in py_func
input=inp, token=token, Tout=Tout, name=name)
File "/home/HashtagPred/hashtag/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_script_ops.py", line 40, in _py_func
name=name)
File "/home/HashtagPred/hashtag/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
op_def=op_def)
File "/home/HashtagPred/hashtag/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2336, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/HashtagPred/hashtag/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1228, in __init__
self._traceback = _extract_stack()
InvalidArgumentError (see above for traceback): 0-th value returned by pyfunc_0 is double, but expects float
[[Node: PyFunc = PyFunc[Tin=[DT_STRING], Tout=[DT_FLOAT], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](DecodeCSV)]]
Exception in thread Thread-6:
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google โค๏ธ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.