x-plug / mplug Goto Github PK

Hi. I don't understand how to use this pre-trained model for image captioning. Am I supposed to clone the github repo and then somehow load the pre-trained model? It would be extremely beneficial if you could provide me with some sort of notebook on how to achieve this

The code in this repo shows that baseline reward is calculated by averaging reward of generated captions. However, the original version of scst as well as some other scst implementation (e.g., in VALOR) calculate the baseline reward with greedy-search-generated caption. Is there any reference or explanation about current implementation in this repo? Really appreciate it if I obtain any help.

and how to find it? thanks

Dear authors,

I finetuned mPLUG Base on VQAv2 but only get around 75% accuracy instead of the around 80% reported in the readme.

Could you kindly upload the finetuned checkpoints for VQA? I am benchmarking your model and would prefer to benchmark the strongest possible version.

Best,

Hi, I encounter a problem. I think you did not release the ViT-L-14 or ViT-B-16 models.
Can you slove these problem?
thank you

请问在modelsope中公布的模型是论文中SOTA的模型吗，我这边使用模型介绍中的模型和caption输出方法，在coco-val 5k上统计bleu4和cider指标，都是低于论文指标的。
是模型的原因、测试方法不对或者是测试集测试工具没对齐吗？

你好，请问pretrain什么时候开放源码，看到readme里写的 coming soon，想在自己的数据上试一下效果。感谢

The links of pre-trained models are invalid. Could you provide a new one?

Hi authors,

Thanks for your great work. Are there any chance for you to release the fintuned COCO/Flikr30k checkpoint for image-retrieval task?

Thanks a lot.

Hi, thanks for your work!

I used your model with modelscope, but I didn't find which model size is the default setting in modelscope, I only know is its named mplug_visual-question-answering_coco_large_en.

And I found that there is memory leak with following code.

pipeline_vqa = pipeline(Tasks.visual_question_answering, model=model_id)
for image_path in image_paths:
    count = count + 1
    raw_image = Image.open(image_path).convert('RGB')
    image_list.append(raw_image)
    image_name_list.append(os.path.basename(image_path))
    question = "what is the man doing in the picture?"
    input_vqa = {
        'image': image_path,
        'question': question,
    }

    text = pipeline_vqa(input_vqa)  # {'text': 'talking on phone'}
    text = text['text']
    result_list.append(text)