tidedra / vl-rlhf

A RLHF Infrastructure for Vision-Language Models

• 👋 Hi, I’m @TideDra
• 👀 I’m interested in Artificial Intelligence.
• 📫 Reach me by sending email to: [email protected]

vl-rlhf's Issues

不使用lora报错

不使用lora会报错
Exception has occurred: FileNotFoundError (note: full exception trace is shown but execution is paused at: )
[Errno 2] No such file or directory: '/cache/zy/VL-RLHF/src/vlrlhf/models/InternLMXC2/..utils.common.py'

微调internXC2报错

File "/home/ma-user/anaconda3/envs/dpo/lib/python3.10/site-packages/transformers/configuration_utils.py", line 264, in getattribute
return super().getattribute(key)
AttributeError: 'InternLMXcomposer2Config' object has no attribute 'image_token_index'

Reproduction of InternLM-XComposer2

Hi,

Thank you for sharing a great work.
I'm trying to reproduce the performance of InternLM-XComposer2 + DPO + VLFeedback, but I found that the baseline performance (InternLM-Xcomposer2-VL-7b) you reported is slightly different from the performance in the original paper.
Can I know why?
Also, the dpo_internlmxc2vl7b.sh file in ./scripts folder is the command for reproducing your InternLM-Xcomposer2-VL-7b-DPO model?
If not, could you share the script file or config file to reproduce InternLM-Xcomposer2-VL-7b-DPO model.
Thank you again for the nice work.

[BUG] Value Error noqa:E501

When I using dpo_llava finetune on custom dpo dataset, after several steps, I run into the following error message

ValueError: The input provided to the model are wrong. The number of image tokens is 3 while the number of image given to the model is 4. This prevents correct indexing and breaks batch generation.

How does that happen since I have checked my dataset in strictly following format

Any suggestion on how to modify code to train single textual modality

Nice code base.

Any suggestion on how to modify code to train single textual modality

微调qwen爆内存

您好，使用原始代码在2张A100 80G上面微调qwen，显存占用两张卡上都只有919M，但是在数据加载过程中？内存占用一直在增加，直到180多G后内存爆了，程序终止。请问这个问题怎么解？
训练log：

内存占用：

Support for InstructBlipPPOTrainer

Hey Guys!

First of all, thank you for this amazing repo!

Do you know if there will be support for InstructBlipPPOTrainer?

微调LLaVA报错

[rank1]: Traceback (most recent call last):
[rank1]:   File "/home/nfs04/chengkz/VL-RLHF/src/vlrlhf/dpo.py", line 146, in <module>
[rank1]:     dpo_trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
[rank1]:   File "/home/data_91_d/anaconda3/envs/chengkz_lvlm/lib/python3.10/site-packages/transformers/trainer.py", line 1885, in train
[rank1]:     return inner_training_loop(
[rank1]:   File "/home/data_91_d/anaconda3/envs/chengkz_lvlm/lib/python3.10/site-packages/transformers/trainer.py", line 2216, in _inner_training_loop
[rank1]:     tr_loss_step = self.training_step(model, inputs)
[rank1]:   File "/home/nfs04/chengkz/VL-RLHF/src/vlrlhf/base/trainer.py", line 305, in training_step
[rank1]:     loss_step = super().training_step(model, inputs)
[rank1]:   File "/home/data_91_d/anaconda3/envs/chengkz_lvlm/lib/python3.10/site-packages/transformers/trainer.py", line 3238, in training_step
[rank1]:     loss = self.compute_loss(model, inputs)
[rank1]:   File "/home/data_91_d/anaconda3/envs/chengkz_lvlm/lib/python3.10/site-packages/trl/trainer/dpo_trainer.py", line 1081, in compute_loss
[rank1]:     loss, metrics = self.get_batch_loss_metrics(model, inputs, train_eval="train")
[rank1]:   File "/home/data_91_d/anaconda3/envs/chengkz_lvlm/lib/python3.10/site-packages/trl/trainer/dpo_trainer.py", line 1022, in get_batch_loss_metrics
[rank1]:     ) = self.concatenated_forward(model, batch)
[rank1]:   File "/home/nfs04/chengkz/VL-RLHF/src/vlrlhf/models/Llava/__init__.py", line 502, in concatenated_forward
[rank1]:     pixel_values=concatenated_batch["pixel_values"],
[rank1]: KeyError: 'pixel_values'

请问这个报错该如何解决？

支持cogvlm2模型的强化学习训练吗

请问支持对internvl-1-5的微调吗？如果可以的话显存应该预留多少

strip may break chat template

in process_batch, raw_text.strip() can drop special token like \n or , which can be a part of chat template.

Support for SFT InternLM-XComposer2?

Does this repo support for SFT InternLM-XComposer2 ?

There is no sft_InternLM_XC.sh in the scripts. Therefore I want to check if the repo support for SFT InternLM-XComposer2.

Thank you!