lizhaoliu-lec / cg-vlm Goto Github PK

This is a brillint work! I want to follow up this work, so I am wondering following questions:

1. That authors claimed that only 1%~10% data is used. There is two stages in the whole training process: pre-training(558K instruction following data in LLaVA-1.5) and visual instruction tuning(665K instruction following data). Is reduction of data usage only in visual instruction tuning or both stages? How many pre-training data is used?
2. In which stage the contrastive loss applies? That authors metioned "vision-language alignment stage". However, I don't understand which stage it actually means (pre-training or Visual IT)

Hi, I am wondering how the gradient of the generative loss propagates to the visual adapter if the LLM is frozen, during the pre-training stage.