GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest π₯ Demo π₯
Single-Region Understanding
Multiple-Region Understanding
GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest
Shilong Zhang*, Peize Sun*, Shoufa Chen*, Min Xiao, Wenqi Shao ,Wenwei Zhang, Kai Chen, Ping Luo
(*Equal Contribution)
[Demo] [Paper] [δΈζδ»η»]
- [July 7] All training and inference code has been released, you can try demo here π₯π₯π₯
- Clone the
GPT4RoI
git clone https://github.com/jshilong/gpt4roi.git
cd gpt4roi
- Create the env
conda create -n gpt4roi python=3.10 -y
conda activate gpt4roi
pip install --upgrade pip # enable PEP 660 support
pip install setuptools_scm
pip install --no-cache-dir -e .
# please use conda re-install the torch, pip may loss some runtime lib
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
- Install the
flash-attn
package
pip install ninja
pip install flash-attn --no-build-isolation
- install the
mmcv-1.4.7
package Make sure that yournvcc -V
is consistent with cudatookit version ofpython -c "import torch;print(torch.version.cuda)
.
cd mmcv-1.4.7
MMCV_WITH_OPS=1 pip install -e .
Our dataset includes RefCOCO, RefCOCO+, RefCOCOg, Visual Genome, Flickr30K entities, and the VCR dataset. We are sincerely grateful to the creators of these datasets, especially for the VCR dataset, for their forward-thinking in creating these dataset.
The dataset section of this repository may appear somewhat messy, especially the VCR part(still finishing), which may cause GPT4RoI not be very user-friendly. We are currently working on formulating the datasets into a unified format and will be accompanying them with stronger models. Please stay tuned for updates.
You can download the corresponding dataset from the official website and organize it as follows. Afterwards, you can modify the gpt4roi/configs/dataset_config.json
file to select the specific dataset you want to use:
GPT4RoI
βββ data
β βββ coco_det
β β βββ annotations
β β β βββinstances_train2017.json
β β βββ train2017/
β βββ mdetr_annotations
β β βββfinetune_refcoco_train.json
β β βββfinetune_refcoco+_train.json
β β βββfinetune_refcocog_train.json
β β βββfinal_flickr_mergedGT_train.json
β βββ coco_imgs/
β βββ flickr30k-images/
β βββ visual_genome
β β βββtrain.json
β β βββvg_all/
β βββ llava
β β βββ llava_instruct_150k.json
β β βββ llava_150k_bbox_pred_results.pkl
β βββ vcr
β β βββ train.jsonl
β β βββ vcr1images/
- coco_imgs should contains all coco image(you can soft link them to this directory.
- We use Visual_Genome_Dataset_V1.2, you should soft all vg images to
vg_all
. llava_150k_bbox_pred_results.pkl
contains the detection predicted results with EVA-02-DET. We appreciate their work.
coming soon.
We release coming soon. weights as delta weights to comply with the LLaMA model license. You can add our delta to the original LLaMA weights to obtain the LLaVA weights.
Instructions:
- Get the original LLaMA weights in the huggingface format by following the instructions here.
- Use the following scripts to get LLaVA weights by applying our delta coming soon.
This conversion command needs around 30 GB of CPU RAM.
python3 -m llava.model.apply_delta \
--base /path/to/llama-7b \
--target /output/path/GPT4RoI-7B-v0 \
--delta jshilong/GPT4RoI-7B-v0
GPT4RoI is trained on 8 A100 with the following code.
You should modify the gpt4roi/configs/dataset_config.json
to make sure you only use the dataset of stage1.
bash train_stage1.sh
You should modify the gpt4roi/configs/dataset_config.json
to make sure you only use the dataset of stage2.
bash train_stage2.sh
Please install Gradio Box first.
python gpt4roi/app.py
prompt format in GPT4RoI
You should always use<region1>, <region2>...
to refer the new bounding box in the image when you first draw them. Then you can use normalregion 1
in the conversation to refer the instance.- You should always click the
clear all
buttul and waiting the clear process finished before you start a new conversation.
Multiple Rounds of Dialogue
- LLaVA: The codebase we built upon.
- Vicuna: The LLM we used.
- VCR: We get strong region reasoning ability from this forward thinking dataset.
If you find GPT4RoI useful for your your research and applications, please cite using this BibTeX:
@misc{zhang2023gpt4roi,
title={GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest},
author={Shilong Zhang and Peize Sun and Shoufa Chen and Min Xiao and Wenqi Shao and Wenwei Zhang and Kai Chen and Ping Luo},
year={2023},
eprint={2307.03601},
archivePrefix={arXiv},
primaryClass={cs.CV}
}