This repository contains the code and data for the paper titled "GeoGPT4V: Towards Geometric Multi-modal Large Language Models with Geometric Image Generation"
-
Clone this repository.
-
Install Package.
conda create -n geogpt4v python=3.10 -y conda activate geogpt4v pip install -r requirements.txt
-
Install Wolfarm engine. Please follow its offical tutorial.
-
Download following open-source datasets or use your own datasets:
-
Transform the dataset into the following format and save it as a jsonline file:
# multi-choice quetion example {"id": 1, "question": "For the pair of similar figures, find the area of the green figure.", "choices": ["20.4", "28.6", "56.0", "78.4"], "answer": "D", "image": "image path"} # open-ending quetsion example {"id": 2, "question": "Prove that △ABC is congruent to △DEF.", "answer": "Because AB = DE, BC = EF, and ∠ABC is equal to ∠DEF, △ABC is congruent to △DEF.", "image": "image path"}
-
[optional] If you are using your own dataset, please modify the function 'construct_prompt' in './pipeline/gen_instruction_mp.py'.
Run the scripts in the following order, remember to modify the dataset path and API key in the scripts.
- sh scripts/gen_instruction_gpt4v_mp.sh
- sh scripts/gen_image_mp.sh
- sh scripts/rerank_gp4v_mp.sh
- sh scripts/filter.sh
You can use following models' offical training code or use your own code. You can run scripts/convert_format.sh to transform the GeoGPT4V dataset into the required format for model training.
The GeoGPT4V dataset will be available soon.
You can find the data example in data/geogpt4v_data_example.jsonl.
The checkpoints will be available soon.