๐ [Read our arXiv Paper] ย ๐ [Project Page]
Jianwei Yang*โ, Hao Zhang*, Feng Li*, Xueyan Zou*, Chunyuan Li, Jianfeng Gao
* Core Contributors ย ย ย ย โ Project Lead
We present Set-of-Mark (SoM) prompting, simply overlaying a number of spatial and speakable marks on the images, to unleash the visual grounding abilities in the strongest LMM. GPT-4V.
- [10/23] We released the SoM toolbox code for generating set-of-mark prompts for GPT-4V. Try it out!
- Release vision benchmarks used in our paper
Fascinating applications of SoM and LMMs:
- Set-of-Mark Prompting by Roboflow: Reimplementation of SoM by @SkalskiP from Roboflow
- Set-of-Mark Prompting for UI Navigation Agent: A really brilliant work using GPT-4V and SoM as a web copilot!
Our method compiles the following models to generate the set of marks:
- Mask DINO: State-of-the-art closed-set image segmentation model
- OpenSeeD: State-of-the-art open-vocabulary image segmentation model
- GroundingDINO: State-of-the-art open-vocabulary object detection model
- SEEM: Versatile, promptable, interactive and semantic-aware segmentation model
- Semantic-SAM: Segment and recognize anything at any granularity
- Segment Anything: Segment anything
We are standing on the shoulder of the giant GPT-4V (playground)!
- Install segmentation packages
# install SEEM
pip install git+https://github.com/UX-Decoder/Segment-Everything-Everywhere-All-At-Once.git@package
# install SAM
pip install git+https://github.com/facebookresearch/segment-anything.git
# install Semantic-SAM
pip install git+https://github.com/UX-Decoder/Semantic-SAM.git@package
- Download the pretrained models
sh download_ckpt.sh
- Run the demo
python demo_som.py
Potential solutions for some common issues:
Users can select which granularity of masks to generate, and which mode to use between automatic (top) and interactive (bottom). A higher alpha blending value (0.4) is used for better visualization.
SoM enables interleaved prompts which include textual and visual content. The visual content can be represented using the region indices.
![Screenshot 2023-10-18 at 10 12 18](https://private-user-images.githubusercontent.com/34880758/276056435-f5e0c0b0-58de-4b60-bf01-4906dbcb229e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQxMTA5ODQsIm5iZiI6MTcyNDExMDY4NCwicGF0aCI6Ii8zNDg4MDc1OC8yNzYwNTY0MzUtZjVlMGMwYjAtNThkZS00YjYwLWJmMDEtNDkwNmRiY2IyMjllLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDIzMzgwNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTNhNjNkYWVmZjdjYmUyMDcxNWY4NDJhZGY0OTM0ZmZlMzRkMTdhODdmNWI3Y2RhODlmZDVlZDJjMTk3ZTM4OGYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.iq_a_HbwUxXc8uysfnKmebnovU6RlnI7wcXjvmuX498)
![Screenshot 2023-10-18 at 10 10 41](https://private-user-images.githubusercontent.com/34880758/276055888-033cd16c-876c-4c03-961e-590a4189bc9e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQxMTA5ODQsIm5iZiI6MTcyNDExMDY4NCwicGF0aCI6Ii8zNDg4MDc1OC8yNzYwNTU4ODgtMDMzY2QxNmMtODc2Yy00YzAzLTk2MWUtNTkwYTQxODliYzllLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDIzMzgwNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTRhMmQ5MWMwODZhYzNjMzJkOThjN2Y0OTA4OWIxMmNlMzE3NzAxMGJmN2E0ZDViYWY5YzJkZTQ3YTYxMGZmZDYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.uEayFXDqiVUWnU7TWr-b0vP9RV5-sZoqQuO0dI49juE)
In comparison to GPT-4V without SoM, adding marks enables GPT-4V to ground the reasoning on detailed contents of the image (Left). Clear object cross-image references are observed on the right. 17
![Screenshot 2023-10-18 at 10 18 03](https://private-user-images.githubusercontent.com/34880758/276057292-8b112126-d164-47d7-b18c-b4b51b903d57.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQxMTA5ODQsIm5iZiI6MTcyNDExMDY4NCwicGF0aCI6Ii8zNDg4MDc1OC8yNzYwNTcyOTItOGIxMTIxMjYtZDE2NC00N2Q3LWIxOGMtYjRiNTFiOTAzZDU3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDIzMzgwNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTg4MDI0ZGRlNGYxNTllN2FjMGY3OWU1MTE3OGQ1NDhkY2RmMTk3YWI4Y2VjZTExMGRkYjcxZDcxMTYwMjIzMzgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.6XCyQ61zh4aYodHObbFz14skthG6jednMENnPTuXPyA)
Case study on solving CAPTCHA. GPT-4V gives the wrong answer with a wrong number of squares while finding the correct squares with corresponding marks after SoM prompting.
![Screenshot 2023-10-18 at 10 18 44](https://private-user-images.githubusercontent.com/34880758/276057392-dc753c3f-ada8-47a4-83f1-1576bcfb146a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQxMTA5ODQsIm5iZiI6MTcyNDExMDY4NCwicGF0aCI6Ii8zNDg4MDc1OC8yNzYwNTczOTItZGM3NTNjM2YtYWRhOC00N2E0LTgzZjEtMTU3NmJjZmIxNDZhLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDIzMzgwNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWEyYjcyOTI4NmEzYmNlYWEyNWRlYmU2ZGVlMWZmMmEwNTA2NzAyOTY0YTIyZTA0OTBiYTZkZjRiMmFiZWRhYTUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.HH2oaRAA5x0OyLEZxQiG3BZNIfiqf-j6e4o_dkx9jHA)
Case study on an image of dish for GPT-4V. GPT-4V does not produce a grounded answer with the original image. Based on SoM prompting, GPT-4V not only speaks out the ingredients but also corresponds them to the regions.
![Screenshot 2023-10-18 at 10 19 12](https://private-user-images.githubusercontent.com/34880758/276057456-88188c90-84f2-49c6-812e-44770b0c2ca5.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQxMTA5ODQsIm5iZiI6MTcyNDExMDY4NCwicGF0aCI6Ii8zNDg4MDc1OC8yNzYwNTc0NTYtODgxODhjOTAtODRmMi00OWM2LTgxMmUtNDQ3NzBiMGMyY2E1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDIzMzgwNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWM1Njc1NzIxOThmNmNlOTllOWM1NzM1N2UxOTBiZmIwZDFlMjAxODFmOWQ4N2RkN2UxYjUyOTQwM2UzZDE0ZjgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.GQuwEfzdGrC3BkD7jL9xbifYMyEBnEFRg4vbVZ5XHCo)
SoM-pormpted GPT-4V gives very precise suggestions while the original one fails, even with hallucinated foods, e.g., soft drinks
![Screenshot 2023-10-18 at 10 19 39](https://private-user-images.githubusercontent.com/34880758/276057520-9b35b143-96af-41bd-ad83-9c1f1e0f322f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQxMTA5ODQsIm5iZiI6MTcyNDExMDY4NCwicGF0aCI6Ii8zNDg4MDc1OC8yNzYwNTc1MjAtOWIzNWIxNDMtOTZhZi00MWJkLWFkODMtOWMxZjFlMGYzMjJmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDIzMzgwNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTNmNGZiOTJlYzAyYTNjMzI3M2ZiMWExMjZiM2Y5YTdlMTk0MDAyNjkyMjllZGM0ZDJhMTc5NTc0Mjc0Y2Q3MDImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.JBd4yh9MPOPWMIIek51jJ5HVMDueGs_zqlyB7NC2vG0)
![Screenshot 2023-10-18 at 10 20 03](https://private-user-images.githubusercontent.com/34880758/276057600-0bc86109-5512-4dee-aac9-bab0ef96ed4c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQxMTA5ODQsIm5iZiI6MTcyNDExMDY4NCwicGF0aCI6Ii8zNDg4MDc1OC8yNzYwNTc2MDAtMGJjODYxMDktNTUxMi00ZGVlLWFhYzktYmFiMGVmOTZlZDRjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDIzMzgwNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWEyZDRlZWI1NWQ0NTBhYmY1ZTViYmRlZWE1NzFiMzg4Mjk5OTJkMzdmOTkzZWIyMDJkZGQ0Y2U2MzczZGZlNjQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.K8-xBs7RnSpoHlWPhlOOCseuL_Z59IAfbSoff6UkjI4)
GPT-4V with SoM gives a reasonable suggestion on how to achieve a goal in a gaming scenario.
![Screenshot 2023-10-18 at 10 21 24](https://private-user-images.githubusercontent.com/34880758/276057806-7f139250-5350-4790-a35c-444ec2ec883b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQxMTA5ODQsIm5iZiI6MTcyNDExMDY4NCwicGF0aCI6Ii8zNDg4MDc1OC8yNzYwNTc4MDYtN2YxMzkyNTAtNTM1MC00NzkwLWEzNWMtNDQ0ZWMyZWM4ODNiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE5VDIzMzgwNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTczYjU4ZDYzZmRmMGZjNDcwZjQyZGFlNGNjODFhMjNlZTM0M2EzMjVkM2YyYzMyMzFmN2ViNWJmNWY2MjU2NzAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.5QiDel3JulMRmMfEFIVTqJtoq0OTVObIwPFDD6pSvgE)
We conduct experiments on various vision tasks to verify the effectiveness of our SoM. Results show that GPT4V+SoM outperforms specialists on most vision tasks and is comparable to MaskDINO on COCO panoptic segmentation.
If you find our work helpful for your research, please consider citing the following BibTeX entry.
@article{yang2023setofmark,
title={Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V},
author={Jianwei Yang and Hao Zhang and Feng Li and Xueyan Zou and Chunyuan Li and Jianfeng Gao},
journal={arXiv preprint arXiv:2310.11441},
year={2023},
}