autodistill / autodistill-gpt-4v

GPT-4V(ision) module for use with Autodistill.

Autodistill GPT-4V Module

This repository contains the code supporting the GPT-4V base model for use with Autodistill.

GPT-4V, developed by OpenAI, is a multi-modal language model. With GPT-4V, you can ask questions about images in natural language. The autodistill-gpt4v module enables you to classify images using GPT-4V.

This model uses the gpt-4-vision-preview API announced by OpenAI on November 6th, 2023.

Note

Using this project will incur billing charges for API calls to the OpenAI GPT-4 Vision API. Refer to the OpenAI pricing page for more information and to calculate your expected pricing. This package makes one API call per image you want to label.

Read the full Autodistill documentation.

Read the GPT-4V Autodistill documentation.

Installation

To use GPT-4V with autodistill, you need to install the following dependency:

pip3 install autodistill-gpt-4v

Quickstart

from autodistill_gpt_4v import GPT4V

# define an ontology to map class names to our GPT-4V prompt
# the ontology dictionary has the format {caption: class}
# where caption is the prompt sent to the base model, and class is the label that will
# be saved for that caption in the generated annotations
# then, load the model
base_model = GPT4V(
    ontology=CaptionOntology(
        {
            "person": "person",
            "a forklift": "forklift"
        }
    ),
    api_key="OPENAI_API_KEY"
)
base_model.label("./context_images", extension=".jpeg")

License

This project is licensed under an MIT license.

🏆 Contributing

We love your input! Please see the core Autodistill contributing guide to get started. Thank you 🙏 to all our contributors!

autodistill-gpt-4v's Issues

dead link in readme

Hi.

The link to the GPT-4V Autodistill documentation in the readme file, just before the installation section is redirection to a dead link (Page Not Found).

Simple code has an issue

Hello,

I am trying to test the code snippet posted on the web. However, I get the following error:

Labeling ../../People/cussion-with-a-colleague-in-an-office-characteristics-of-a-determined-person.j
Traceback (most recent call last):
File "/media/M2Disk/Projects/CompVis/ObjectDetection/AutoDistill/GPT4/autodistill-gpt-4v/genlabels.py", line 10, in
base_model.label(input_folder=folder_name, output_folder='mldata')
File "/media/M2Disk/Projects/CompVis/ObjectDetection/AutoDistill/autodistillenv/lib/python3.11/site-packages/autodistill/detection/detection_base_model.py", line 52, in label
detections = self.predict(f_path)
^^^^^^^^^^^^^^^^^^^^
TypeError: GPT4V.predict() missing 1 required positional argument: 'classes'

UnboundLocalError: local variable 'class_ids' referenced before assignment

In predict def it throws this error when in the else check condition:
UnboundLocalError: local variable 'class_ids' referenced before assignment