Rate limit. about chatdev

[tbels89]

@sjetha thanks for the tenacity suggestion. i was able to implement and using gpt4. i reviewed the log and it did have some rate limit hits, but because of the retry, never killed the process and chatdev was able to complete the task (not exactly what i was hoping it would output, but at least not hitting rate limits). for everyone wondering, here's what i did:

file: chatdev-main/camel/model_backend.py

import: from tenacity import retry, wait_exponential

under the OpenAIModel class, just above the run function, i added the following line of code:
@Retry(wait=wait_exponential(multiplier=0.02, max=20))

[flipflop4]

For those doing a direct copy and paste of @tbels89 suggested modification, I believe the 'R' in @Retry should be lower case, i.e., @Retry(...)

Edit: comments automatically capitalise the letter following '@' so '@ retry(...)' without space

[wghuang-aigc]

Why has this been closed? it never works for me - always errors for GPT 4 - I fought hard to get access :)

quote your api_key in your code

[Abemelech]

@GBurgardt Try adding in file: chatdev-main/camel/agent/role_playing.py

The following at the top of the file

from tenacity import retry, wait_exponential

And the following

@retry(wait=wait_exponential(multiplier=60, max=100))

on top of this function

def step(

Line 235 (unless there was any change)

If this doesn't work. Copy your error code into GPT-4 and ask it where you should add @retry(wait=wait_exponential(multiplier=60, max=100)) and it should guide you to the right direction

[hemangjoshi37a]

refer to this page for rate limits https://platform.openai.com/account/rate-limits

[fasterinnerlooper]

I followed the example using Tenacity here, and I was able to limit the requests made to the GPT-4 endpoints.
I'll need to dial in the variables, but it works out the gate without any modifications.
https://platform.openai.com/docs/guides/rate-limits/error-mitigation

[dipique]

If this doesn't work. Copy your error code into GPT-4 and ask it where you should add @retry(wait=wait_exponential(multiplier=60, max=100)) and it should guide you to the right direction

He would but he has hit his rate limit.

[belacmu]

Having the same issue here with gpt-4, not sure why this is closed.

[wghuang-aigc]

Why has this been closed? it never works for me - always errors for GPT 4 - I fought hard to get access :)

quote your api_key in your code

It will work ，you can try it

[thinkwee]

Thank you. We will check it out.
By the way, in our test we haven't met this rate limit error, is it because some kinds of batch running in your scenario?

[IIEleven11]

I am hitting the rate limit error too. I was using my GPT_4 API key. I tried the 32K version and got a different error. No batch script, just running from from inside VsCode terminal/windows 11.
"python run.py --task "write a me this example program" --name "Example Name" --model 'GPT_4'

Tracebacks below.

GPT 4 Traceback

[OpenAI_Usage_Info Receive]
prompt_tokens: 453
completion_tokens: 74
total_tokens: 527

Traceback (most recent call last):
File "H:\PROGRAMS\chatdev\venv\Lib\site-packages\tenacity_init_.py", line 382, in call
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "H:\PROGRAMS\chatdev\ChatDev\camel\utils.py", line 145, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\PROGRAMS\chatdev\ChatDev\camel\agents\chat_agent.py", line 191, in step
response = self.model_backend.run(messages=openai_messages)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\PROGRAMS\chatdev\ChatDev\camel\model_backend.py", line 69, in run
response = openai.ChatCompletion.create(*args, **kwargs,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\PROGRAMS\chatdev\venv\Lib\site-packages\openai\api_resources\chat_completion.py", line 25, in create
return super().create(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\PROGRAMS\chatdev\venv\Lib\site-packages\openai\api_resources\abstract\engine_api_resource.py", line 153, in create
response, _, api_key = requestor.request(
^^^^^^^^^^^^^^^^^^
File "H:\PROGRAMS\chatdev\venv\Lib\site-packages\openai\api_requestor.py", line 298, in request
resp, got_stream = self._interpret_response(result, stream)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\PROGRAMS\chatdev\venv\Lib\site-packages\openai\api_requestor.py", line 700, in _interpret_response
self._interpret_response_line(
File "H:\PROGRAMS\chatdev\venv\Lib\site-packages\openai\api_requestor.py", line 763, in _interpret_response_line
raise self.handle_error_response(
openai.error.RateLimitError: Rate limit reached for 10KTPM-200RPM in organization org-PcnGPHPM8C05nx9hV0x7Wie6 on tokens per min. Limit: 10000 / min. Please try again in 6ms. Contact us through our help center at help.openai.com if you continue to have issues.

GPT4 32K TRACEBACK

"python run.py --task "write a me this example program" --name "Example Name" --model 'GPT_4_32K'

Traceback (most recent call last):
File "H:\PROGRAMS\chatdev\venv\Lib\site-packages\tenacity_init_.py", line 382, in call
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "H:\PROGRAMS\chatdev\ChatDev\camel\utils.py", line 145, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\PROGRAMS\chatdev\ChatDev\camel\agents\chat_agent.py", line 191, in step
response = self.model_backend.run(messages=openai_messages)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\PROGRAMS\chatdev\ChatDev\camel\model_backend.py", line 69, in run
response = openai.ChatCompletion.create(*args, **kwargs,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\PROGRAMS\chatdev\venv\Lib\site-packages\openai\api_resources\chat_completion.py", line 25, in create
return super().create(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\PROGRAMS\chatdev\venv\Lib\site-packages\openai\api_resources\abstract\engine_api_resource.py", line 153, in create
response, _, api_key = requestor.request(
^^^^^^^^^^^^^^^^^^
File "H:\PROGRAMS\chatdev\venv\Lib\site-packages\openai\api_requestor.py", line 298, in request
resp, got_stream = self._interpret_response(result, stream)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\PROGRAMS\chatdev\venv\Lib\site-packages\openai\api_requestor.py", line 700, in _interpret_response
self._interpret_response_line(
File "H:\PROGRAMS\chatdev\venv\Lib\site-packages\openai\api_requestor.py", line 763, in _interpret_response_line
raise self.handle_error_response(
openai.error.InvalidRequestError: The model gpt-4-32k does not exist or you do not have access to it. Learn more: https://help.openai.com/en/articles/7102672-how-can-i-access-gpt-4.

[loprx]

Thank you. We will check it out.

By the way, in our test we haven't met this rate limit error, is it because some kinds of batch running in your scenario?

As they said, this problem occurs when using the GPT4 model.

[IIEleven11]

refer to this page for rate limits https://platform.openai.com/account/rate-limits

Is this a joke? I understand what rate limits are. I am trying to use Open Interpreter with a gpt4 api key and I cannot because there is no functionality that allows for waiting the required amount of time and or working around the limit in some other way.

[Nbtguyoriginal]

FORK THE Repo

Change the code to use v2 completion endpoint
May fix the issue

[Storhemulen]

I get the same error. Works fine with GPT3.5 but if I try GPT4 it crashes to:
openai.error.RateLimitError: Rate limit reached for 10KTPM-200RPM

[wghuang-aigc]

I get the same error. Works fine with GPT3.5 but if I try GPT4 it crashes to:
openai.error.RateLimitError: Rate limit reached for 10KTPM-200RPM

[tbels89]

also experiencing this issue using GPT4. is there somewhere in the code where i can add a rate limit or adjust how requests are sent to the API so that i can run this and ensure rate limit isn't reached so my project can complete? i prefer to use GPT4 because 1. i have access and 2. it seems to perform better, particularly with coding, from my experience.

[fasterinnerlooper]

also experiencing this issue using GPT4. is there somewhere in the code where i can add a rate limit or adjust how requests are sent to the API so that i can run this and ensure rate limit isn't reached so my project can complete? i prefer to use GPT4 because 1. i have access and 2. it seems to perform better, particularly with coding, from my experience.

Hi @tbels89,
If you follow the link in my comment above, you can see an example that uses the Tenacity library. The examples that are there need to be tweaked because it backs off quite aggressively, but it was working well for me.

[tbels89]

Hi @tbels89, If you follow the link in my comment above, you can see an example that uses the Tenacity library. The examples that are there need to be tweaked because it backs off quite aggressively, but it was working well for me.

@sjetha i had a look but i'm not sure where in the program to put the tenacity. what file(s) did you modify to get this working? can you show an example of your implementation? i'm still pretty new to this and trying to figure it all out.

[karingijs2727]

Can some one show an example code or tell how to fix it?

[loprx]

Can some one show an example code or tell how to fix it?

You can add sleep code in the request openai API.

[sakher]

Why has this been closed? it never works for me - always errors for GPT 4 - I fought hard to get access :)

[sebib96]

Like @sakher mentioned, add his code and mess with variables. I added:

@Retry(wait=wait_exponential(min=1, max=60)) above the OpenAIModel class and my issues were fixed (using 3.5 turbo)

[GBurgardt]

Still facing rate limit error with GPT-4. Any confirmed fix?

[fasterinnerlooper]

I think it was closed likely because there is an easy solution that involves modifying the code as outlined in other comments in this Issue. If someone wants to put a PR together than gives this work-around to all then that would be the next logical step.

[gennadiyvorgy1]

Solved like this
`class OpenAIModel(ModelBackend):
r"""OpenAI API in a unified ModelBackend interface.""""""

def __init__(self, model_type: ModelType, model_config_dict: Dict) -> None:
    super().__init__()
    self.model_type = model_type
    self.model_config_dict = model_config_dict




@retry(
    tries=10, # Number of tries
    delay=3, # Delay between attempts in seconds
    max_delay=30, # Maximum delay between attempts
    backoff=2, # Coefficient of exponential growth of delay
    jitter=(0, 1) # Delay randomization
)
def run(self, *args, **kwargs) -> Dict[str, Any]:
    string = "\n".join([message["content"] for message in kwargs["messages"]])
    encoding = tiktoken.encoding_for_model(self.model_type.value)
    num_prompt_tokens = len(encoding.encode(string))
    gap_between_send_receive = 15 * len(kwargs["messages"])
    num_prompt_tokens += gap_between_send_receive


    num_max_token_map = {
        "gpt-3.5-turbo": 4096,
        "gpt-3.5-turbo-16k": 16384,
        "gpt-3.5-turbo-0613": 4096,
        "gpt-3.5-turbo-16k-0613": 16384,
        "gpt-4": 8192,
        "gpt-4-0613": 8192,
        "gpt-4-32k": 32768,
    }
    num_max_token = num_max_token_map[self.model_type.value]
    num_max_completion_tokens = num_max_token - num_prompt_tokens
    self.model_config_dict['max_tokens'] = num_max_completion_tokens


    # Add a delay before sending the request
    sleep(3)


    response = openai.ChatCompletion.create(*args, **kwargs, model=self.model_type.value, **self.model_config_dict)


    log_and_print_online(
        "**[OpenAI_Usage_Info Receive]**\nprompt_tokens: {}\ncompletion_tokens: {}\ntotal_tokens: {}\n".format(
            response["usage"]["prompt_tokens"], response["usage"]["completion_tokens"],
            response["usage"]["total_tokens"]))
    if not isinstance(response, Dict):
        raise RuntimeError("Unexpected return from OpenAI API")
    return response`

[qianyouliang]

if your api only support gpt-3.5,your api rate limit is 3times/minute,so in order to success running,you must revise your code（ChatDev\camel\model_backend.py） ,like this!

while True:            
            try:
                response = openai.ChatCompletion.create(*args, **kwargs, model=self.model_type.value, **self.model_config_dict)
                break
            except AttributeError:
                response = openai.chat.completions.create(*args, **kwargs, model=self.model_type.value, **self.model_config_dict)
                break
            except Exception as exc:
                print("Rate limit exceeded, waiting for 60 seconds...")
                time.sleep(60)

[fasterinnerlooper]

@qianyouliang Can you add this as a PR? We'll see if these guys will approve it since it's a good addition to the codebase.

[SuzdalenkoAlexey]

client = OpenAI(api_key=settings.IA_API_KEY,)
chat_completion = client.chat.completions.create(
messages=[
{ "role": "user", "content": 'esto es una prueba todo ok', }
],
model="gpt-3.5-turbo"
)

Error code: 429 - {'error': {'message': 'Rate limit reached for gpt-3.5-turbo in organization org-rxenTzjMkf2Po8dsexe410Mg on requests per min (RPM): Limit 3, Used 3, Requested 1. Please try again in 20s. Visit