program with knn and annotate requires too many tokens about dspy

[okhat]

456171 tokens, wow! That seems incredibly large. Something went really wrong in your run.

Are you able to share more details? I think some of your parameters have to be really off, maybe the number of demonstrations.

Also, this seems like it's using turbo. Is that true?

[ylchin]

yes, it's using turbo, but even when i was running text-davinci-002, the token window often exceeded

[okhat]

I see. Before we dig into that, why/how is it 456,00 tokens? I'd understand if it was 5,000 for example. Are you able to share more information? How many demonstrations are you using?

[ylchin]

i'm using 5 demonstrations

[okhat]

Can you reduce it to 1?

[ylchin]

even when I reduced the number of demos to 1
x.demos = dsp.annotate(multihop_attempt)(demos, k=1, return_all=True)
it still exceeds the count
InvalidRequestError: This model's maximum context length is 4097 tokens. However, your messages resulted in 456675 tokens. Please reduce the length of the messages.

[okhat]

Can you return_all = False?

[ylchin]

ive tried and the same thing still returns

[okhat]

Okay… can you print the prompt inside the generate function? This is a bit absurd, with 450,000 tokens! That’s like 200 times more than it normally is

[ylchin]

I'm not sure why but after the first retrieval of passages for the annotation of the demos after sampling, the program passes all the questions and answers of the entire training set to gpt, including those that were not in demos

[okhat]

Ah okay. That makes sense. It’s got to do with your annotate call. Is it setting k=1 and return_all=False? If so, you can add an assert that the length of the output is 1 demo.

[ylchin]

in multihop_demonstrate, i have changed it to

def multihop_demonstrate(question: str, x: dsp.Example, n: int) -> dsp.Example:
    demos = knn_search(question, train, dev, n)
    x.demos = dsp.annotate(multihop_attempt)(demos, k=1, return_all=False)
    assert len(x.demos) == 1, "Output length is not 1"
    return x

but the same error is appearing. the other variable is the knn_search function, but it works with all the other programs in the notebook
for reference, the function was formulated following the knn_demonstrations notebook

def knn_search(question: str, train: list[Example], dev: list[Example], k):
    dsp.settings.configure(vectorizer=dsp.NaiveGetFieldVectorizer())
    knn_func = dsp.knn(train)

    def find_query_location(value, data):
        for index, item in enumerate(data):
            if isinstance(item, dict):
                # Check if the value is in the dictionary
                if value in item.values():
                    return [index]
                else:
                    # Recursively search the dictionary
                    result = find_query_location(value, item)
                    if result:
                        return [index] + result
            elif isinstance(item, list):
                # Recursively search the list
                result = find_query_location(value, item)
                if result:
                    return [index] + result
            else:
                return None
    
    QUERY_IDX = find_query_location(question, dev)[0]
    knn_res_dev = knn_func(dev[QUERY_IDX], k)
    
    print(question)  # the query itself
    # similar questions
    print('-'*80)
    print([train.index(i) for i in knn_res_dev])
    for i in [train[train.index(i)].question for i in knn_res_dev]:
        print(i)
    
    return knn_res_dev