Cannot use annotations/citations as CONTEXT about bootcamp

[shahules786]

Hi @adorosario, Shahul from Ragas here. Not sure If I understand your concern currently. But the context in ragas refers to retrieved context. Ragas metrics (that require context like faithfulness) can only be used with this information being available.

[zc277584121]

@adorosario thanks for your advice. As far as I know, openai provides context: https://platform.openai.com/docs/assistants/how-it-works/message-annotations
so it seem not a black box RAG agent.

[adorosario]

@zc277584121

@adorosario thanks for your advice. As far as I know, openai provides context: https://platform.openai.com/docs/assistants/how-it-works/message-annotations so it seem not a black box RAG agent.

Sorry - I think you might be mis-understanding the difference between "annotations" and "context".

Context is the (possibly) thousands of tokens/words from the knowledge base that is used to create the response. For example, in our RAG platform, we could technically use tens of thousands of words of context (think of it as 20-30 pages) to create the response. This CONTEXT is never shown to the client in a black-box RAG service like OpenAI assistants or CustomGPT.ai

Annotations are tiny snippets of text from the knowledge base that were used to construct the answer .. they are typically about 10% (if at all) of the context that was used.

Using them interchangeably in a black-box RAG would be wrong and skew your benchmarks.

@shahules786

Shahul from Ragas here. Not sure If I understand your concern currently. But the context in ragas refers to retrieved context. Ragas metrics (that require context like faithfulness) can only be used with this information being available.

Shahul -- yes - in a black box RAG service like OpenAI assistants or CustomGPT.ai, the retrieved CONTEXT is never available to the client -- so any benchmark metric that involves retrieved CONTEXT cannot be computed (which ragas does correctly). Using annotations and context interchangeably to compute any metric like faithfulness would be wrong - that is the reason for raising this issue.