joontju / rageval

Evaluation tools for Retrieval-augmented Generation (RAG) methods.

Rageval

Evaluation tools for Retrieval-augmented Generation (RAG) methods.

Rageval is a tool that helps you evaluate RAG system. The evaluation consists of six sub-tasks, including query rewriting, document ranking, information compression, evidence verify, answer generating, and result validating.

Definition of tasks and metrics

1. The generate task

The generate task is to answer the question based on the contexts provided by retrieval modules in RAG. Typically, the context could be extracted/generated text snippets from the compressor, or relevant documents from the re-ranker. Here, we divide metrics used in the generate task into two categories, namely answer correctness and answer groundedness.

(1) Answer Correctness: this category of metrics is to evaluate the correctness by comparing the generated answer with the groundtruth answer. Here are some commonly used metrics:

• Answer F1 Correctness: is widely used in the paper (Jiang et al.), the paper (Yu et al.), the paper (Xu et al.), and others.
• Answer NLI Correctness: also known as claim recall in the paper (Tianyu et al.).
• Answer EM Correctness: also known as Exact Match as used in the paper (Ivan Stelmakh et al.).
• Answer Bleu Score: also known as Bleu as used in the paper (Kishore Papineni et al.).
• Answer Ter Score: also known as Translation Edit Rate as used in the paper (Snover et al.).
• Answer chrF Score: also known as character n-gram F-score as used in the paper (Popovic et al.).
• Answer Disambig-F1: also known as Disambig-F1 as used in the paper (Ivan Stelmakh et al.) and the paper (Zhengbao Jiang et al.).
• Answer Rouge Correctness: also known as Rouge as used in the paper (Chin-Yew Lin).

(2) Answer Groundedness: this category of metrics is to evaluate the groundedness (also known as factual consistency) by comparing the generated answer with the provided contexts. Here are some commonly used metrics:

• Answer Citation Precision: also known as citation precision in the paper (Tianyu et al.).
• Answer Citation Recall: also known as citation recall in the paper (Tianyu et al.).
• Context Reject Rate: also known as reject rate in the paper (Wenhao Yu et al.).

2. The rewrite task

The rewrite task is to reformulate user question into a set of queries, making them more friendly to the search module in RAG.

3. The search task

The search task is to retrieve relevant documents from the knowledge base.

(1) Context Adequacy: this category of metrics is to evaluate the adequacy by comparing the retrieved documents with the groundtruth contexts. Here are some commonly used metrics:

(2) Context Relevance: this category of metrics is to evaluate the relevance by comparing the retrieved documents with the groundtruth answers. Here are some commonly used metrics:

• Context Recall: also known as Context Recall in RAGAS framework.

Installation

git clone https://github.com/gomate-community/rageval.git
cd rageval
python setup.py install

Usage

import rageval as rl

test_set = rl.datasets.load_data('ALCE', task='')
metric = rl.metrics.ContextRecall()
model = rl.models.OpenAILLM()
metric.init_model(model)

results = metric._score_batch(teset_set)

Contribution

Please make sure to read the Contributing Guide before creating a pull request.

About

This project is currently at its preliminary stage.