Ragaaf - adding new metric 'context recall' (#184)

* small fix for ragas.py Signed-off-by: aasavari <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed error when metrics arg is used Signed-off-by: aasavari <[email protected]> * updated README Signed-off-by: aasavari <[email protected]> * added key features Signed-off-by: aasavari <[email protected]> * edited formatting Signed-off-by: aasavari <[email protected]> * improved readability Signed-off-by: aasavari <[email protected]> * improved note in model section Signed-off-by: aasavari <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * spell check Signed-off-by: aasavari <[email protected]> * adding context recall Signed-off-by: aasavari <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: aasavari <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
opea-project · Nov 2, 2024 · cc7cebd · cc7cebd
1 parent f995c9c
commit cc7cebd
Showing 1 changed file with 13 additions and 0 deletions.
diff --git a/evals/metrics/ragaaf/prompt_templates/context_recall.py b/evals/metrics/ragaaf/prompt_templates/context_recall.py
@@ -0,0 +1,13 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+
+class ContextRecall:
+    name = "context_recall"
+    required_columns = ["context", "ground_truth"]
+    template = """- Context Recall: context recall measures how well ground truth is supported by the context.
+  - Score 1: None of facts/steps/aspects in the ground truth are supported by the context.
+  - Score 2: Only a small part of facts/steps/aspects in the ground truth are supported by the context.
+  - Score 3: About half of facts/steps/aspects in the ground truth are supported by the context.
+  - Score 4: All main/important facts/steps/aspects in the ground truth are supported by the context but some of the details are not supported by the context.
+  - Score 5: All of facts/steps/aspects in the ground truth are supported by the context. """