-
Notifications
You must be signed in to change notification settings - Fork 52
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Breaking change] Add support for all Granite Guardian risks #1576
Conversation
I don't agree on calling 'question ' to the default |
The RAG Granite Guardian metrics are duplicated now, but not exactly. The current Granite Guardian RAG metric is defined differently in the catalog. @elronbandel let me know if this is the way I should define the metrics in the prepare file and how should I adapt the example. |
I also changed the Granite-based LLM evaluators. I removed the '0' from the version as it looked awkward. Let me know if it is to late to make that change and I will revert it. |
@arielge FYI, we are adding all the missing Granite Guardian risks. |
552fb4a
to
84e15e9
Compare
a5cc196
to
2a403f2
Compare
src/unitxt/metrics.py
Outdated
|
||
def prepare(self): | ||
self.reduction_map = {"mean": [self.main_score]} | ||
def verify_guardian_config(self, task_data): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The following lines are very complicated. Can we instead use Inheritance? So we have BaseGuerdianGraniteMetric
Then RagGuerdianGraniteMetric
where minimal things are changed? Not sure if possible, but I think it will might make this code section as well as it usage more simple
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it makes sense, I will implement it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@elronbandel done
d570a32
to
0aa47f1
Compare
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
cf95b88
to
0cf1858
Compare
Signed-off-by: Martín Santillán Cooper <[email protected]>
Signed-off-by: Martín Santillán Cooper <[email protected]>
0cf1858
to
c451c57
Compare
Signed-off-by: Martín Santillán Cooper <[email protected]>
b4f77b3
to
2548490
Compare
This PR adds support for all available Granite Guardian risks, extending current support of RAG risks, plus bug fixes and several improvements. In summary, this PR:
GraniteGuardianWMLMetric
was re-implemented andGraniteGuardianRagRisk
can be used in its place.GraniteGuardianBase
and classes with nameGraniteGuardian<risk_type>Risk
inherit from it by implementing methodsverify_granite_guardian_config()
andprocess_input_fields()
.LogProbInferenceEngine
that offers Granite Guardian models.