Learn how to customize your LLM-powered metrics with Continuous Learning via Human Feedback.
As you start using Galileo Preset LLM-powered metrics (e.g. Context Adherence or Instruction Adherence),
or start creating your own LLM-powered metrics via Autogen, you might not always agree with the results.
False positives or False Negatives in metric values are often due to domain edge cases that aren’t handled
in the metric’s prompt.Galileo helps you address this problem and adapt and continuously improve metrics via Continuous Learning
via Human Feedback.
As you identify mistakes in your metrics, you can provide ‘feedback’ to ‘auto-improve’ your metrics. Your
feedback gets translated (by LLMs) into few-shot examples that are appended to the Metric’s prompt. Few-shot
examples help your LLM-as-a-judge in a few ways:
Examples with your domain data teach it what to expect from your domain.
Concrete examples on edge cases teach your LLM-as-a-judge how to deal with outlier scenarios.
This process has shown to increase accuracy of metrics by 20-30%.
CLHF-ed metrics are scoped to the project. I.e. you can have different teams customizing the same metric in different ways and not impact each other’s projects.
When entering feedback, enter a critique of the explanation generated by the erroneous metric. Be as precise
as possible in your critique, outlining the exact reason behind the desired metric value.