Customizing your LLM-powered metrics via CLHF
Learn how to customize your LLM-powered metrics with Continuous Learning via Human Feedback.
As you start using Galileo Preset LLM-powered metrics (e.g. Context Adherence or Instruction Adherence), or start creating your own LLM-powered metrics via Autogen, you might not always agree with the results. False positives or False Negatives in metric values are often due to domain edge cases that aren’t handled in the metric’s prompt.
Galileo helps you address this problem and adapt and continuously improve metrics via Continuous Learning via Human Feedback.
How it works
As you identify mistakes in your metrics, you can provide ‘feedback’ to ‘auto-improve’ your metrics. Your feedback gets translated (by LLMs) into few-shot examples that are appended to the Metric’s prompt. Few-shot examples help your LLM-as-a-judge in a few ways:
- Examples with your domain data teach it what to expect from your domain.
- Concrete examples on edge cases teach your LLM-as-a-judge how to deal with outlier scenarios.
This process has shown to increase accuracy of metrics by 20-30%.
What to enter as feedback
When entering feedback, enter a critique of the explanation generated by the erroneous metric. Be as precise as possible in your critique, outlining the exact reason behind the desired metric value.
How to use it
See this video on how to use Continuous Learning via Human Feedback to improve your metric accuracy:
Which metrics is this supported on?
- Context Adherence
- Instruction Adherence
- Correctness
- Any LLM-as-a-judge generated via Galileo’s Autogen feature