Learn how to customize your LLM-powered metrics with Continuous Learning via Human Feedback.
As you start using Galileo Preset LLM-powered metrics (e.g. Context Adherence or Instruction Adherence), or start creating your own LLM-powered metrics via Autogen, you might not always agree with the results. False positives or False Negatives in metric values are often due to domain edge cases that aren’t handled in the metric’s prompt.
Galileo helps you address this problem and adapt and continuously improve metrics via Continuous Learning via Human Feedback.
As you identify mistakes in your metrics, you can provide ‘feedback’ to ‘auto-improve’ your metrics. Your feedback gets translated (by LLMs) into few-shot examples that are appended to the Metric’s prompt. Few-shot examples help your LLM-as-a-judge in a few ways:
This process has shown to increase accuracy of metrics by 20-30%.
When entering feedback, enter a critique of the explanation generated by the erroneous metric. Be as precise as possible in your critique, outlining the exact reason behind the desired metric value.
See this video on how to use Continuous Learning via Human Feedback to improve your metric accuracy:
Learn how to customize your LLM-powered metrics with Continuous Learning via Human Feedback.
As you start using Galileo Preset LLM-powered metrics (e.g. Context Adherence or Instruction Adherence), or start creating your own LLM-powered metrics via Autogen, you might not always agree with the results. False positives or False Negatives in metric values are often due to domain edge cases that aren’t handled in the metric’s prompt.
Galileo helps you address this problem and adapt and continuously improve metrics via Continuous Learning via Human Feedback.
As you identify mistakes in your metrics, you can provide ‘feedback’ to ‘auto-improve’ your metrics. Your feedback gets translated (by LLMs) into few-shot examples that are appended to the Metric’s prompt. Few-shot examples help your LLM-as-a-judge in a few ways:
This process has shown to increase accuracy of metrics by 20-30%.
When entering feedback, enter a critique of the explanation generated by the erroneous metric. Be as precise as possible in your critique, outlining the exact reason behind the desired metric value.
See this video on how to use Continuous Learning via Human Feedback to improve your metric accuracy: