Understand the distinctions between Context Adherence and Instruction Adherence metrics in Galileo Evaluate to assess generative AI outputs accurately.
These two metrics sound similar but are built to measure different things.
Metric | Intention | How to Use | Further Reading |
---|---|---|---|
Context Adherence | Was the information in the response grounded on the context | Low adherence means improve context | Link |
Instruction Adherence | Did the model follow its instructions | Low adherence means improve prompt | Link |
Instruction Adherence is a Chainpoll-powered metric. Context Adherence has two flavors: Plus (Chainpoll-powered), or Luna (powered by in-house Luna models).
Context Adherence refers to whether the output matches the context it was provided. It is not looking at the steps, but rather at the full context. This is more useful in RAG use-cases where you are providing additional information to supplement the output. With this metric, correctly answering based on the provided information will return a score closer to “1”, and output information which is not supported by the input would return a score closer to “0”.
You can use Instruction Adherence to gauge whether the instructions in your prompt, such as “you are x, first do y, then do z” aligns with the output of that prompt. If it does, then Instruction Adherence will return that the steps were followed correctly and a score closer to “1”. If it fails to follow instructions, Instruction Adherence will return the reasoning and a score closer to “0”.
Understand the distinctions between Context Adherence and Instruction Adherence metrics in Galileo Evaluate to assess generative AI outputs accurately.
These two metrics sound similar but are built to measure different things.
Metric | Intention | How to Use | Further Reading |
---|---|---|---|
Context Adherence | Was the information in the response grounded on the context | Low adherence means improve context | Link |
Instruction Adherence | Did the model follow its instructions | Low adherence means improve prompt | Link |
Instruction Adherence is a Chainpoll-powered metric. Context Adherence has two flavors: Plus (Chainpoll-powered), or Luna (powered by in-house Luna models).
Context Adherence refers to whether the output matches the context it was provided. It is not looking at the steps, but rather at the full context. This is more useful in RAG use-cases where you are providing additional information to supplement the output. With this metric, correctly answering based on the provided information will return a score closer to “1”, and output information which is not supported by the input would return a score closer to “0”.
You can use Instruction Adherence to gauge whether the instructions in your prompt, such as “you are x, first do y, then do z” aligns with the output of that prompt. If it does, then Instruction Adherence will return that the steps were followed correctly and a score closer to “1”. If it fails to follow instructions, Instruction Adherence will return the reasoning and a score closer to “0”.