Context vs. Instruction Adherence | Galileo Evaluate FAQ
Understand the distinctions between Context Adherence and Instruction Adherence metrics in Galileo Evaluate to assess generative AI outputs accurately.
What are Instruction Adherence and Context Adherence
These two metrics sound similar but are built to measure different things.
- Context Adherence: Detects instances where the model stated information in its response that was not included in the provided context.
- Instruction Adherence: Detects instances where the model response did not follow the instructions in its prompt.
| Metric | Intention | How to Use | Further Reading |
|---|---|---|---|
| Context Adherence | Was the information in the response grounded on the context | Low adherence means improve context | Link |
| Instruction Adherence | Did the model follow its instructions | Low adherence means improve prompt | Link |
Instruction Adherence is a Chainpoll-powered metric. Context Adherence has two flavors: Plus (Chainpoll-powered), or Luna (powered by in-house Luna models).
Context Adherence
Context Adherence refers to whether the output matches the context it was provided. It is not looking at the steps, but rather at the full context. This is more useful in RAG use-cases where you are providing additional information to supplement the output. With this metric, correctly answering based on the provided information will return a score closer to “1”, and output information which is not supported by the input would return a score closer to “0”.
Instruction Adherence
You can use Instruction Adherence to gauge whether the instructions in your prompt, such as “you are x, first do y, then do z” aligns with the output of that prompt. If it does, then Instruction Adherence will return that the steps were followed correctly and a score closer to “1”. If it fails to follow instructions, Instruction Adherence will return the reasoning and a score closer to “0”.