Skip to main content

​
What are Instruction Adherence and Context Adherence

These two metrics sound similar but are built to measure different things.
  • Context Adherence: Detects instances where the model stated information in its response that was not included in the provided context.
  • Instruction Adherence: Detects instances where the model response did not follow the instructions in its prompt.
MetricIntentionHow to UseFurther Reading
Context AdherenceWas the information in the response grounded on the contextLow adherence means improve contextLink
Instruction AdherenceDid the model follow its instructionsLow adherence means improve promptLink
Instruction Adherence is a Chainpoll-powered metric. Context Adherence has two flavors: Plus (Chainpoll-powered), or Luna (powered by in-house Luna models).

​
Context Adherence

Context Adherence refers to whether the output matches the context it was provided. It is not looking at the steps, but rather at the full context. This is more useful in RAG use-cases where you are providing additional information to supplement the output. With this metric, correctly answering based on the provided information will return a score closer to “1”, and output information which is not supported by the input would return a score closer to “0”.

​
Instruction Adherence

You can use Instruction Adherence to gauge whether the instructions in your prompt, such as “you are x, first do y, then do z” aligns with the output of that prompt. If it does, then Instruction Adherence will return that the steps were followed correctly and a score closer to “1”. If it fails to follow instructions, Instruction Adherence will return the reasoning and a score closer to “0”.
⌘I