Overview
Action Completion addresses the common pain points of agent performance by measuring whether AI agents are actually helping users achieve their end goal rather than just providing responses. Action Completion is successful when all of the below are true: :- The agent provides a complete response that addresses a question
- The agent provides a confirmation of successful action in the case of a request
- The response is coherent and factually accurate
- The response comprehensively addresses every aspect of the user’s request
- The response avoids contradicting tool outputs
- The response summarizes all relevant parts returned by tools
Action Completion at a glance
Calculation method
If Action Completion returnsfalse, the agent did not accomplish every user goal.
1
Additional Requests
Multiple requests are sent to an LLM using a carefully designed chain-of-thought prompt that adheres to the definition above.
2
Judgment Responses
The LLM generates multiple distinct responses, each containing:
- An explanation
- A final judgment: “Yes” (goal accomplished) or “No” (goal not accomplished)
3
Result
The evaluator returns
true when the agent accomplishes every user goal and false otherwise.4
Explanation Surfacing
One explanation is surfaced, chosen to align with the majority judgment among the responses.
This metric requires multiple LLM calls to compute, which may impact usage and billing.
How to improve Action Completion scores
To optimize your agent’s performance and ensure high Action Completion scores, focus on comprehensive goal accomplishment and complete response generation.Common issues and solutions
Best practices for optimization
- Track Progress Over Time: Monitor Action Completion scores across different versions of your agent to identify trends and ensure continuous improvements in task completion capabilities.
- Analyze Failure Patterns: When Action Completion scores are low, examine specific steps or scenarios where agents fail to meet user goals. Use this analysis to identify and address systematic issues.
- Combine with Other Metrics: Use Action Completion alongside other agentic metrics, such as Action Advancement, to get a comprehensive view of your assistant’s effectiveness and identify areas for improvement.
- Test Edge Cases: Create evaluation datasets that include complex, multi-step tasks to thoroughly assess your agent’s ability to handle challenging scenarios and advance user goals effectively.
When optimizing for Action Completion, ensure you’re not sacrificing other important aspects like safety, factual accuracy, or user experience in pursuit of task completion.
Comparison to other metrics
Performance Benchmarks
We evaluated Action Completion against human expert labels on an internal dataset of agentic conversation samples using top frontier models.GPT-4.1 Classification Report
Benchmarks based on internal evaluation dataset. Performance may vary by use case.
Related Resources
If you would like to dive deeper or start implementing Action Completion, check out the following resources:Examples
- Action Completion Examples - Log in and explore the “Action Completion” Log Stream in the “Preset Metric Examples” Project to see this metric in action.