Improving AI Agent Reliability with Amazon Bedrock AgentCore Evaluations

‍

Building an AI agent that works in a demo is relatively easy. Building one that continues to work reliably as users, prompts, tools, models, and business requirements change is much harder. An agent can produce a helpful answer in one interaction and still make the wrong tool selection in another, miss part of the user’s goal, or gradually become less consistent as the system changes over time.

‍

This is why evaluating an agent requires more than looking at a few sample conversations or relying on simulated test cases. You need a way to measure agent behavior across real user interactions with the agent: complete sessions, individual responses, and tool calls, and you need those measurements to be available both during development and in production. Amazon Bedrock AgentCore Evaluations provides a managed way to do that. It uses agent telemetry to assess agent performance with built-in or custom evaluators and integrates the results with Amazon CloudWatch for analysis and monitoring.

‍

If you are new to the platform itself, our earlier post From Prototype to Production-Ready AI Agents with Amazon Bedrock AgentCore covers the AgentCore building blocks that this evaluation layer sits on top of.

‍

Start with the Agent Telemetry

AgentCore Evaluations works by evaluating the telemetry generated while your agent runs. The service supports agent frameworks such as Strands and LangGraph through OpenTelemetry (OTel) and OpenInference instrumentation, allowing agent traces to be converted into a format that can be analyzed by the evaluation service [1].

‍

This means the first step is making sure your agent is properly instrumented and that observability is enabled. When the agent runs, its traces contain the information needed to reconstruct the interaction, including user prompts, assistant responses, tool calls, and other execution details. For agents running on AgentCore Runtime, this telemetry can be integrated directly with AgentCore Observability. Agents built elsewhere can also be evaluated as long as their telemetry is available in the expected CloudWatch format [2].

‍

Once that telemetry is available, AgentCore Evaluations can turn individual interactions into measurable signals about how the agent is performing.

‍

Measure What Your Agent Is Actually Doing

One of the useful aspects of AgentCore Evaluations is that evaluation does not have to mean a single score for an entire application. Different evaluators operate at different levels of an agent interaction.

‍

Built-in evaluators cover common dimensions such as helpfulness, correctness, goal success rate, tool selection accuracy, and tool parameter accuracy. Goal success rate evaluates whether the agent achieved the user’s objectives across a session, while correctness focuses on an individual response and tool selection and parameter accuracy evaluate specific tool calls. This makes it possible to look at the agent both as a complete conversation and as a sequence of individual decisions [3].

‍

That distinction matters in practice. An agent may ultimately produce a reasonable answer while still taking an unnecessary or incorrect tool action along the way. A session-level metric can tell you whether the overall task was completed, while trace- and tool-level metrics can help explain where the interaction went wrong.

‍

AgentCore Evaluations currently provides 13 built-in evaluators covering areas such as response quality, safety, task completion, and tool usage, so teams can start evaluating an agent without building an evaluation system from scratch [4].

‍

Use LLMs as a Judge, but Control the Cost

Many of these evaluations use the strategy known as “LLM as a judge”. Instead of relying only on deterministic rules, the evaluator is given the relevant agent context and asked to score the interaction against a defined set of criteria. This makes it possible to assess qualities such as helpfulness or correctness that are difficult to capture with simple pass-or-fail checks.

‍

The trade-off is that evaluating every interaction can become expensive, particularly for agents that handle large volumes of traffic. AgentCore Evaluations addresses this by allowing online evaluations to sample sessions and apply filtering rules, so you can evaluate a representative portion of production traffic rather than every interaction. You can also run on-demand evaluations against specific traces when you need to investigate a particular issue or validate a change [5].

‍

In practice, it is also useful to think carefully about the judge model itself. Using a different model from the one powering your agent can provide another perspective on the agent’s behavior and reduce the risk of evaluating the system against the same model assumptions that produced the response.

‍

Build Evaluations for Your Own Definition of Quality

Built-in evaluators are a useful starting point, but every agent eventually develops requirements that are specific to its domain. This is where custom evaluators become important.

‍

With a custom LLM-based evaluator, you define the evaluation instructions, the judge model, and the scoring schema. AgentCore provides prompt templates that can be extended with the context of the interaction being evaluated. At the session level, that context can include the conversation history, assistant responses, and tool calls across the session. At the trace level, the evaluator can receive the previous conversation together with the current assistant response. At the tool level, it can evaluate the available tools, prior context, and the specific tool call under evaluation [3].

‍

This creates a useful pattern for evaluation design. Instead of writing a completely different prompt for every interaction, you can define a reusable evaluation instruction and let AgentCore populate the relevant runtime information before sending it to the judge model. You can then define your own scoring scale and choose whether the evaluator operates at the session, trace, or tool-call level.

‍

For example, a customer support agent could have a custom metric that evaluates whether responses follow a particular escalation policy. A financial agent could be evaluated against a domain-specific communication standard. These are the kinds of requirements that generic quality metrics cannot fully capture.

‍

Apart from the “LLM as a judge”, AgentCore Evaluations also supports code-based custom evaluators through AWS Lambda. This is useful when the evaluation can be expressed as deterministic logic, such as checking a structured response, applying a business rule, calling an external system, or calculating a custom metric without using an LLM [6].

‍

Use Ground Truth When “Probably Correct” Is Not Enough

There are cases where a general-purpose LLM judge is not enough. An answer can sound reasonable and still be wrong according to the exact rules of your business or domain.

‍

Ground truth evaluations provide a way to make those requirements explicit. Instead of asking the evaluator to decide what a good response looks like based only on its general knowledge, you provide a known expected outcome that the agent can be compared against. AgentCore supports ground truth for scenarios such as expected responses, behavioral assertions, and expected tool trajectories [7].

‍

For example, imagine an agent that handles insurance claims. A generic correctness evaluator may determine that a response sounds appropriate, but your organization may have a specific expected answer for a particular claim scenario. By supplying that expected response as ground truth, the evaluation can focus on whether the agent meets your domain-specific definition of correctness rather than relying solely on a more subjective interpretation. You can also use natural language assertions with the goal success rate metric to define specific conditions that the agent should satisfy throughout an entire session. The evaluator can then assess whether the agent’s behavior met those conditions across the conversation. For example, you could define an assertion such as: “The agent must verify the customer’s policy details before providing a claim decision.” The evaluator can then examine the complete session and determine whether the agent followed this requirement, even if the final response itself appears correct.

‍

This makes ground truth particularly useful for benchmark datasets, regression testing, and validating changes where you need a consistent reference point. It also helps turn agent quality from a subjective discussion into an objective metric.

‍

Bring Evaluation Results into CloudWatch

The evaluation process becomes even more useful when it is connected to the observability data generated by your agent. As mentioned earlier, agent frameworks such as Strands can be instrumented with OpenTelemetry (OTel) to generate traces and spans for agent interactions and send that telemetry to Amazon CloudWatch [8]. AgentCore Evaluations can then use this telemetry to reconstruct the relevant sessions, traces, and spans and run the configured evaluators against them. Depending on the evaluator, an evaluation can be applied at the session, trace, or span level.

‍

The resulting assessments and scores are displayed in the Evaluations dashboard within the GenAI observability section of Amazon CloudWatch. This gives you a view of how your agent is performing over time, with evaluation trends that can help identify potential quality issues. Instead of looking at individual interactions in isolation, you can monitor metrics such as correctness, helpfulness, or goal success rate and see how they change as your agent evolves. You can also drill down from the evaluation results to the specific sessions, traces, or spans that were evaluated, making it easier to understand the behavior behind a particular score.

‍

These evaluation results can also be treated as CloudWatch metrics. This means you can create alarms based on evaluation thresholds and use them as another signal for monitoring your agent in production. For example, a sustained drop in a correctness metric could trigger an alarm, prompting the team to investigate the underlying sessions, traces, and spans to understand what changed [9].

‍

This makes evaluation part of the same monitoring and troubleshooting workflow used for the rest of your application. Rather than treating evaluation as a separate testing activity, teams can use the scores to identify potential problems, investigate the underlying agent behavior, and diagnose issues using the telemetry available in CloudWatch. The result is a continuous feedback loop between agent behavior, evaluation, and production observability.

‍

‍

From Agent Monitoring to Continuous Improvement

The most useful way to think about AgentCore Evaluations is not as a final test that happens once before deployment. It is part of the development and operational lifecycle of an agent. During development, on-demand [5] and batch [10] evaluations can help establish a baseline and test changes against a known set of interactions. In production, online evaluations can continuously sample live traffic and track how quality changes over time. When a low-scoring interaction is identified, the underlying trace can be investigated, the agent can be updated, and the evaluation can be run again to validate the change. This creates a more practical feedback loop for agent engineering. Instead of asking whether an agent is “good enough,” you can measure specific dimensions of quality (including custom ones), identify where failures occur, and use those results to guide improvements.

‍

What This Means for Your Agent

Agentic systems are becoming more capable, but that capability also makes them harder to evaluate with traditional application testing. An agent can take different paths to solve the same problem, make multiple tool calls, and produce responses whose quality depends on the context of the entire interaction. An isolated response or even a single successful session might appear to be working as expected, but it may not expose the edge cases and unexpected behaviors that emerge across real user interactions.

‍

Amazon Bedrock AgentCore Evaluations gives you a way to make that behavior measurable. Built-in evaluators provide a starting point for common quality dimensions. Custom evaluators let you define metrics around your own business requirements. Ground truth allows you to compare behavior against known expectations, while CloudWatch brings the resulting quality metrics together with the telemetry needed to understand what happened.

‍

What’s Next

Building a reliable agent is not only about choosing the right foundation model or designing the right tools. It is also about having a repeatable way to understand whether the agent is continuously behaving in the way that you expect. Amazon Bedrock AgentCore Evaluations provides that foundation with managed evaluators, custom evaluation logic, ground truth support, and integration with CloudWatch observability. Regardless of whether you are validating an agent during development or monitoring one in production, the same evaluation approach can help turn agent behavior into insightful metrics.

‍

If you are building an AI agent on AWS and want to understand how to define the right evaluation metrics for your use case, connect your telemetry, or build a feedback loop around agent quality, let’s look at how AgentCore Evaluations can fit into your architecture.

‍

References

‍

[1] “Evaluate agent performance with Amazon Bedrock AgentCore Evaluations”, AWS Docs,

https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/evaluations.html

‍

[2] “Telemetry setup and delivery”, AWS Docs,

https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/supported-frameworks-telemetry.html

‍

[3] “Prompt templates”, AWS Docs,

https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/prompt-templates-builtin.html

‍

[4] “Amazon Bedrock AgentCore Evaluations is now generally available”, AWS News,

https://aws.amazon.com/about-aws/whats-new/2026/03/agentcore-evaluations-generally-available/

‍

[5] “On-demand evaluation”, AWS Docs,

https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/on-demand-evaluations.html

‍

[6] “Custom code-based evaluator”, AWS Docs,

https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/code-based-evaluators.html

‍

[7] “Ground truth evaluations”, AWS Docs,

https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/ground-truth-evaluations.html

‍

[8] “Observe your agent applications on Amazon Bedrock AgentCore Observability”, AWS Docs,

https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability.html

‍

[9] “Agent details - Evaluations”, AWS Docs,

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/session-traces-evaluations.html

‍

[10] “Batch evaluation”, AWS Docs,

https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/batch-evaluations.html

‍