CertBrain
Courses
Home
Blog
Sign in
← Back to decks
Sign in to save your progress, vote, and build your own decks.Sign in

Evaluating RAG, Agents and Workflows

Most real applications don't call a foundation model once and show the answer. A RAG system searches your documents first, an agent may call several tools, and a workflow chains many steps. When the final answer is wrong, you need to know which step failed. So evaluating an FM-based application…

10 cards
Study this deck

Evaluating RAG, Agents and Workflows

Most real applications don't call a foundation model once and show the answer. A RAG system searches your documents first, an agent may call several tools, and a workflow chains many steps. When the final answer is wrong, you need to know which step failed. So evaluating an FM-based application means scoring each component, not just the end result. Amazon Bedrock evaluates RAG systems, and Amazon Bedrock AgentCore Evaluations scores agents. The exam tests whether you can match a symptom to the step and metric that diagnose it.

Evaluate Each Step, Not Just the Answer

A wrong answer from a RAG application has two possible causes. Either the search step found the wrong passages, or the model wrote a poor answer from good passages. The fixes are completely different: better chunking, embeddings or source data for the first, and a better prompt or model for the second. AWS's guidance recommends evaluating every step of an application so that feedback lands on the part that needs work.

Retrieval Quality

Bedrock's retrieve-only RAG evaluation scores just the search step. It works on an Amazon Bedrock Knowledge Base or, if you supply its results, an external RAG system. Its built-in metrics are:

  • Context relevance: are the retrieved passages relevant to the question?
  • Context coverage: do the passages contain all the information in the expected answer? This needs ground truth in your test dataset.

Low scores here mean the problem is in finding information. Changing the model that writes the answers won't help.

Answer Quality

A retrieve-and-generate evaluation scores the passages and the written answer together. Its built-in metrics include correctness, completeness, helpfulness and logical coherence. Three more are especially useful for spotting RAG problems:

  • Faithfulness: does the answer stick to the retrieved passages, or add claims they don't support? Low faithfulness is the RAG version of a hallucination.
  • Citation precision: are the passages the answer cites actually the right ones?
  • Citation coverage: is the answer fully supported by its citations, with none missing?

There are also responsible-AI metrics for harmfulness, stereotyping and refusal. All of them are scored by an evaluator LLM.

Evaluating Agents

An agent can produce a polite final reply even after taking the wrong steps. So agent evaluation looks at the path as well as the outcome. AgentCore Evaluations uses LLM-as-a-judge to score agent traces, which record each step the agent took. It can score whether the agent achieved the user's goal across a whole session, and whether it chose the right tool at each step. AWS's FAQ names built-in evaluators such as goal success rate and tool selection accuracy, and you can add custom evaluators for your own criteria.

Workflows and Production Monitoring

Testing before launch isn't enough, because quality can drift once real users arrive or a prompt changes. AgentCore Evaluations runs in three ways:

  • Online evaluation continuously scores a sample of live sessions (for example, 10%) and shows trends on dashboards.
  • On-demand evaluation scores specific interactions you pick by ID, which is useful for investigating a complaint or testing a fix.
  • Batch evaluation scores many stored sessions in one job, for example to compare quality before and after a prompt change.

On the Exam

SymptomWhere to look
Retrieved passages are off-topicRetrieval: context relevance
Passages are fine, answer adds unsupported factsGeneration: faithfulness
Answer cites the wrong sourcesCitation precision
Agent's reply reads well but it used the wrong toolTool selection accuracy
Need ongoing scoring of live trafficOnline evaluation

The trap is fixing the wrong component, such as swapping the model when retrieval scores show the search step is failing.

Quiz me on this
A company's RAG assistant gives wrong answers about its HR policies. A retrieve-only evaluation shows low context relevance scores. Where should the team focus first?
  1. Swapping the generator model for a larger foundation model
  2. Raising the temperature so the model explores more answers
  3. Adding a guardrail that blocks off-topic responses
  4. ✓ The retrieval step, such as chunking and search
A RAG evaluation shows high context relevance and context coverage. However, reviewers notice that answers often include claims that appear in none of the retrieved passages. Which metric pinpoints this problem?
  1. Context coverage, because the passages are missing information
  2. ✓ Faithfulness, because answers add unsupported claims
  3. Context relevance, because the wrong documents are being retrieved
  4. Refusal, because the model is declining to answer questions
Which type of Amazon Bedrock RAG evaluation scores only what a knowledge base finds, without judging any generated answers?
  1. Retrieve and generate
  2. A human-based model evaluation
  3. ✓ Retrieve only
  4. An automatic summarization task
A travel company's agent handles booking changes. Its final replies read well, but logs show it often calls the hotel search tool when customers ask to change flights. Which evaluation measure targets this problem MOST directly?
  1. ✓ Tool selection accuracy
  2. ROUGE against a reference itinerary
  3. Helpfulness of the final reply alone
  4. Average tokens used per session
A retailer's shopping agent is live on Amazon Bedrock AgentCore. After recent prompt updates, quality seems to be slipping, and the team wants ongoing scores for a sample of real sessions, with trends shown over time. Which approach meets this need?
  1. On-demand evaluation of a few interactions picked by their trace IDs
  2. A Bedrock human-based model evaluation job with an internal team
  3. ✓ AgentCore online evaluation that samples a share of live sessions
  4. A Bedrock automatic evaluation job with a built-in dataset
Retrieve-only vs retrieve-and-generate RAG evaluation
Retrieve-only scores just the passages found. Retrieve-and-generate scores the passages and the written answer together.
Context relevance and context coverage
Relevance asks whether the retrieved passages fit the question. Coverage asks whether they hold all the information in the expected answer.
Faithfulness (RAG metric)
Whether an answer sticks to the retrieved passages instead of adding unsupported claims. Low faithfulness signals hallucination.
Citation precision and citation coverage
Precision asks whether the cited passages are the right ones. Coverage asks whether the answer is fully supported by its citations.
Amazon Bedrock AgentCore Evaluations
Scores agent traces with LLM-as-a-judge, such as goal success and tool selection accuracy, through online, on-demand or batch evaluation.