Ways to Evaluate a Foundation Model

10 cards

Ways to Evaluate a Foundation Model

Before a company trusts a foundation model with real work, it needs evidence that the model does that work well. There are three broad ways to get that evidence: score the model automatically against test data, ask people to rate its answers, or use a second model as a judge. Amazon Bedrock offers all three as evaluation jobs. The exam expects you to pick the right approach from a scenario's constraints, such as speed, cost, subjectivity and who must sign off.

Why Evaluation Is Harder for Generative AI

A traditional classifier has one right label per example, so you simply count correct answers. A foundation model writes free text. Two very different summaries can both be good, and a fluent answer can still be wrong. That's why generative AI evaluation usually combines several methods rather than relying on a single score. AWS's guidance treats the evaluation setup itself as something to design, version and keep improving, like any other part of the application.

Benchmark and Golden Datasets

A benchmark dataset is a public, standard set of test prompts used to compare models on general skills, such as answering trivia or summarizing news. Bedrock's automatic jobs include several built-in datasets of this kind. Benchmarks are useful for a first comparison, but a model that tops a public leaderboard may still struggle with your company's documents and tasks.

So teams also build their own golden dataset: a set of realistic prompts with expert-checked answers, drawn from real user logs, examples written by domain experts and carefully reviewed synthetic cases. The quality of this dataset limits the quality of the whole evaluation. If its "correct" answers are wrong, every score built on it is misleading.

Automatic Evaluation

Automatic evaluation scores responses with computed metrics, with no people involved. In Bedrock, an automatic (programmatic) job runs a model against a built-in dataset or your own prompt dataset for a task such as text generation, summarization, question answering or classification. It reports metrics such as accuracy, robustness (whether small changes to the prompt change the answer) and toxicity. It's fast, cheap and repeatable, which makes it ideal for screening models early. But it can't judge subjective qualities such as tone.

Human-in-the-Loop Evaluation

In human-in-the-loop evaluation, people rate or compare responses. A Bedrock human-based job sends prompts and responses to a work team you choose, such as your own employees or industry experts. Workers rate each response using methods like thumbs up or down, a 1-5 scale, or picking the better of two responses. One job can compare up to two models. Human review is the best guide for subjective criteria and high-stakes decisions, but it's slow and expensive, and different reviewers can disagree.

A Judge Model in the Middle

LLM-as-a-judge uses a second model, the evaluator, to score a generator model's responses and explain each score. Bedrock offers built-in judge metrics such as correctness, helpfulness and harmfulness, and lets you define custom metrics. A judge scales far better than people, but it can carry its own biases. So AWS recommends checking a judge against human ratings and keeping people as the final gate for critical releases. The next lesson covers judge metrics in detail.

On the Exam

Scenario clue	Likely answer
Quick first comparison, standard tasks, no test data of your own	Automatic job with a built-in dataset
Subjective quality, brand tone, expert sign-off required	Human-based evaluation with your own work team
Thousands of responses, no reference answers, need explanations	LLM-as-a-judge
Public scores look good but real tasks fail	Build a golden dataset from your own work

A common trap is assuming a high public benchmark score proves a model fits your use case. Benchmarks measure general skill; only your own test set measures your task.

Quiz me on this

A marketing team is choosing between two foundation models for writing product announcements. The deciding factor is whether the text sounds like the company's brand, and policy says the company's own brand specialists must make that call. Which evaluation approach fits these requirements?

An automatic evaluation job that uses a built-in dataset
✓ A human-based evaluation job staffed by the brand specialists
A BLEU score against previously published announcements
Raising the temperature setting and reviewing which model varies more

A startup wants a fast first check of how accurate and how toxic three candidate models are at summarizing text. It has no test data of its own yet and no budget for reviewers. Which approach should it use?

A human-based evaluation job with a team of contractors
Fine-tuning each model and then comparing them in production
An A/B test that shows each model's summaries to real customers
✓ A Bedrock automatic evaluation job with a built-in dataset

An insurer picked the model with the best scores on public benchmark datasets. In a pilot, the model keeps misreading the insurer's claim letters. What should the insurer do FIRST to evaluate models properly for this use case?

✓ Build a test set of real claim letters checked by experts
Switch to whichever model currently leads a public leaderboard
Ask the model vendor for its latest published benchmark results
Increase the maximum output length so the model can explain more

A bank scores 50,000 chatbot responses every night with an LLM-as-a-judge setup. Its risk team now requires that a person approves quality before any new chatbot version reaches customers. Which approach meets this requirement MOST efficiently?

Have employees manually review all 50,000 responses every night instead of using the judge
Use the same model that powers the chatbot as the judge, since it knows the expected style best
✓ Keep the judge for broad nightly scoring and add human review as the final release gate
Stop the nightly scoring and rely on customer complaints after each release goes live

Which type of Amazon Bedrock evaluation job uses a second model to score another model's responses and explain each score?

A model evaluation job that uses human workers
✓ A model evaluation job that uses a judge model
A programmatic job that uses a built-in dataset
A batch inference job

Benchmark dataset

A ready-made set of test prompts, like Bedrock's built-in datasets, for a quick comparison. A custom dataset tailored to your use case tests real fit.

Golden evaluation dataset

Your own realistic prompts with expert-checked answers, drawn from real logs, expert examples and reviewed synthetic cases.

Automatic (programmatic) evaluation in Amazon Bedrock

Scores a model with computed metrics such as accuracy, robustness and toxicity, using built-in or custom prompt datasets. Fast and cheap, but can't judge tone.

Human-based evaluation in Amazon Bedrock

A work team you choose rates or compares responses (thumbs, scales, choices). Best for subjective criteria; slow and costly.

LLM-as-a-judge

A second model scores another model's responses and explains each score. Scales well, but should be checked against human ratings.