A foundation model can score well on every quality metric and still fail the business. The summaries might be excellent, yet nobody saves any time; the chatbot might be helpful, yet customers still phone support. Business alignment means checking that an AI application moves the outcome it was built for. The exam names productivity, user engagement, task completion rate, user satisfaction and cost per interaction as the metrics to use. It expects you to pick the one that matches a scenario's stated goal.
From Model Scores to Business Outcomes
Engineering teams track technical metrics such as ROUGE or faithfulness. Leaders care about hours saved, customers served and money spent. AWS's guidance warns about the gap between the two and recommends making every technical metric traceable to a business outcome. Start from the goal the project was approved for, then choose the metric that would show it being met.
Productivity and Engagement
Productivity measures whether people get their work done faster or better. Examples are the time an analyst takes to draft a report, or how many cases a support agent closes per day. Compare before and after the AI tool, or between people who use it and people who don't.
User engagement measures how much people use the application: active users, return visits, sessions per week. Engagement shows whether people adopt the tool, but it isn't the goal itself. People can spend more time in an app because it's confusing. Treat engagement as a sign of adoption, then check it against the outcome you actually want.
Task Completion Rate
Task completion rate is the share of user sessions in which the user achieved what they came to do: the return was booked, the question was answered, the form was filed. It's often the clearest alignment metric for assistants and agents, because it measures the outcome rather than the wording. A low rate tells you to investigate where users give up, such as being passed to a human or leaving partway through.
User Satisfaction
User satisfaction captures how users feel about the experience. It's usually gathered through thumbs up or down ratings, short surveys or a customer satisfaction (CSAT) score. It picks up problems that offline tests miss: answers that are technically correct but too long, too slow, or in the wrong tone. When satisfaction falls while quality scores stay high, trust the users and look again at what you're testing.
Cost per Interaction
Cost per interaction is the average cost of serving one request or conversation, including tokens, infrastructure and any human follow-up. On its own it only tells you about spending. It becomes useful when paired with an outcome. A cheaper model that completes fewer tasks may cost more per successful task than an expensive one.
Building a KPI Dashboard
AWS recommends tracking return on investment continuously rather than calculating it once at launch, because both costs and value change over time. Its suggested ROI dashboard combines financial metrics, such as cost per interaction and total infrastructure spend, with value metrics, such as hours saved, revenue gained and improvements in CSAT score.
On the Exam
| Stated goal | Metric to pick |
|---|---|
| Save employees time | Productivity, such as time per task |
| Users finish what they came to do | Task completion rate |
| Users happy with the experience | User satisfaction or CSAT |
| Keep running costs in check | Cost per interaction, paired with an outcome |
| People adopting the tool | Engagement, but not as proof of value |
The trap is treating a quality score or an engagement number as proof of business value. Always check it against the goal.