A generative AI project succeeds when it moves a business number, not when its model scores well in a lab. This lesson covers the metrics businesses use to judge generative AI: return on investment (ROI), efficiency, conversion rate, average revenue per user (ARPU), customer lifetime value (CLV), accuracy and cross-domain performance. The exam gives you a goal and asks which metric shows whether it was met.
Start From the Business Goal
Pick the metric that matches what the project was meant to change. A shopping assistant built to increase sales should be judged on sales, not on how many people chatted with it. A tool built to save staff time should be judged on time saved. Technical scores, such as a model's accuracy on a test set, are useful inputs, but leadership asks about business results.
Efficiency and ROI
Efficiency metrics capture time and effort saved: hours of agent work avoided, tickets resolved without a person, documents drafted per day.
ROI compares the value a project delivers with what it costs. For generative AI, costs keep changing: you pay per token or per hour of capacity, usage grows or shrinks, and models get updated. So ROI shouldn't be calculated once at launch and forgotten. AWS guidance recommends tracking it continuously on a dashboard, setting costs such as cost per interaction and total infrastructure spend against value such as hours saved, revenue gained and customer satisfaction.
Revenue Metrics
- Conversion rate: the share of visitors or users who complete a goal, such as making a purchase or signing up. It suits shopping assistants and personalized offers.
- Average revenue per user (ARPU): total revenue in a period divided by the number of users. It shows whether features like personalized recommendations make each customer spend more.
- Customer lifetime value (CLV): the total revenue a business expects from a customer over the whole relationship. It suits goals about loyalty and retention, where the payoff comes over years.
Quality Metrics
Accuracy asks how often outputs are correct against an agreed standard. It matters most where wrong answers are costly. Customer satisfaction scores show how users feel about the experience.
Cross-domain performance asks how well one model performs across different subject areas or tasks, such as HR questions, legal summaries and code help. A model that performs well across many areas can serve several teams from one platform, which spreads its cost.
Picking the Right Metric
| The business wants to know... | Metric |
|---|---|
| Is it worth what we're spending? | ROI |
| Are we saving staff time? | Efficiency, such as hours saved |
| Are more visitors buying? | Conversion rate |
| Is each customer spending more? | ARPU |
| Are customers worth more over the years? | CLV |
| Can one model serve many teams? | Cross-domain performance |
On the Exam
Distractors are usually real metrics that answer a different question. Usage counts, token counts and technical scores like BLEU describe activity or model quality, not business value. Match the metric to the exact goal in the scenario, and remember that ROI weighs value against cost.