CertBrain
Courses
Home
Blog
Sign in
← Back to decks
Sign in to save your progress, vote, and build your own decks.Sign in

Model and Business Metrics

A model can be judged in two ways: how well it predicts, and whether it's worth what it costs. Model performance metrics such as accuracy, precision, recall and F1 score measure the first. Business metrics such as cost per user and return on investment measure the second. The exam asks you to pick…

9 cards
Study this deck

Model and Business Metrics

A model can be judged in two ways: how well it predicts, and whether it's worth what it costs. Model performance metrics such as accuracy, precision, recall and F1 score measure the first. Business metrics such as cost per user and return on investment measure the second. The exam asks you to pick the right metric for a scenario, which usually comes down to which mistake is more costly.

Right and Wrong Predictions

Take a model that flags card transactions as fraud. Each prediction lands in one of four boxes, often drawn as a confusion matrix:

  • True positive: flagged as fraud, and it really was fraud.
  • False positive: flagged as fraud, but it was a genuine purchase (a false alarm).
  • False negative: not flagged, but it was fraud (a miss).
  • True negative: not flagged, and it was genuine.

Every metric below is a different way of counting these boxes.

Accuracy, Precision and Recall

  • Accuracy is the share of all predictions that were right.
  • Precision asks: of everything the model flagged, how much was really fraud? High precision means few false alarms.
  • Recall asks: of all the real fraud, how much did the model catch? High recall means few misses.

Worked example. Out of 1,000 transactions, 20 are fraud. The model flags 25, and 15 of those are real fraud. Precision is 15 out of 25, which is 60%. Recall is 15 out of 20, which is 75%. Accuracy is 98.5%: 15 caught frauds plus 970 genuine purchases left alone, out of 1,000. But a lazy model that never flags anything would still score 98% accuracy while catching no fraud at all. That's why accuracy misleads when one outcome is rare.

F1 Score

Precision and recall usually pull against each other. Flag more transactions and you catch more fraud (recall rises), but raise more false alarms (precision falls). The F1 score combines both into one number that is only high when both are high. Use it when you need a single figure and both kinds of mistake matter.

Business Metrics

A model with great scores can still be a bad investment. Business metrics connect the model to the goal:

  • Cost per user: what it costs to serve each user, including inference and infrastructure.
  • Development costs: the people, data and computing time spent building it.
  • Customer feedback: whether users find the results helpful, through ratings, complaints or surveys.
  • Return on investment (ROI): the value the model creates, such as fraud losses avoided or hours saved, compared with what it costs.

On the Exam

If the costly mistake is...Prioritise
Missing a real case (disease, fraud, safety defect)Recall
A false alarm (wrongly blocking a customer, spam filter hiding real email)Precision
Both, and you need one numberF1 score
Outcomes are balanced and equally costlyAccuracy can be fine

The classic trap is choosing accuracy for a rare-event problem. If 99% of cases are "normal", a model can be 99% accurate and useless.

Quiz me on this
A hospital uses a model to flag scans that may show a serious disease, so a doctor can review them. Missing a real case is far worse than sending a healthy patient's scan for an extra review. Which metric should the hospital prioritise?
  1. Precision
  2. Accuracy
  3. ✓ Recall
  4. Cost per user
An email provider's spam filter sometimes moves important customer emails into the spam folder, and users are complaining. The team wants to measure and reduce exactly this kind of error. Which metric should it focus on improving?
  1. ✓ Precision
  2. Recall
  3. Development costs
  4. Training data size
A payments company tests a fraud model on 10,000 transactions, of which only 50 are fraudulent. The vendor highlights that the model is 99.5% accurate. Why should the company be cautious about this claim?
  1. Accuracy can only be measured on data the model was trained on
  2. Accuracy above 99% always proves that the model has overfitted its training data
  3. Accuracy is a business metric rather than a performance metric
  4. ✓ A model that never flags fraud would also score about 99.5% accuracy
A support team's AI assistant has strong precision and recall in testing. Leadership now asks whether the project is worth continuing, compared with what it costs to build and run. Which metric answers that question MOST directly?
  1. F1 score
  2. ✓ Return on investment (ROI)
  3. Recall
  4. Accuracy on a held-out test set
Precision
Of everything the model flagged as positive, the share that really was positive. High precision means few false alarms.
Recall
Of all the real positive cases, the share the model caught. High recall means few misses.
F1 score
A single score combining precision and recall; it is only high when both are high.
Why can accuracy mislead?
When one outcome is rare, a model that always predicts the common outcome scores high accuracy while missing every rare case.
Return on investment (ROI) for an ML model
The value a model creates (losses avoided, hours saved, revenue gained) compared with what it costs to build and run.