Bias and variance are the two main ways a model's predictions go wrong. Too much bias means the model is too simple and underfits. Too much variance means it has memorised its training data and overfits. This lesson shows how to read each from a scenario, what fixes each one, and why a model that looks accurate overall can still fail a whole group of people.
Two kinds of error
When a model makes mistakes on new data, the cause is usually one of two things. Bias, in the statistical sense, is error from assumptions that are too simple: the model can't capture the real pattern, so it's wrong in a consistent way. Variance is error from being too sensitive to the training data: the model has learned the details and noise of the examples it saw, so its predictions swing when the data changes.
Making a model more complex usually lowers bias but raises variance. The goal is the balance point where error on new data is lowest. Note that this statistical "bias" differs from bias in the fairness sense, which means outcomes that are skewed against a group. The two can be connected, as the last section shows.
High bias means underfitting
An underfit model does poorly on its training data and on new data, because it never learned the pattern in the first place. Typical causes are a model that is too simple, too few useful input features, or training that stopped too soon. Typical fixes are a more capable model, better features, or more training passes (epochs).
High variance means overfitting
An overfit model scores very well on its training data but much worse on new data. That gap between training and test results is the signature. Typical causes are a very complex model, too little training data, or training for too long.
Fixes include more training data, data augmentation, early stopping (ending training before the model starts learning noise), regularization (penalising unhelpful complexity) and simplifying the model. To detect overfitting before launch, teams use k-fold cross-validation: they split the data into several parts, train on all but one, test on the one held out, and rotate until every part has been the test set.
Effects on demographic groups
A single overall accuracy number can hide serious problems. Imagine a model that is 95% accurate overall, where one group makes up only 3% of the data. The model can be badly wrong for that group while the average still looks fine. Both kinds of error can cause this:
- Variance for small groups. With few examples of a group, the model's behaviour for that group rests on a handful of cases, noise included, so it generalises poorly to new members of the group.
- Bias for distinct groups. A model too simple to capture how one group's pattern differs will apply the majority's pattern to everyone.
The result is inaccuracy that falls unevenly, which becomes a fairness problem. The fix starts with measuring accuracy separately for each group, then usually means collecting more representative data.
On the exam
| Symptom | Diagnosis | Typical fix |
|---|---|---|
| Poor on training and new data | High bias, underfitting | More capable model, better features, train longer |
| Great on training data, poor on new data | High variance, overfitting | More data, augmentation, early stopping, regularization |
| Good overall, poor for one group | Under-representation | Per-group metrics, more data for that group |
Don't mix up the fixes: early stopping and regularization treat overfitting, and they make underfitting worse.