A model learns its view of the world from its training data, so a responsible AI system starts with a responsible dataset. This lesson covers the four dataset characteristics the exam names (inclusive, diverse, curated and balanced), what goes wrong when each one is missing, and how to spot the problem from a scenario's symptoms.
Why the dataset comes first
Machine learning models copy the patterns in their data, including the unfair ones. If the data under-represents a group of people, or carries old human prejudice in its labels, the model learns that and repeats it at scale. AWS's documentation gives a simple example: a model trained mostly on middle-aged people may be less accurate for younger and older people. Fixing a dataset before training is usually cheaper and more effective than patching a biased model afterwards. That's why dataset questions on the exam usually point to a data fix, not a model fix.
Inclusive and diverse data
An inclusive dataset contains examples from every group the system will serve: ages, regions, languages, accents, skin tones, income levels, device types and so on. A diverse dataset also covers the variety within those groups: different writing styles, lighting conditions, edge cases and unusual requests.
The test is simple. Compare the people and situations in the data with the people and situations the model will meet in production. Any group or situation that's missing or rare in the data is where the model will be weakest. A speech app trained only on studio recordings will struggle in noisy cars. A face-matching system trained mostly on one skin tone will make more mistakes on others.
Curated data sources
Curated data has been chosen, checked and cleaned on purpose, rather than collected in bulk and used as-is. Curation means knowing where each source came from and whether you have the right to use it. It also means removing duplicates, errors, offensive content and out-of-date material, and checking that labels are accurate.
Scraping large amounts of public web text is cheap, but it brings in whatever that text contains: slang, insults, stereotypes and wrong answers. A smaller, vetted set of sources often produces a safer and more accurate model than a larger, unvetted one.
Balanced datasets
A balanced dataset has enough examples of each outcome and each group for the model to learn all of them well. Imbalance comes in two common forms:
- Outcome imbalance. Fraud might be 1 transaction in 1,000, so a model can score 99.9% accuracy by never predicting fraud.
- Group imbalance. One region might supply 90% of the examples, so the model fits that region well and the others poorly.
Teams rebalance by collecting more examples of the rare cases, by re-weighting or resampling the data, or by data augmentation. Data augmentation creates modified copies of existing examples, such as flipped or rotated images. AWS notes that augmented data copies the bias of its source, so the source should be checked first.
On the exam
| Symptom in the scenario | Missing characteristic |
|---|---|
| Model works well for most users but poorly for one group | Inclusive or balanced data |
| Model fails on unusual but real situations | Diverse data |
| Model repeats offensive or wrong content | Curated sources |
| High accuracy, but it never catches the rare event | Balanced outcomes |
The trap is "add more data". More of the same skewed data keeps the skew. The fix is more of the right data.