Some models let you see exactly how they reach an answer, while others give you an answer with no visible reasoning behind it. This lesson explains the difference between transparent, explainable models and opaque ones, how transparency differs from explainability, and how teams add explanations to models that can't explain themselves. The exam tests whether you can tell which kind of model, or which kind of explanation, a scenario needs.
Interpretable Models and Black Boxes
An interpretable model is one a person can follow from input to output. In a linear regression you can read how much each input pushes the result up or down. In a small decision tree you can trace the exact path of yes-or-no splits that led to a prediction. Nothing about the decision is hidden.
An opaque model, often called a black box, is one whose internal workings are too complex for a person to follow. A deep neural network spreads what it has learned across millions or billions of numbers, and a foundation model even more so. These models are often more accurate on hard problems such as images, speech and free text, but nobody can point to the rule that produced a particular output.
Transparency and Explainability Are Different Ideas
AWS treats these as two separate responsible AI dimensions.
- Explainability means having ways to understand the system's behavior: why it produced this output for this input.
- Transparency means giving people the information they need to make informed choices about using the system at all: what it's for, what it can't do, what data it was built on and how it was tested.
A model can be transparent without being explainable. A company might publish clear documentation about a black-box model's purpose and limits, yet still be unable to say why it made one specific decision. The reverse also happens: a simple model can be fully interpretable while its owner shares nothing about it with the people it affects.
Explaining an Opaque Model After the Fact
When a complex model is needed for accuracy but its decisions must still be explained, teams add post-hoc explanation methods. Feature-attribution techniques such as SHAP and LIME estimate how much each input contributed to one prediction, for example "debt-to-income ratio counted most against this application". These explanations are approximations of the model's behavior, not a view of its true inner logic, so AWS recommends testing them with real users to make sure they help rather than mislead.
Explanations for Generative AI
Generative AI needs different techniques, because there are no tidy input features to rank. Common approaches include:
- Source attribution: a retrieval augmented generation (RAG) system shows which documents an answer came from. Amazon Bedrock Knowledge Bases returns citations that link each part of an answer to the source chunks it used.
- Confidence indicators: showing how certain the system is, so users know when to double-check.
- Agent traces: recording each step an agent took, such as which tool it called and what came back. Amazon Bedrock AgentCore Observability shows this step-by-step path for agents.
On the Exam
| Scenario clue | Likely answer |
|---|---|
| Regulators need the exact reason for each decision on tabular data | An inherently interpretable model, such as a decision tree or linear model |
| A complex model must stay, but each decision still needs reasons | Post-hoc feature attribution on that model |
| Users must check where a generated answer came from | Source citations from the RAG system |
| People need to decide whether to use the system at all | Transparency documentation about purpose and limits |
A common trap is treating transparency and explainability as the same thing. Published documentation is transparency; a reason for one specific output is explainability.