A foundation model can be trained in four main ways: pre-training, fine-tuning, continued pre-training and distillation. Each one changes something different about the model and needs a different kind of data. For the exam, the skill is reading a scenario and naming the method that matches what the business wants to change.
Pre-training builds general ability
Pre-training creates a model from scratch. The model reads an enormous, broad collection of mostly unlabeled text, or images, and learns by repeatedly predicting what comes next. It changes every part of the model, because the model starts with no knowledge at all. The result is a general-purpose model with wide knowledge but no particular specialty. Few organizations do this themselves, because it takes huge datasets and a great deal of computing power.
Fine-tuning changes behavior on a task
Fine-tuning starts from a model that's already pre-trained and trains it further on a much smaller set of labeled examples. Each example pairs an input with the output you want, such as a support question and an ideal answer. The model's internal settings, called weights, are adjusted so its answers move toward the examples. Fine-tuning mainly changes how the model behaves: its format, tone, or skill at one kind of task. In Amazon Bedrock, this is supervised fine-tuning, and it produces a private custom copy of the model.
Continued pre-training adds domain knowledge
Continued pre-training also starts from a pre-trained model, but it keeps going with the pre-training method on new, unlabeled text from one field, such as medical research or a company's engineering manuals. Because no labels are needed, a business can use documents it already has. What changes is the model's knowledge and vocabulary, not its task skills. AWS's guidance for Amazon Nova notes that it works best with very large volumes of domain text, and that it should usually be followed by instruction tuning so the model can use what it learned.
Distillation makes a smaller model smarter
Distillation transfers skill from a large, capable model, the teacher, to a smaller, faster model, the student. In Amazon Bedrock Model Distillation, you supply example prompts for your use case. Bedrock has the teacher answer them, then fine-tunes the student on those answers. What changes is the student. It gets close to the teacher's quality on that one use case, while keeping its own speed and lower cost per request.
| Method | Starts from | Data | Mainly changes |
|---|---|---|---|
| Pre-training | Nothing | Huge, broad, unlabeled | Everything |
| Fine-tuning | Pre-trained model | Small, labeled | Task behavior |
| Continued pre-training | Pre-trained model | Large, unlabeled, one domain | Domain knowledge |
| Distillation | Small student model | Prompts, answered by a teacher | Student's quality |
On the exam
Match the clue to the method. "Labeled examples of the ideal answer" means fine-tuning. "Lots of unlabeled documents from our industry" means continued pre-training. "Same quality but faster and cheaper" means distillation. "Build our own model from nothing" means pre-training, which is almost never the right answer for a business with a budget. Cost comparisons between these methods, and against RAG, are covered in the Customization Cost Tradeoffs lesson.