CertBrain
Courses
Home
Blog
Sign in
← Back to decks
Sign in to save your progress, vote, and build your own decks.Sign in

Customization Cost Tradeoffs

There are several ways to make a foundation model better at your task, and they differ enormously in cost, effort and time. From cheapest to most expensive, they are: in-context learning (putting instructions and examples in the prompt), Retrieval Augmented Generation (RAG), fine-tuning, model…

10 cards
Study this deck

Customization Cost Tradeoffs

There are several ways to make a foundation model better at your task, and they differ enormously in cost, effort and time. From cheapest to most expensive, they are: in-context learning (putting instructions and examples in the prompt), Retrieval Augmented Generation (RAG), fine-tuning, model distillation, and pre-training a model from scratch. The exam often asks for the lowest-cost approach that still meets a requirement, so knowing this ladder, and what each step actually changes, is essential.

In-Context Learning

In-context learning means guiding the model through the prompt alone: clear instructions plus a few examples of the input and desired output. This is also called few-shot prompting. No training happens and the model's weights don't change, so there's no upfront cost and you can change behavior in minutes. The trade-off is that the instructions and examples add input tokens to every single request. At very high volumes, a long block of examples repeated on every call becomes a real running cost.

Retrieval Augmented Generation

RAG adds knowledge rather than behavior. Relevant passages from your documents are retrieved and added to each prompt. There's still no model training, but there are new running costs: creating embeddings for your documents, storing them in a vector database, and the extra input tokens from the retrieved passages on every request. RAG is the usual answer when the model needs company facts that change over time.

Fine-Tuning

Fine-tuning trains an existing model further on your labeled examples, changing its weights so it performs better at a specific task, style or format without long instructions in every prompt. Costs move upfront: in Amazon Bedrock you pay for training based on the tokens processed, a monthly storage charge for the custom model, and inference for the custom model. You also need good labeled data and must repeat the work when requirements change.

Model Distillation

Distillation transfers skill from a large, capable "teacher" model to a smaller, faster "student" model for one specific use case. In Amazon Bedrock, you provide prompts for your use case; Bedrock generates responses from the teacher and uses them to fine-tune the student. There's an upfront cost, including inference charges for the teacher's responses, but the result is a model that's cheaper and faster to run at high volume while staying close to the teacher's accuracy on that task.

Pre-Training

Pre-training builds a foundation model from scratch on a huge general dataset. It takes massive computing resources and months of time, and AWS notes that it can cost millions of dollars. For almost every business, using and adapting an existing foundation model is far cheaper and faster.

ApproachChanges the model?Upfront costOngoing cost driver
In-context learningNoNoneExtra prompt tokens per call
RAGNoLow (embeddings, vector store)Storage and retrieved tokens
Fine-tuningYesMedium (training)Custom model storage and inference
DistillationYes (student)Medium (teacher calls, training)Low: small model inference
Pre-trainingNew modelVery highHosting a large model

On the Exam

Start at the bottom of the ladder and stop at the first approach that meets the requirement. New behavior or format with a few examples: in-context learning. Current company facts: RAG. Consistent specialized behavior at scale: fine-tuning. Large-model quality at small-model cost: distillation. Pre-training is almost never the right answer.

Quiz me on this
A team needs a foundation model to return product descriptions in a fixed three-line format. It wants the cheapest approach, with no model training. What should it try FIRST?
  1. Pre-train a brand-new model on the company's product descriptions
  2. Fine-tune the model on thousands of examples
  3. Distill a large model into a smaller one
  4. ✓ Add instructions and a few examples to the prompt
An e-commerce firm sorts millions of customer emails a day with a large foundation model. Accuracy is excellent, but inference is too slow and too expensive. It wants a smaller model that keeps nearly the same accuracy on this one task. Which approach fits BEST?
  1. Use RAG to add more context to each email
  2. ✓ Distill the large model into a smaller student model
  3. Pre-train a new compact model from scratch on the emails
  4. Raise the large model's temperature setting
Which list orders these customization approaches from LOWEST to HIGHEST typical cost and effort?
  1. ✓ In-context learning, RAG, fine-tuning, pre-training
  2. RAG, in-context learning, pre-training, fine-tuning
  3. Fine-tuning, in-context learning, RAG, pre-training
  4. In-context learning, fine-tuning, pre-training, RAG
An insurer's claims tool sends 25 long worked examples in every prompt to get its specialized output. The task is stable and runs at very high volume, and input token costs are now the largest expense. Which change MOST directly removes the need for those examples?
  1. Add an Amazon Bedrock knowledge base for the examples
  2. Pre-train a new model on the insurer's claims
  3. ✓ Fine-tune the model on labeled examples of the task
  4. Raise the maximum length setting for responses
A company replaces a plain chatbot with a RAG design using an Amazon Bedrock knowledge base. Which new costs should it expect?
  1. Monthly custom model storage plus a new training job for every document update
  2. Months of large-scale compute to build a new model
  3. Teacher model inference to generate synthetic training data
  4. ✓ Embeddings, vector storage and extra input tokens per request
In-context learning
Steering a model with instructions and examples in the prompt. No training, but the examples add tokens to every call.
Fine-tuning costs in Amazon Bedrock
Training (tokens processed), monthly storage for the custom model, and inference for the custom model.
Model distillation
A large teacher model's responses fine-tune a smaller student model, giving cheaper, faster inference for one use case.
Pre-training
Building a foundation model from scratch. Takes months and can cost millions; rarely the right business choice.
Customization cost ladder
Prompt engineering (in-context learning), then RAG, then fine-tuning or continued pre-training, then building a custom model. Stop at the first rung that works.