Preparing Data for Fine-Tuning

A fine-tuned model can only be as good as the data it learns from. Preparing that data means curating it, labeling it, making sure it represents the real job, getting the size right, and governing who can see and use it. This lesson covers each step, plus reinforcement learning from human feedback…

10 cards

Study this deck

Preparing Data for Fine-Tuning

Curation and quality

Data curation means choosing, cleaning and checking examples before training. That includes removing duplicates, fixing errors, cutting irrelevant or outdated material, and making the examples consistent. A model learns from mistakes just as faithfully as from good examples, so a wrong answer in the training set teaches the model to be wrong. AWS's guidance for continued pre-training calls data quality the most important factor in whether training succeeds.

Labeling and format

Supervised fine-tuning needs labeled examples: inputs paired with the outputs you want. In Amazon Bedrock, they go into JSON Lines files, one example per line, typically a prompt and its ideal completion, stored in Amazon S3. The labels must be accurate and consistent. If two reviewers would write very different ideal answers to the same prompt, the model gets mixed signals. A separate validation set, held back from training, lets you check the model on examples it hasn't seen.

Size and representativeness

More data isn't automatically better, and each model sets minimum and maximum dataset sizes for customization. What matters most is that the examples are representative: they should look like the real requests the model will face, across every type of user, topic and edge case. If a support dataset only contains billing questions, the model will be weak on technical ones. Under-representing some groups of people can also build bias into the model.

Governance

Training data is often sensitive, so it needs the same care as any important business data. Teams should know where each dataset came from and whether they have the right to use it. They should restrict access with least-privilege permissions, encrypt it, and remove personal information the model doesn't need. AWS's security guidance warns that a model trained on personal data can leak it later. It suggests tools like Amazon Macie to find sensitive data in Amazon S3, and customer managed keys in AWS KMS to encrypt training data and the resulting custom model.

Reinforcement learning from human feedback

RLHF uses human judgment to make a model's answers more helpful, accurate and safe. People write good example answers, and then rank several model responses to the same prompt from best to worst. Those rankings train a separate reward model that predicts what humans would prefer. The language model is then trained to produce answers the reward model scores highly. RLHF handles qualities that are hard to define with rules, such as tone and helpfulness, but it needs a lot of careful human effort.

On the exam

Expect scenarios where a fine-tuned model behaves badly and you must find the data cause. Wrong answers mean poor curation or labels. Weakness on certain topics or groups means data that isn't representative. Leaked personal details point to missing governance. When the goal is answers people prefer, especially on tone or helpfulness, the clue is human ranking, which means RLHF.

Quiz me on this

A bank fine-tuned a model on support conversations, but it often gives outdated fee amounts. The team finds that many training examples were copied from old transcripts that quote fees the bank has since changed. Which data preparation step was missed?

Converting the data to JSON Lines format
Increasing the number of training examples
Removing personal information from the transcripts
✓ Curating the data to remove outdated examples

A model developer wants its chatbot's answers to match what people find most helpful and polite. It plans to have reviewers rank several responses to each prompt from best to worst, then use those rankings to steer training. What is this approach called?

✓ Reinforcement learning from human feedback (RLHF)
Continued pre-training on domain text
Domain adaptation fine-tuning on documents
Retrieval Augmented Generation (RAG) with a knowledge base

A telecom company fine-tuned a support model using only examples from its English-speaking urban customers. After launch, the model performs poorly for customers in rural areas, whose questions are mostly about coverage and installation. Which data problem best explains this?

The dataset was too large for the chosen model
✓ The training data wasn't representative of all users
The labels weren't stored as JSON Lines in Amazon S3
The data wasn't encrypted with a customer managed key

A healthcare company plans to fine-tune a model on patient support transcripts stored in Amazon S3. Its compliance team worries the model could later repeat patients' personal details. Which step should come first?

✓ Use Amazon Macie to find personal data and remove what isn't needed
Switch to a larger base model with a longer context window
Turn on Amazon Bedrock model invocation logging for the custom model
Add more transcripts so personal details become a smaller share

Data curation

Selecting and cleaning training examples, for example removing duplicates and filtering out low-quality content before training.

Fine-tuning data format in Amazon Bedrock

JSON Lines files in Amazon S3, one example per line, usually a prompt and its ideal completion.

Representative training data

Examples that mirror the real requests and users the model will face, across topics, groups and edge cases.

Validation dataset

Examples set aside and never used for training, so you can measure how well the model performs on data it hasn't seen.

RLHF

Humans rank model responses, a reward model learns those preferences, and the language model is trained to score well.

Governing fine-tuning data

Know its origin and usage rights, use least-privilege access, encrypt it, and remove unneeded personal information.