Preparing a Dataset for Fine-Tuning

The situation

A team wants to fine-tune a foundation model on Amazon Bedrock so it drafts internal support replies in the house voice: a particular structure, a fixed sign-off, a calm tone, and a habit of quoting the ticket reference back to the customer. Prompt engineering got them most of the way, but the system prompt has swollen to hundreds of words of tone instructions and worked examples, it is expensive on every call, and the model still drifts out of voice about one reply in ten. Fine-tuning is the right tool for baking in behaviour that a prompt keeps having to re-teach.

They have three years of resolved tickets in a data warehouse: roughly forty thousand agent replies, of varying quality, written by dozens of people across several eras of the style guide. The instinct is to throw all forty thousand at the job and let scale sort it out. That instinct is the problem. A large fraction of those replies are off-voice, contradictory, or full of customer names, addresses, and card fragments. Some near-duplicate replies would land in both the training and the evaluation data. Feeding the model that heap teaches it the average of every era’s style, including the bad ones.

What they actually need is a small, deliberately curated set of examples that shows the model exactly the behaviour they want, in exactly the format the model expects, with nothing in it that leaks private data or inflates the evaluation score. The size of the raw pile is a distraction; the shape and cleanliness of the curated set is the whole job.

What actually matters

The first thing worth naming is what fine-tuning changes and what it does not. Fine-tuning on labelled examples teaches behaviour, format, and tone: how to respond, in what structure, with what voice. It does not reliably install fresh facts. If the model needs to know current pricing, this week’s policy, or a specific customer’s history, that knowledge belongs in retrieval at inference time, not baked into weights that go stale the moment they are trained. The clean mental split is that fine-tuning shapes how the model answers and retrieval supplies what it answers about; a serious support assistant usually wants both, fine-tuning for voice and structure, retrieval for the facts.

Given that, quality and consistency beat volume by a wide margin. A few hundred to a few thousand examples that are clean, representative, and formatted identically will produce a better tuned model than tens of thousands of noisy ones. The reason is that the model learns the regularities in the set, so every inconsistency is a lesson too. If half the examples end with a sign-off and half do not, the model learns that the sign-off is optional and produces it half the time. If the reference number is formatted three different ways across the set, it learns all three and picks unpredictably. Inconsistent labels and formatting do not average out to something reasonable; they teach the model to be inconsistent.

Consistency is not the same as sameness, though, and this is the tension to hold. The set has to be consistent in format and voice while still covering the real distribution of the task, including the edge cases. If every example is a simple happy-path cancellation, the model handles cancellations beautifully and falls apart on a billing dispute or an angry complaint. Coverage means the set spans the intents, tones, and awkward shapes the model will actually meet in production, each rendered in the one consistent house format. Representative of the real spread, uniform in presentation.

Then there is safety and hygiene, which is where careless datasets do real damage. Support text is dense with personal data: names, emails, addresses, phone numbers, card fragments, order histories. That has to be removed or redacted before anything is staged for training, because a model trained on it can regurgitate it, and because the raw data sitting in a bucket is a liability of its own. Duplicates and near-duplicates have to go, since they over-weight whatever pattern they repeat. And most quietly damaging of all is leakage between the training split and the validation split: if a reply or a close paraphrase appears in both, the validation numbers look wonderful and mean nothing, because the model is being tested on what it memorised.

Finally, the mechanics. Bedrock fine-tuning wants the data as JSONL, one example per line, with fields matching the target model’s expected schema, split into a training file and a separate validation file, both staged in Amazon S3 for the job to read. And the result has to be judged on a held-out set the model never saw during training, because training lossLoss curveThe plot of training error over time; the gap between the training and validation lines is how you spot memorising rather than learning. tells you the model fit the data, not that it does the job.

What we’ll filter on

1. Task fit, is the goal behaviour, format, and tone (fine-tuning’s job) rather than fresh factual knowledge (retrieval’s job)?
2. Example format, are the records labelled prompt-completion pairs in JSONL, matching the target model’s expected schema, with a train and a validation split?
3. Consistency, is every example formatted and labelled the same way, so the model learns one regular pattern rather than several conflicting ones?
4. Coverage, does the set span the real task distribution and its edge cases rather than repeating the happy path?
5. Safety and hygiene, is PII removed, are duplicates gone, and is the train/validation split free of leakage?
6. Evaluation, is there a held-out set to measure the tuned model against, separate from anything it trained on?

The dataset landscape

The pieces you assemble a fine-tuning set from, and what each one is for:

1. Labelled prompt-completion pairs. The core unit for fine-tuning that teaches a task: an input and the exact output you want the model to have produced. For the support case, the prompt is the ticket context and the completion is the ideal house-voice reply. The model learns to map the one onto the other. This is supervised fine-tuning, and it is what most cert scenarios mean by fine-tuning.

2. JSONL in the model’s schema. One JSON object per line, with the field names the target model expects. Different model families on Bedrock want different shapes, so the record schema is not universal; a set formatted for one model may need reshaping for another. The rule is to match the documented schema of the specific model you are tuning, not a generic template.

3. The train split. The bulk of the curated examples, the data the job actually learns from. This is where consistency and coverage have to be right, because everything in here is a lesson.

4. The validation split. A separate, smaller slice the training job reads during tuning to watch how the model generalises as it learns, so you can catch overfittingOverfittingWhen a model stops learning the general pattern in your data and starts memorising the individual examples. while the job runs. It must not overlap the training split. Bedrock can carve one out automatically if you do not supply it, but a curated split you control is better because you can guarantee no leakage.

5. The held-out evaluation set. Examples set aside before training and never shown to the job at all, kept to judge the finished model. This is distinct from the validation split, which the job sees during training; the evaluation set is the honest final exam. Keep it representative of production and never let a curated training example or its paraphrase drift into it.

6. Continued pre-training data. A different mechanism for a different goal. Continued pre-training feeds the model large volumes of unlabelled domain text (no prompt-completion pairs, just raw documents) to steep it in a domain’s vocabulary and patterns. It is unsupervised, it wants volume rather than hand-labelled precision, and it teaches familiarity rather than a specific input-output behaviour. Reach for it to adapt a model to a specialised corpus, not to teach it to answer support tickets in a house voice.

7. Preference or reward data. Pairs or rankings of better-versus-worse responses used by preference-tuning methods to push a model toward preferred outputs. Worth knowing it exists and that it is a distinct data shape from supervised prompt-completion pairs; it is not what a straightforward Bedrock supervised fine-tuning job consumes.

Side by side

Reading the table against the support goal: the job wants prompt-completion pairs in the model’s JSONL schema, split into train and validation with no overlap, plus a held-out set kept back for the final judgement. Continued pre-training is the wrong tool here because the goal is a specific reply behaviour, not general domain steeping; the model already knows English and support, it just needs to learn this team’s voice.

The picks in depth

Curating the pairs is where the real work sits, and it starts with throwing most of the raw data away. From forty thousand replies, the team should select a few hundred to a couple of thousand that genuinely exemplify the house voice, rewriting where a good reply has a formatting wart and dropping anything off-voice, contradictory, or thin. Every surviving example gets normalised to one format: the same structure, the same sign-off, the reference rendered one way. The prompt side has to be consistent too, carrying the same fields in the same order, because the model learns the shape of the input as much as the output. Curating down and normalising is worth more than any amount of extra volume.

Coverage is the counterweight that stops curation from collapsing into a monoculture. Before finalising, check the set against the real intent and tone distribution: cancellations, billing disputes, technical problems, complaints, simple thanks, the awkward multi-part message. Each should appear enough times to teach its pattern, each in the one house format. A set that is consistent but narrow tunes a model that is confident and wrong the moment it meets an intent it never saw.

Hygiene runs across the whole set before anything is staged. Redact or remove PII from both the prompt and completion sides, using pattern-based detection for the obvious identifiers and a review pass for the rest; a purpose-built service such as Amazon Comprehend can flag PII entities at scale, but the point is that no live personal data reaches the training bucket. De-duplicate, including near-duplicates that differ only in a name or a date, because repeated examples silently over-weight their pattern. Then split into train and validation and actively check for leakage across the boundary, deduplicating across the split and not just within each file; the fastest way to a validation score that lies is a reply and its paraphrase landing on opposite sides.

The format and staging are the mechanical finish. Write the examples as JSONL, one object per line, with field names matching the target model’s documented schema, since the shape differs by model family and a mismatched schema fails the job or trains on garbage. Put the training file and the validation file in S3, point the Bedrock fine-tuning job at them, and grant the job’s role read access to the bucket. Bedrock will carve out a validation split itself if you do not supply one, but a split you control is safer precisely because you can guarantee the no-leakage property the automatic carve cannot know about.

Evaluation closes the loop and has to be honest. Judge the tuned model on the held-out set that never touched training, comparing its replies against the original prompt-engineered baseline on the things that actually matter: voice adherence, structural correctness, the sign-off, the reference quote, and whether it invented any facts. A drop in training loss is not success; success is the tuned model beating the baseline on held-out examples on the metrics you set out to improve. If it does not, the answer is almost always in the data (coverage gaps, residual inconsistency, too few examples of a hard intent) rather than in the training settings.

A worked example: from raw tickets to a staged split

Start with a raw warehouse row, which is exactly what must not go into training:

Agent reply, ticket 44821:
"Hi Dana Whitfield, cancelled your Pro plan (card ending 4417,
dana.whitfield@example.com). - Marcus"

It is off-format (no house structure, no reference quoted back, an ad-hoc sign-off) and it is full of PII. Curated, normalised, and redacted, it becomes one clean prompt-completion pair. The completion carries the house voice; the prompt carries a consistent set of fields; the personal data is gone:

{"prompt": "Intent: cancellation\nPlan: Pro\nReference: 44821\nTone: neutral\nDraft a house-voice reply.", "completion": "Thanks for getting in touch. I've cancelled your Pro plan, effective at the end of your current billing period. Your reference for this is 44821. If there's anything else we can help with, just reply here.\n\nBest,\nThe Support Team"}

Every other surviving example gets the same treatment: same prompt fields, same reply structure, same sign-off, reference always rendered the same way, no live names or card fragments anywhere. Then the set is split, with a leakage check across the boundary so no reply and its paraphrase straddle the two files, and staged in S3 as two files the job reads:

The shape of the win is that the model no longer needs a hundreds-of-words tone preamble on every call, because the voice is in the weights; the prompt shrinks to the ticket context, latency and per-call cost drop, and drift falls because the behaviour was trained rather than requested. The facts the reply depends on still come from retrieval at inference time, because those change and fine-tuning would only freeze a stale copy.

What’s worth remembering

1. Fine-tuning teaches behaviour, format, and tone, not fresh facts; when the answer depends on knowledge that changes, pair the tuned model with retrieval rather than baking the facts into weights.
2. Quality and consistency beat volume; a few hundred to a few thousand clean, consistent examples usually out-teach a large noisy set.
3. Every inconsistency in the set is a lesson, so inconsistent formatting or labels teach the model to be inconsistent; normalise structure, sign-off, and field order across all examples.
4. Consistency is not sameness; the set must still cover the real task distribution and its edge cases, each rendered in the one house format.
5. Supervised fine-tuning wants labelled prompt-completion pairs in JSONL, with field names matching the target model’s expected schema, since the shape differs by model family.
6. Split into train and validation, and actively check for leakage across the boundary; a reply and its paraphrase on opposite sides makes the validation score meaningless.
7. Remove or redact PII and drop duplicates and near-duplicates before staging, because a model can regurgitate what it trained on and repeats over-weight their pattern.
8. Stage the training and validation files in S3 for the Bedrock job to read, and give the job’s role access to the bucket.
9. Continued pre-training is a different mechanism: large volumes of unlabelled domain text to steep a model in a corpus, not labelled pairs that teach a specific input-output behaviour.
10. Judge the tuned model on a held-out set it never saw, on the metrics you set out to improve; falling training loss shows it fit the data, not that it does the job.