Exam Room · Advanced GenAI

Lab: Fine-Tune a Model and Read the Loss Curves

August 05, 2026 · 20 min read

Generative AI Development · part of The Exam Room

This is the second lab in the managed track of the hands-on labs. The first ten build things by hand against the model API; the managed track buys them instead. The theory post on tuning fine-tuning laid out the knobs and what a training-versus-validation loss curve looks like when a run goes wrong. Nothing on this blog has actually run one. The full lab is in lab-12-fine-tuning.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours. It also sweeps this lab’s out-of-band Bedrock resources, the custom model and any serving capacity, which a stack delete never sees.

The scenario

Greenbox support has a few hundred prompt-and-completion pairs that capture how their replies should sound, and a smaller set they kept back. Every reply does the same four things: opens with the subscriber’s first name, states the fact in one plain sentence, gives the next action, and closes with “Any trouble, just reply to this email.” Today that style is enforced by a long few-shot preamble on every call, which they pay for on every token of every request.

They want a model that answers that way without being told. The first run used the defaults and came out sounding like the base model. The second cranked the passes right up and came out parroting the training replies word for word. The model they want is somewhere in between, and the two curves the job writes to S3 are how they find it without launching twenty jobs and squinting at the output.

What you’re given

CloudFormation builds two things: a bucket that holds the datasets and receives the job output, and the IAM service role Amazon Bedrock assumes to read one and write the other. That role is worth a look, because it is where the confused-deputy conditions live: the trust policy names bedrock.amazonaws.com and pins aws:SourceAccount to your account with aws:SourceArn restricted to model customization jobs, so nothing else can borrow it.

There is deliberately no CloudFormation resource for the job. A customization job is a one-shot piece of work rather than a standing resource, so scripts/train.sh launches it and polls:

aws bedrock create-model-customization-job \
  --job-name "$JOB_NAME" \
  --custom-model-name "$CUSTOM_MODEL_NAME" \
  --role-arn "$ROLE_ARN" \
  --base-model-identifier amazon.nova-micro-v1:0 \
  --customization-type FINE_TUNING \
  --hyper-parameters '{"epochCount":"2","learningRate":"0.00001","learningRateWarmupSteps":"2"}' \
  --training-data-config '{"s3Uri":"s3://BUCKET/data/training.jsonl"}' \
  --validation-data-config '{"validators":[{"s3Uri":"s3://BUCKET/data/validation.jsonl"}]}' \
  --output-data-config '{"s3Uri":"s3://BUCKET/output/"}'

Three details in there earn their place. The hyperParameters values are strings, not numbers, because the API takes a string-to-string map. The validationDataConfig is optional and is the entire reason you get a second curve; leave it out and the job still succeeds, still reports a training loss, and tells you nothing about whether the model generalised. And the set of available HyperparameterA training setting you choose before the run (epochs, learning rate, batch size), as opposed to a weight the run learns. belongs to the base model rather than to fine-tuning: Amazon Nova Understanding models expose epochCount (1 to 5, default 2), learningRate (1e-6 to 1e-4, default 1e-5) and learningRateWarmupSteps (0 to 100, default 10), and that is the lot. There is no batchSize and no learningRateMultiplier on Nova. Those exist on other families: Cohere Command has batch size and early stopping; Meta Llama pins batch size at 1. Read the table for the model you picked before you plan a sweep, or you will go looking for a dial that is not there.

data.py generates the three JSONL files, 300 training pairs, 60 validation, 40 held back, all disjoint. Nova wants the conversational fine-tuning shape, one JSON object per line:

{"schemaVersion": "bedrock-conversation-2024",
 "system": [{"text": "You are a Greenbox support agent."}],
 "messages": [{"role": "user", "content": [{"text": "Hi, Rosa here. Can I move my delivery to Friday?"}]},
              {"role": "assistant", "content": [{"text": "Rosa, your delivery day is now Friday. ..."}]}]}

The older {"prompt": ..., "completion": ...} shape is a different format for a different family of models. Mixing them up fails the job hours after you launched it, which is the expensive way to learn the difference.

src/plot_curves.py already finds the CSVs, parses them, and does the coordinate maths. The gaps are the two functions that matter.

Lab 12 solution architecture A CloudFormation stack contains an S3 bucket holding the training and validation JSONL files and receiving the job output, plus the IAM service role Amazon Bedrock assumes. The model customization job, the custom model it produces, and the optional serving deployment are created by the API rather than by the stack: the job reads both datasets, writes its metrics CSVs back under the output prefix, and produces the custom model. On your machine, data.py writes the datasets and plot_curves.py draws the two loss curves from the downloaded CSVs. CloudFormation stack: genai-lab-12 Amazon Bedrock created by the API, not the stack Your machine data.py writes the three JSONL files; plot_curves.py draws the two curves deploy.sh uploads both files S3 data bucket SSE-S3, no public access data/training.jsonl 300 pairs data/validation.jsonl 60 pairs output/ the two metrics CSVs and the job artifacts fetch-metrics.sh brings the CSVs back down Customization role trusted by bedrock.amazonaws.com, pinned by aws:SourceAccount and aws:SourceArn the job reads them writes the metrics back Customization job Nova Micro base launched by train.sh produces Custom model billed monthly until deleted Served on demand optional Part B, deleted at the end

Your task

Turn two CSVs into a chart and a verdict. The job writes them under the output prefix:

model-customization-job-<id>/
    training_artifacts/step_wise_training_metrics.csv
    validation_artifacts/post_fine_tuning_validation/validation_metrics.csv

Both carry step_number, epoch_number and perplexity. The third column is training_loss in one file and validation_loss in the other. Write render_svg() to plot both against step number with a marker on the lowest validation point, and verdict() to say which of three pictures this is:

best_step, best_epoch, best_loss = min(validation, key=lambda r: r[2])
last_step, _, last_loss = validation[-1]

if (training[0][2] - training[-1][2]) / training[0][2] < 0.15:
    return "Underfitting. The model barely moved, so it will sound like the base."

rise = last_loss - best_loss
if best_step >= 0.85 * last_step or rise <= 0.02 * best_loss:
    return "Healthy. Both fell together and validation has not turned up."

return (f"Overfitting from step {best_step}. Validation climbed {rise:.4f} after "
        f"its low point while training loss kept falling. The best model this "
        f"run produced was at step {best_step}, in epoch {best_epoch}.")

Plain Python and hand-rolled SVG, no matplotlib, so it runs on a bare install and the output is a text file you can drop into a pull request.

Deploy and prove it

Costs first, because they are the reason this lab is split three ways.

The free path. Two complete sets of metrics CSVs ship with the lab, laid out exactly as Bedrock writes them: one healthy run, one that overfits. scripts/test.sh runs your code against both, needs no AWS account, and makes no AWS calls. If all you want is the skill, this is the whole lab.

cd lab-12-fine-tuning
./scripts/test.sh                # your version
SRC=solution ./scripts/test.sh   # the reference answer

Part A, paid, hours. A real customization job over 300 short records on Nova Micro is billed per token processed multiplied by the epoch count, so the training charge is small but not zero, and the custom model that results is billed for storage every month until you delete it. The job runs for hours rather than minutes. train.sh prints what it is about to spend and will not launch without a confirmation.

./scripts/deploy.sh              # bucket, role, datasets uploaded. Cents.
./scripts/train.sh               # confirms, launches, polls
./scripts/fetch-metrics.sh       # downloads the CSVs and plots them
EPOCH_COUNT=5 ./scripts/train.sh # rerun with a different shape

Part B, optional, gated. Using the model is a separate cost decision from making it. A custom model deployment serves it on demand, billed per token with no hourly charge, at rates that match the base model’s (the -custom-model SKUs in the AWS Price List API equal the base on-demand SKUs for Nova Micro and Lite; on the Bedrock pricing page they sit under Model customization, not the on-demand tables). For this comparison that is a fraction of a US cent; that path is available for Nova custom models in us-east-1 and Llama 3.3 70B in us-west-2. Everything else needs Provisioned ThroughputReserved Bedrock capacity bought by the hour for a fixed term, paid for whether traffic fills it or not. , which bills by the hour from creation to deletion whether you send it a token or not, at tens of US dollars an hour for a single model unit. serve-and-compare.sh supports both, prints the cost before it does anything, refuses to move until you type a confirmation, and deletes the serving capacity on exit, on Ctrl-C, and on error.

./scripts/serve-and-compare.sh                   # on-demand, per token
MODE=provisioned ./scripts/serve-and-compare.sh  # one no-commitment model unit
./scripts/teardown.sh

It runs held-out messages through the base model and the custom model side by side. Teardown deletes deployments and Provisioned Throughputs first, because those are the ones with a meter on them, then the custom model, then the bucket and the stack. A custom model is not a CloudFormation resource, so deleting the stack leaves it behind, still billing; aws bedrock delete-custom-model is what removes it, and teardown runs that for you unless you ask it not to.

Reading the two curves

The shipped samples are the exercise. Open both SVGs side by side and the difference is not subtle.

The healthy run is two epochs over the 300 examples. Training loss starts at 2.20 and lands at 0.59. Validation starts at 2.21, bottoms at 0.73 near the end of the run, and finishes at 0.75. Both curves fall together, flatten, and stay flattened. Nothing here says stop early, and nothing says another epoch would help much either, because a curve that has gone flat has stopped paying for the passes you are buying.

The overfitting run is the same 300 examples with the epoch count pushed to five. Training loss goes from 2.18 down to 0.07, which read alone looks like the better run: the model is fitting its training data almost perfectly. Validation tells the other half. It falls to 0.79 at step 320, in the third epoch, and then climbs steadily for the rest of the run to finish at 1.26. That divergence, one curve still falling while the other turns up, is the model switching from learning the general pattern to memorising the specific rows. The model in hand at the last step is worse than the model that existed at step 320, and the run has no memory of step 320 unless early stopping kept it.

So the correction for the parrot is fewer passes, not more forceful ones. Set the epoch count near the turning point, or turn on early stopping where the base model supports it and let the validation curve decide. The correction for a run where both curves stay high and flat is the opposite: more epochs, or a higher learning rate if more passes still will not move it. Same instrument, opposite readings, which is why guessing from the output alone gets expensive.

One thing the curves cannot tell you is whether the model is worth shipping. That is what Part B is for, and why the held-out set never goes near the job.

What’s worth remembering

  1. A model customization job is launched by API, not by CloudFormation, and the two things it needs from you are a service role Bedrock can assume and S3 locations for training, validation and output.
  2. validationDataConfig is optional and is the only reason you get a second curve; without it the job succeeds, reports a training loss, and says nothing about generalisation.
  3. The hyperParameters map takes string values, and which keys are valid is a property of the base model: Amazon Nova exposes epochCount, learningRate and learningRateWarmupSteps, with no batch size and no learning-rate multiplier.
  4. The metrics land as step_wise_training_metrics.csv under training_artifacts/ and validation_metrics.csv under validation_artifacts/post_fine_tuning_validation/, both carrying step number, epoch number and perplexity alongside the loss.
  5. Training loss falling is not evidence of a good model, because a model can always fit its own training data harder; the validation curve is what tells you when that progress stopped being real.
  6. Where validation loss bottoms out and turns up is the best model the run produced, and everything after it is a worse model with a better training loss.
  7. Making a custom model and serving it are separate charges: on-demand custom model deployment bills per token, Provisioned Throughput bills hourly from creation to deletion, and the custom model itself bills monthly for storage until you delete it explicitly.
  8. The loss curve says the run was healthy; a held-out comparison against the base model says the result is better, and those are different claims decided by different evidence.

These posts are LLM-aided. Backbone, original writing, and structure by Craig. Research and editing by Craig + LLM. Proof-reading by Craig.