Lab: Build a Data-Quality Gate

This is one of the hands-on labs that run alongside these posts. The scaffolding is low now: you get the plumbing and write the decision. The full lab is in lab-07-quality-gate.zip.

Before your first lab, do the one-time, once-per-account setup: run the zip’s preflight.sh to confirm your account is ready, then deploy the lab reaper, a standing backstop that auto-deletes any lab you forget to tear down after 24 hours.

The scenario

Before documents reach a knowledge base, something has to stop the bad ones. Raw support-ticket records land in s3://bucket/incoming/. Most are fine; some have an empty body, a nonsense priority, a missing email, or are not even valid JSON. Ingest those and retrieval quietly degrades, an answer grounded in a half-empty record is worse than no answer. A gate reads each record, decides, and routes it: good records to clean/, bad ones to quarantine/ with the reasons attached. Only clean/ goes on to be embedded.

What you’re given

An S3 bucket, a Lambda with least-privilege access to it, six seeded records (three good, three broken in different ways), and the handler’s reading, routing, and reporting. The gap is validate().

Your task

Write the rules that decide fitness. validate(record) returns a pair: True and an empty list when the record is fit, False and a reason string for every rule it breaks. Match the seeded samples with four checks: an id that is present and non-empty, a body of at least ten characters, a priority drawn from VALID_PRIORITIES, and an email with an @ in it. Collect the reasons as you go rather than bailing at the first failure, so a record that breaks two rules is quarantined with both.

A record that breaks a rule is quarantined with its reasons; a clean one passes. The malformed-JSON record is caught before validate even runs, because a parse failure is a different kind of bad.

Deploy and prove it

cd lab-07-quality-gate
./scripts/deploy.sh
./scripts/test.sh
./scripts/teardown.sh

You get {"clean": 3, "quarantined": 3}, three objects under each prefix, and T-5’s quarantine note naming its two failures. Nothing is deleted; the bad records are held with an explanation.

When you want the reference answer, deploy it with SRC=solution ./scripts/deploy.sh, or unfold it here:

def validate(record):
    reasons = []
    if not record.get("id"):
        reasons.append("missing id")
    body = record.get("body") or ""
    if len(body) < 10:
        reasons.append("body missing or too short")
    if record.get("priority") not in VALID_PRIORITIES:
        reasons.append("priority not one of high/medium/low")
    if "@" not in (record.get("email") or ""):
        reasons.append("email missing or malformed")
    return (len(reasons) == 0, reasons)

The ideas the exam cares about

• A gate routes, it does not delete. Quarantine with reasons keeps the data and makes the failure auditable, which is what an ingestion pipeline for a regulated system needs.
• The rules are the declarative part. Here they are a few lines of Python; AWS Glue Data Quality lets you express the same intent as DQDL rules against a Data Catalog table and get a quality score, and SageMaker Model Monitor runs the drift version against live inference data. The mechanism you built by hand is what those services manage.
• Malformed input is a distinct failure from a rule failure. Catch the parse error first; both belong in quarantine, but for different reasons, and conflating them hides real problems.
• The gate belongs before embedding. That is the cheapest place to stop bad data. Re-embedding a corpus you later discover was poisoned is the expensive fix, so the quality check is an ingestion-time concern, not a clean-up job.

What’s worth remembering

1. Validate data before it is embedded; the gate is the cheapest place to stop bad records, and re-embedding later is the costly fix.
2. Route, do not drop: quarantine failures with their reasons so nothing is lost and the pipeline is auditable.
3. Separate a parse failure (malformed input) from a rule failure (valid but unfit); both quarantine, for different reasons.
4. The rules are declarative intent; AWS Glue Data Quality (DQDL, with a score) is the managed version of what you wrote by hand.
5. SageMaker Model Monitor applies the same constraint idea to live inference data as drift detection, a different stage of the same lifecycle.
6. A knowledge base is only as trustworthy as the gate in front of it; quality is an ingestion property, not an afterthought.