When the envisioning session at Lodgewise scored its candidates, pulling key terms out of lease PDFs landed in the parking lot rather than the pilot list. Not because it was hard to build, extraction is one of the cleaner shapes of AI, but because it had a data problem written on the note: there were thousands of leases and zero agreed answers to check an extractor against. The decision was right. An extractor you can’t measure is one you can’t trust to write into the system of record, and re-keying a wrong bond amount is worse than re-keying it by hand.
So this build does what the parking note asked first. It builds the thing that was missing, then the model.
The output you want
The terms the agency re-keys by hand are a short, structured list, which is exactly what makes this an extraction job rather than a generation one:
{
"rent_amount": 540.00,
"rent_frequency": "weekly",
"bond_amount": 2160.00,
"lease_start": "2026-02-01",
"lease_end": "2027-01-31",
"notice_period_days": 21,
"pets_allowed": true,
"managing_agent": "Lodgewise"
}
Closed, typed fields. A number is a number, a date is a date, pets_allowed is a boolean. That typing is what lets you validate the output instead of trusting it, the same discipline as the triage classifier: the model proposes, the schema checks, and anything that doesn’t fit the shape is a flag, not a silent write.
Building the gold set first
The parked prerequisite is unglamorous and unavoidable: a senior property manager reads a few hundred real leases and records the correct value of every field. That’s the gold set, and it’s the asset that turns “the demo looked right” into “we measure 98.4% on bond amount and 91% on notice period.” Without it you are shipping a feeling.
Two things make the labelling pay for itself. It is a one-time cost that licenses every future change to the prompt or the model, because each one is graded against the same answers. And the disagreements the labellers have with each other, is a “per week” rent written as the weekly or the monthly figure?, surface the genuine ambiguities in the documents before the model trips on them, which sharpens both the prompt and the agency’s own data standards.
Reading the PDF
Leases arrive as PDFs, some born-digital, many scanned. Rather than wrestle layout, hand each page to a multimodal model as an image, the same approach as the multi-modal assistant (for clean text-only documents, an OCR step first is cheaper; mixed estates are simpler to treat uniformly as images):
import json, boto3
brt = boto3.client("bedrock-runtime", region_name="ap-southeast-2")
MODEL_ID = "anthropic.claude-3-5-sonnet-20240620-v1:0"
SYSTEM = """Extract lease terms from the page images. Return one JSON
object with exactly these keys: rent_amount (number), rent_frequency
(one of weekly, fortnightly, monthly), bond_amount (number),
lease_start (YYYY-MM-DD), lease_end (YYYY-MM-DD), notice_period_days
(integer), pets_allowed (boolean), managing_agent (string).
For any field you cannot find or are unsure of, use null and lower the
matching confidence. Do not guess a number you cannot see. Also return
a "confidence" object mapping each field to your confidence 0..1.
Reply with one JSON object and nothing else."""
def extract(page_images: list[bytes]) -> dict:
content = [{"image": {"format": "png", "source": {"bytes": img}}}
for img in page_images]
content.append({"text": "Extract the lease terms."})
resp = brt.converse(
modelId=MODEL_ID,
system=[{"text": SYSTEM}],
messages=[{"role": "user", "content": content}],
inferenceConfig={"temperature": 0, "maxTokens": 800},
)
return json.loads(resp["output"]["message"]["content"][0]["text"])
Temperature zero, because extraction wants the value on the page, not a plausible one. The instruction to return null and a low confidence rather than guess is the single most important line: a missing field the human fills in is a minor cost, while a confidently wrong bond amount written silently into the ledger is the failure this whole design exists to avoid.
Validate, then trust
The model’s JSON is a proposal. Coerce and check every field against its type before anything downstream sees it, and treat a coercion failure as a low-confidence flag rather than an error to swallow:
from datetime import date
def validate(result: dict) -> dict:
conf = result.get("confidence", {})
clean, flags = {}, []
def take(key, caster):
try:
clean[key] = caster(result.get(key))
except (TypeError, ValueError):
clean[key] = None
if clean[key] is None or conf.get(key, 0) < 0.85:
flags.append(key)
take("rent_amount", float)
take("bond_amount", float)
take("notice_period_days", int)
take("lease_start", date.fromisoformat)
take("lease_end", date.fromisoformat)
clean["rent_frequency"] = (result.get("rent_frequency")
if result.get("rent_frequency") in {"weekly","fortnightly","monthly"} else None)
clean["pets_allowed"] = result.get("pets_allowed")
clean["_review_fields"] = flags
return clean
_review_fields is the autonomy rung made concrete. The envisioning session pinned extraction at draft for review, so the output of this pipeline is never a write to the system of record; it’s a pre-filled form with the uncertain fields highlighted for a person to confirm. A clean, high-confidence extraction is a few seconds of glancing and confirming; an uncertain one puts the human exactly where their attention is worth most.
Measuring it against the gold set
Now the parked prerequisite pays off. Run the extractor over the held-out gold leases and score per field, because the fields don’t carry equal risk and a single accuracy number would hide that:
def evaluate(gold):
fields = ["rent_amount", "bond_amount", "notice_period_days",
"lease_start", "lease_end", "rent_frequency", "pets_allowed"]
for f in fields:
right = sum(validate(extract(g["pages"])).get(f) == g[f] for g in gold)
print(f"{f}: {right/len(gold):.1%}")
Bond and rent amounts need to be near-perfect because they touch money; a notice period the model finds 90% of the time and flags the rest for review is fine, because the flags route to a human anyway. The per-field numbers tell you which fields to trust enough to pre-confirm and which always warrant a glance, and they set the bar that every later prompt tweak or model change has to clear, scored as a job the way evaluating LLM output sets out.
What it feeds
Once the gold set exists and the numbers hold, the parked idea is a pilot. Confirmed extractions flow into the system of record, which means they also flow into the tenant question-answering pilot as clean structured facts: “your bond is $2,160” can come from a typed field a human confirmed rather than a clause the model had to interpret on the spot. One parked idea, unblocked by building the boring data asset first, quietly makes another pilot better. That is usually how the parking lot pays out: not in one breakthrough, but in prerequisites met one at a time.