The AI Use Case Envisioning session at Lodgewise, a residential property-management agency, picked two pilots. The first, maintenance triage, was a classifier. This is the second, and a deliberately different shape of AI so the team learns two things from one quarter: answering the dozen questions tenants ask over and over (notice periods, who pays for what, pets, bin day, breaking a lease early) from the lease and the tenant handbook.
Again, the envisioning pins are the spec:
- Capability: retrieve and answer. Find the relevant clause, answer from it, cite it. Retrieval-augmented generation, the pattern set out in choosing between chains, retrieval, and agents.
- Autonomy: suggest. The system informs the tenant; it never does anything. It doesn’t vary the lease, agree to a repair, or commit the agency. When it isn’t sure, it says so and hands the question to a property manager.
- Cost of being wrong: a tenant acting on bad information about their own tenancy. That makes two requirements non-negotiable: every answer is grounded in a real clause the tenant can see, and a tenant can only ever retrieve their own lease.
Why this isn’t “just a chatbot”
The temptation is to paste the leases into a prompt and wrap a chat box around a model. That builds the two failure modes straight in. A model answering from its training, ungrounded, will state a notice period that sounds plausible and is wrong for this state and this lease. And a single shared knowledge base will, the first time someone asks the right question, cheerfully quote one tenant’s lease to another.
So the real work here is two guarantees, and the model is almost the easy part:
- Isolation. A retrieval can only return chunks from the asking tenant’s own documents.
- Grounding. An answer is built only from retrieved clauses, cites them, and refuses when the clauses don’t cover the question.
Get those two right and you have a useful pilot. Get either wrong and you have an incident.
The knowledge base
Lodgewise already holds every lease as a PDF and a single tenant handbook. Ingest them into a Bedrock Knowledge Base, which handles chunking, embedding, and the vector store (the trade-offs in that choice are their own decision, see picking a vector store for Bedrock RAG). The one thing you must not skip is metadata: every chunk carries the tenancy_id it came from, and handbook chunks are tagged as shared.
{
"tenancy_id": "T-10488",
"doc_type": "lease",
"property": "12 Marri St",
"source_uri": "s3://lodgewise-leases/T-10488/lease.pdf"
}
That tenancy_id is the load-bearing field in the entire system. Without it, retrieval is a free-for-all across every lease the agency holds. With it, retrieval can be fenced to one tenancy at query time. The handbook, being the same for everyone, is tagged doc_type: handbook and left readable by all.
Retrieving the right clauses
When a tenant asks a question, retrieve against the knowledge base with a filter that pins results to their tenancy or the shared handbook, and nothing else:
import boto3
agent = boto3.client("bedrock-agent-runtime", region_name="ap-southeast-2")
KB_ID = "lodgewise-tenancy-docs"
def retrieve(question: str, tenancy_id: str) -> list[dict]:
only_this_tenant = {
"orAll": [
{"equals": {"key": "tenancy_id", "value": tenancy_id}},
{"equals": {"key": "doc_type", "value": "handbook"}},
]
}
resp = agent.retrieve(
knowledgeBaseId=KB_ID,
retrievalQuery={"text": question},
retrievalConfiguration={
"vectorSearchConfiguration": {
"numberOfResults": 6,
"filter": only_this_tenant,
}
},
)
return resp["retrievalResults"]
The tenancy_id comes from the authenticated session, never from the question text, so a tenant can’t ask their way into someone else’s lease. This filter is the isolation guarantee, and it deserves its own test: a fixture that asks tenant A’s session a question only answerable from tenant B’s lease, and asserts zero results. That test failing is a breach; treat it like one.
Answering only from the source
Now hand the retrieved clauses to a capable model (RAG answering is where you spend a little more on the model than classification did, see picking a Bedrock model for high-volume RAG) with an instruction that makes grounding non-optional:
brt = boto3.client("bedrock-runtime", region_name="ap-southeast-2")
MODEL_ID = "anthropic.claude-3-5-sonnet-20240620-v1:0"
SYSTEM = """You answer a tenant's question using ONLY the numbered
clauses provided. Rules:
- Use only the clauses. Do not use general knowledge about tenancy law.
- Cite the clause numbers you used, like [2], at the point you use them.
- If the clauses do not clearly answer the question, reply with exactly:
HANDOFF
and nothing else. Do not guess. A wrong answer about someone's lease
is worse than no answer.
- Keep it short, plain, and neutral. You inform; you never promise the
agency will do anything."""
def answer(question: str, clauses: list[dict]) -> str:
numbered = "\n\n".join(
f"[{i+1}] {c['content']['text']}" for i, c in enumerate(clauses)
)
user = f"Clauses:\n{numbered}\n\nTenant question: {question}"
resp = brt.converse(
modelId=MODEL_ID,
system=[{"text": SYSTEM}],
messages=[{"role": "user", "content": [{"text": user}]}],
inferenceConfig={"temperature": 0, "maxTokens": 500},
)
return resp["output"]["message"]["content"][0]["text"].strip()
Three things do the grounding. Only the clauses forbids the model from answering from training. Cite the clause numbers turns every answer into something checkable, by the tenant and by you. And the explicit HANDOFF token gives the model a clean way to decline, which is the behaviour you most want and the one a chatty model resists most. Temperature zero again: this is a lookup with prose around it, not a brainstorm.
Guardrails: grounding and refusal
The prompt asks for grounding; a guardrail enforces it. Attach a Bedrock guardrail with contextual grounding and relevance checks, the same control from configuring Bedrock guardrails, and a denied-topics policy so the system won’t be talked into giving legal advice or negotiating rent:
def answer(question: str, clauses: list[dict]) -> str:
numbered = "\n\n".join(
f"[{i+1}] {c['content']['text']}" for i, c in enumerate(clauses)
)
resp = brt.converse(
modelId=MODEL_ID,
system=[{"text": SYSTEM}],
messages=[{"role": "user",
"content": [{"text": f"Clauses:\n{numbered}\n\n"
f"Tenant question: {question}"}]}],
inferenceConfig={"temperature": 0, "maxTokens": 500},
guardrailConfig={"guardrailIdentifier": "lodgewise-tenant-qa",
"guardrailVersion": "2"},
)
if resp.get("stopReason") == "guardrail_intervened":
return "HANDOFF"
return resp["output"]["message"]["content"][0]["text"].strip()
The grounding check is a second opinion on whether the answer is actually supported by the retrieved clauses; when it isn’t, the guardrail intervenes and the system falls back to the same HANDOFF path as an unsure model. Two independent mechanisms, the prompt and the guardrail, both have to agree the answer is grounded before a tenant sees it. Belt and braces is the right posture when the cost of a confident wrong answer is a tenant standing on a misquoted clause.
When it doesn’t know
The suggest rung is the whole point, and it lives in what happens around the model. A grounded answer is shown to the tenant with its citations and a quiet line that a property manager is happy to confirm anything. A HANDOFF, whether from no retrieved clauses, an unsure model, or a guardrail intervention, never reaches the tenant as an answer at all:
def respond(question: str, tenancy_id: str) -> dict:
clauses = retrieve(question, tenancy_id)
if not clauses:
return {"action": "handoff", "reason": "no matching clauses"}
text = answer(question, clauses)
if text == "HANDOFF":
return {"action": "handoff", "reason": "not grounded / unsure"}
return {
"action": "answer",
"text": text,
"citations": [c["location"]["s3Location"]["uri"] for c in clauses],
}
A hand-off creates a normal task for the tenant’s property manager, with the question and a note that the system wasn’t confident, so the tenant gets a real person rather than a dead end or a guess. Far from being the failure case, the hand-off rate is a feature you watch: it tells you exactly which questions the documents don’t answer well, which is a backlog for improving the handbook, not just the model.
Measuring groundedness
A retrieve-and-answer pilot is measured differently from a classifier. Accuracy isn’t one number; the questions that matter are was the answer faithful to the clauses, did it cite the right ones, and did it hand off when it should have. Build an eval set of real tenant questions with a librarian’s answer key: the correct clause(s) for each, and a flag for the ones the documents genuinely can’t answer (which should hand off).
def evaluate(qa_set):
n = len(qa_set)
grounded = cited_right = handoff_right = 0
leaked = 0
for x in qa_set:
r = respond(x["question"], x["tenancy_id"])
if x["should_handoff"]:
handoff_right += r["action"] == "handoff"
continue
if r["action"] != "answer":
continue
grounded += judge_faithful(r["text"], x["gold_clauses"])
cited_right += set(r["citations"]) >= set(x["gold_citations"])
# The breach metric: did any answer draw on another tenancy?
leaked += any(c not in x["allowed_sources"] for c in r["citations"])
print(f"faithful: {grounded}/{n} cited right: {cited_right}/{n}")
print(f"handed off when it should: {handoff_right}")
print(f"ISOLATION LEAKS: {leaked}") # must be zero
Faithfulness can be scored by a second model acting as a judge over the answer and the gold clauses, the same eval-as-a-job machinery in evaluating LLM output with Bedrock eval jobs. But the number that can stop the launch on its own is ISOLATION LEAKS: any answer that cited a document outside the asking tenant’s own set is a data breach, and the only acceptable value is zero. A faithful, well-cited, helpful assistant that leaks one lease in a thousand does not ship.
What it bought
After a quarter, the repetitive questions, the ones whose answers were always sitting in the lease but never findable, are largely self-served, with citations the tenant can read for themselves. Property managers get the questions worth a human: the judgement calls, the disputes, the ones the documents don’t cover, surfaced cleanly by the hand-off path instead of buried under bin-day queries.
Two pilots, two capability families, two autonomy rungs, both kept deliberately low and both earning the right to climb only on measured evidence. That is what an envisioning session is supposed to produce: not one impressive demo, but a small portfolio of honest, measurable bets, each with the human in exactly the right place.