We had an LLM pipeline that needed to read messy input and produce a structured, hierarchical output — specifically, a deep, structured understanding of insurance data pulled from PDFs and other source documents. Nothing exotic at first: instruction plus data, straight into the model. It worked, but not well enough. Accuracy sat below where we needed it, and for a while every fix felt like it was chasing the same ceiling.

Iteration one: just get the prompt right

The first move wasn’t clever. We tightened the instruction — clearer structure, explicit steps, less ambiguity about what “correct” meant in edge cases. And instead of eyeballing whether a change helped, we ran everything through an evaluation framework: every LM call got logged in full — the exact input and output, how long it took, how many tokens it used — so every prompt variant became a real, measurable experiment instead of a guess.

This was slow, unglamorous work, and it paid off — steadily, for a while. Then it stopped paying off. We’d rewrite a section, run the eval, and the number wouldn’t move. That’s a plateau, not a bad prompt. No amount of further wordsmithing was going to get us past it — we needed a different lever.

Iteration two: give it something to point at

The next idea was few-shot examples, pulled dynamically per request from a knowledge base via graph-RAG. Whatever case the model was looking at, it got a small set of genuinely similar prior cases to reference alongside the instruction.

The eval score jumped — past the plateau, clean win. Then a few outputs came back that felt off. Values in the response that didn’t trace to anything in the actual input — plausible, well-formed, just borrowed from the wrong place: the retrieved example, not the case being processed. The thing that made retrieval useful — pulling in genuinely close examples — was the same thing that made this dangerous. There’s no token that says ignore the numbers here, just copy the shape.

Iteration three: split generation from verification

The fix wasn’t a smarter generation prompt. It was a second stage. Generation stays exactly as it was in iteration two — instruction, data, and example still go in, and an intermediate output still comes out, leak risk and all. What changes is that the intermediate output no longer counts as final. It has to pass through a validator first, and the validator’s input is deliberately narrow: the original instruction and data for that case, plus the intermediate output it needs to check — and nothing else. It never receives the example.

That’s the whole trick. If the intermediate output contains a value that only exists because it was copied from the example, the validator has no way to confirm it — it isn’t in the data, and the validator has never seen the example to recognize it as “correct but from the wrong place.” It gets flagged. Because the validator is blind to the example by construction, this isn’t a smarter check catching a subtle bug — it’s an architectural constraint making that specific bug impossible to slip past, at this stage.

Concretely, this validator is an LLM-as-judge: given the intermediate output plus the real source for that case — both the source document’s free-text content and the structured data already extracted from it — it fact-checks every claim against the source and returns a hallucination score. Its output isn’t a rewritten version of the analysis; it’s a highlight layer on top of it — every claim marked grounded or not, with a reason.

Two details made this work in practice, not just in theory:

• It has to check both source layers. Checking only the free-text content produces false positives — plenty of correct claims are grounded in the structured data instead, and a text-only check would flag them as unsupported.
• It needs a real taxonomy, not a binary match. We built a 10-rule hallucination taxonomy alongside an explicit list of non-hallucinations — normalization, synonym use, grounded advice, and omissions all needed to be ruled out by name, or the judge would over-flag legitimate phrasing as fabrication. Findings are weighted (major issues count far more than minor ones), and the judge has to list every finding individually with a tally that matches the score — so the score is auditable, not just a number.

We validated this against a real customer-reported hallucination case from earlier triage — a case we already knew the ground-truth answer for. The validator caught exactly what the customer had originally reported: two fabricated coverage-absence claims (the analysis claimed two coverages weren’t present in the documents when they were) and one recommendation for a coverage already included. Same failure, now caught automatically instead of by a customer noticing first.

What three iterations actually taught us

Each iteration was a genuinely different kind of lever, not a bigger version of the last one:

• Prompt clarity has a ceiling. Once you hit it, more editing is just noise.
• Retrieval-augmented context breaks through that ceiling, but it imports a new risk proportional to how good the retrieval is.
• A separate stage that’s structurally blind to the risky input — not just a smarter prompt — is how you contain a risk you can’t prompt your way out of.

None of this would have been visible without eval discipline from day one. The evaluation framework is what caught the plateau in iteration one, and it’s the same infrastructure that let us validate the judge against a known real case in iteration three, instead of trusting it on vibes.

Where it stands now

The validator currently runs offline, against recorded traces — that part’s done. The next steps are already mapped out: run it against live traffic in near-real-time, get its output classified into a trusted accuracy metric, then close the loop entirely — catch and correct a hallucination before the output ships, and eventually fix the underlying templates so certain classes of hallucination stop occurring at all. Detection first, then prevention at the root.