Don't Trust the Fluent Report

The first time my pipeline embarrassed me, the report looked perfect.

It was a weekly competitive brief: five pages, clean structure, confident prose, a chart caption, three bulleted takeaways. I skimmed it, thought this is good, and almost shipped it. Then I clicked through to one of the sources it cited, and the number wasn't there. Not misquoted. Not paraphrased. Invented. The report had pulled a statistic out of thin air and dressed it in a citation to make it look load-bearing.

That's the specific failure mode of generated reports, and it's worse than a typo. A typo you catch. A fluent, well-structured, confidently-cited claim slips past you, because the prose is doing its job: it reads like something a competent analyst wrote. Your eyes relax. You rubber-stamp it. And at scale, fifty reports a week, a hundred, eyeballing isn't a validation strategy. It's a prayer.

Why the obvious checks don't catch this

The first thing I reached for was rules. Does the report have all six sections? Are the citations formatted? Is it under the word limit? Any banned phrases?

Rules are great at shape and useless at truth. They'll confirm the report looks like a report. They will never tell you that section three contradicts section five, that a quoted figure is 40% off the source, or that a "source" was hallucinated entirely. Deterministic checks catch formatting. They can't catch a confident lie, because a confident lie is well-formatted by definition.

For unstructured text you need a check that understands the text. Which means, like it or not, another model.

What the checker actually is

An agentic checker is a second LLM call that reads the generated report and decides whether it holds up. That's the whole job. The popular name for the pattern is LLM-as-a-judge, and it's gone from research curiosity to standard practice in production agent systems.

The non-obvious part is what you feed it. A naive checker gets the report and asks is this good? and, because the report is fluent and the checker is the same kind of model that wrote it, says yes, looks great. Useless. The checker only earns its keep when it's grounded: the report, the source material the report was built from, and a rubric that says exactly what holds up means.

Now the checker isn't opining. It's verifying.

	Naive checker	Grounded checker
Input	the report	report + sources + rubric
Asks	"is this good?"	"does claim X trace to a source?"
Fails fluent lies	no	yes
What it returns	an opinion	a verdict, with reasons

Does every claim in section three trace to a source in the pile? Does the number in the chart match the number in the source? Do the takeaways contradict the body? Is every citation real? Those are questions with answers, and a grounded model can fail them.

Use a different model than the one that wrote the report if you can. Same family, same blind spots.

A concrete walkthrough

My pipeline ingests a pile of articles and filings each week and produces a research brief. The checker sits at the end of it:

def check(report, sources, rubric):
    issues = judge(report, sources, rubric)   # grounded LLM call
    return issues                             # [] = pass, else specific failures

result = check(brief, source_pile, RUBRIC)
if result.issues:
    hold_for_review(brief, result.issues)     # never auto-ship a fail
else:
    publish(brief)

Four lines that matter. The rubric is the part I rewrite constantly: every quantitative claim must appear verbatim or within rounding in a cited source; no two sections may contradict; all six required sections present; every citation resolves to a real document. That last line is the one that would have saved me from the embarrassing report.

Notice what the checker does not do: it doesn't rewrite. On a fail it holds the report and surfaces the specific issues to a human. I learned that one the hard way.

Where it breaks (the part I want you to read)

I'd be selling you something if I stopped at "add a checker." Here's where mine has failed:

• Same model, same blind spots. Early on my checker and writer were the same model. They made the same mistakes in the same direction, so the checker kept blessing garbage. Grounding helped, but switching the checker to a different family helped more. A judge that shares the defendant's worldview isn't a judge.
• Data quality, not model quality. The single biggest jump in my catch-rate came not from a better model but from cleaning the source pile and sharpening the rubric. A checker run against garbage sources with a vague rubric is just a confident second liar. The real work is curation and writing, not prompting.
• Goodhart, on autopilot. The checker optimizes the rubric you wrote, not the truth you meant. I once tightened a rubric until reports started passing every check while still being subtly misleading: technically sourced, technically consistent, and missing the point entirely. Sound familiar? It's the same trap as any loop. The faster you automate, the faster you arrive at a wrong answer that passes.
• Cost and latency. A second grounded pass roughly doubles your bill and adds latency to every report. You will not run it on everything. The honest choice is usually: gate the high-stakes output, sample the rest.
• The fix-loop drift. The temptation is to close the loop: checker fails, send it back, writer fixes, repeat until it passes. Done carelessly, the writer learns to satisfy the checker rather than be right, and you converge on check-passing fiction. If you auto-fix, make the checker's bar strict enough that gaming it is harder than fixing the report honestly.

How to start

You don't need an eval framework with a logo. Pick the one failure mode that has actually bitten you. Write a rubric for just that, three lines. Run a grounded checker over your last twenty reports. Count how often it caught the real failure versus how often it cried wolf. Tune until the signal is trustworthy. Only then think about automating the gate.

The point isn't to disappear overnight. It's to move, one failure mode at a time, from I skim it and hope to something cheaper than me catches the lie.

The real job

A generated report that reads well isn't a generated report that's right, and the gap between those two is exactly where the damage lives. The checker can't prove a report correct. That's not possible, and pretending otherwise is how you ship the confident lie. What it can do is make wrongness cheap to catch: a second pair of eyes that doesn't get tired, isn't impressed by fluent prose, and reads every report instead of skimming the first paragraph.

Get the rubric right and you can trust the pipeline a little more. Get it wrong and you've just built a faster way to rubber-stamp the same lie, now with a checkmark on it.