The expensive, lazy way to get an LLM to extract a structured invoice reliably is to run it three times and vote on the best result.
A better way is measuring how good the result is, and only climb to a more expensive pass if the result wasn't good enough.
The problem: invoices are messy
Restaurant invoices are a worst-case OCR target. When we tested with our initial customers, many invoices were faxed scans, photographed receipts, three-column layouts, handwritten totals, or vendors who scribbled notes in the bottom corner. A single Gemini pass nails maybe 70% of them cleanly. The other 30% need more work but you don't know which 30% in advance since every scenario is different.
So the ladder is adaptive. Each rung only fires if the rung below it produced something the scorer judged inadequate.
Before the ladder: getting text off the page
Before any LLM sees the invoice, the PDF has to become text, and we run two
different parsers depending on what the document actually is. Unstructured is
the fast path: a local, in-process parser that turns digital, text-based PDFs into
structured elements cheaply (the fast strategy). LlamaParse is the heavy
path that takes in faxed scans, photos, multi-column
layouts and returns clean text and tables (the hi_res and
ocr_only strategies).
The text these parsers produce is the "raw text layer" the chain-of-thought pass later reasons over.
Rung 0: a quality score
The whole ladder pivots on one function, score_quality. It starts at 100 and
subtracts for concrete, checkable defects:
score = 100
# negative amount: -5
# tax-rate / tax-amount mismatch: -5
# subtotal + tax ≠ total (math broken): -10
# each is_critical field still missing: -35Most of these are deterministic facts about the numbers, not opinions about the
text. Does subtotal + tax equal total? A broken
equation is a far stronger signal that extraction went wrong than any model-
reported confidence.
The -35 for a missing critical field
is critically, per-org. The score is computed against that org's field_specs — the set of
canonical fields they declared mandatory. A missing freight_amount is a
catastrophe for an org that tracks freight and a non-event for one that doesn't.
The same extraction of the same invoice scores differently for two different
orgs, because "good enough" is a tenant-specific question. The threshold the
ladder compares against (default 70) is therefore measuring against each org's
own definition of complete.
Rung 1 → 2: chain-of-thought
Pass 1 is a standard extraction against a schema that we score. The climb to Pass 2 is guarded:
quality = score_quality(dynamic_result, passes_used=1, field_specs=field_specs)
if (dynamic_result is not None
and quality.score < self._quality_threshold
and text):
retry_result = await self._run_pass2(...)
dynamic_result = confidence_weighted_merge(pass1, quality1, pass2, quality2)
quality = score_quality(dynamic_result, passes_used=2, field_specs=field_specs)Three conditions all have to hold. The score has to be below threshold — a clean
70%+ invoice never pays for Pass 2. And text has to exist, because Pass 2 is a
chain-of-thought reprompt that reasons over the raw parsed text layer; on a
multimodal-only run there's nothing for it to reason over, so we don't pretend.
The merge is where it gets nice. We don't take Pass 2 wholesale. confidence_ weighted_merge combines the two extractions field by field, keeping whichever
pass scored higher per field.
Voting throws away the loser entirely; weighted merge salvages the parts the loser
got right.
Rung 3: stop re-reading the whole invoice
If quality is still low, Pass 3 doesn't re-extract everything again. It computes exactly which fields are still failing and re-extracts only those:
failing_fields = _pass3_targets(dynamic_result, quality, field_specs)
patched = await self._run_pass3(dynamic_result, failing_fields, text)If after two passes you have a
solid extraction missing only po_number, you ask the model one focused question
about po_number instead of redoing the entire extraction process.
The top of the ladder is not an LLM call
After all the probabilistic passes, the final
rung is pure deterministic arithmetic — _apply_math_consistency_fix that
runs after all mutating passes, not just at the end:
# Fills MISSING fields only — never overwrites a value the model extracted.
# First match wins:
# tax_amount = round(subtotal * tax_rate / 100, 2)
# total_amount = round(subtotal + tax, 2)
# subtotal_amount = round(total - tax, 2)
# tax_amount = round(total - subtotal, 2)
# tax_rate = round(100 * tax / subtotal, 2) An invoice's amount fields are mostly just math.
If the model reliably read any two of {subtotal, tax, total},
the third isn't worth to make a Gemini call for extracting a
total. So before the ladder climbs to another
probabilistic rung, it fills every blank that algebra can fill, for free, with a
number that's correct by construction.
It only fills missing fields and never
overwrites a value the model actually extracted, so a real total
from the document is respected over a derived one. And the derived tax_rate is
only accepted if it lands in 0 < rate <= 100.
Why the schema is built at request time
Per-org scoring possible because of the extraction schema built per request from the org's field. specs:
dynamic_model = build_canonical_extraction_model(field_specs)The worker sends a list like field key, display name, section, data type, and a
bag of vendor-specific aliases (invoice_number → ["Invoice No.", "Inv #", "Bill Number"]) merged from system and org mappings. The OCR service compiles
that into a schema on the fly and a new org tracking a custom field needs zero code
changes.
One prompt library, not one giant prompt
The prompt is assembled the same way the schema is which is from parts per request. Instead of maintaining one enormous static string, we keep a library of discrete prompt components, each a single concern, and a factory that composes the ones a given pass actually needs:
@dataclass
class PromptComponents:
base: str = "" # role + core extraction task
requirements: str = ""
field_rules: str = ""
common_errors: str = ""
examples: str = ""
validation: str = ""
field_aliases: str = "" # built from the request's canonical_fields
chain_of_thought: str = "" # added only on Pass 2
format_hints: str = "" # added only on Pass 2
# create_invoice_prompt(...) → _compose() concatenates the non-empty pieces