Getting the number out is the easy half.

Annual reports are written for people. The figures a team actually needs sit in tables that run across pages, with the units in a header three rows up and last year’s restated numbers in the next column.

Getting those figures into a structured form, consistently and across many reports, usually means keying them by hand or trusting an extraction nobody can easily check. The interesting part of this project isn’t pulling a number out of a PDF. It’s being able to say whether the number is right.

• Confidential inputs. Client documents and workbooks can’t be shown or reused, including here.
• Layouts vary. Between companies, and between years for the same company.
• The value is in the tables. That’s exactly where naive text extraction loses rows, merges columns or drops units.
• Calls cost money. Repeating a model or API call for something already answered is waste.
• Someone has to trust the output. It has to be checkable by a person, not just plausible.

The pipeline treats extraction and validation as one system. Text and table structure are extracted from the PDF first. Chunking is table-aware, so a table stays together with its header row and units instead of being cut mid-row by a character limit. Retrieval then narrows each target value down to the passages and tables likely to contain it.

Extracted values go into a fixed schema, and schema checks catch the cheap failures early: wrong type, missing field, implausible unit. Values that pass are reconciled against a reference workbook, which also serves as a fallback source when the PDF can’t be read reliably. What disagrees is surfaced on a dashboard for review rather than quietly overwritten.

I worked on the pipeline described above, in an industry ML engineering role: PDF-first extraction, table-aware chunking, structured outputs and schema checks, reconciliation against Excel references, retrieval, and the dashboards used to review results.

Because this was work for an employer and its clients, the employer, the client and the documents stay out of this write-up.

Three comparisons matter for a system like this, and each needs a defined baseline, a fixed sample and a clear counting rule before its number means anything:

Consistency

Does the same field come out the same way across reports and across re-runs?

Mismatches

How often does an extracted value disagree with the reference, and how is a “mismatch” counted?

Call volume

How many model or API calls does a run need, and how many of them repeat work already done?

I’ve deliberately kept specific improvement figures off this page. They’ll go here once the measurement method behind them (baseline, sample, counting rule) is written up well enough to publish alongside them.

The design target is simple to state: every value either agrees with the reference or shows up as a flagged mismatch for a person to resolve.

Silent disagreement is the failure this pipeline is built to prevent. A wrong number that looks right is worse than a gap. Quantified results are withheld for now, for the reason above.

Validation is part of the product, not a test suite next to it. The reconciliation step is what lets someone use the output without re-checking everything by hand.

Tables deserve their own handling. Most of the hard cases come from how a table is cut, not from how a model reads it.

Next

• Publish a fully synthetic, end-to-end example: an invented report page, the extracted record, and the check that flags it.
• Write up the measurement method so results can be shown with their baseline and sample size.