Data ExtractionDocument AILLM API

LLM Data Extraction: PDFs to Structured JSON (2026)

1 min read

The invoice pipeline ingested 40,000 documents last quarter. The reconciliation report looked perfect — until someone noticed that 3% of the vendor fields were silently wrong: an invoice number transposed, a tax line attached to the wrong row, a currency symbol dropped. Nobody noticed, because the extraction didn’t fail. It succeeded, with the wrong answer.

That’s the specific horror of document extraction: failures are silent. There’s no 500 error when a schema field comes back empty-but-validated, no exception when a table column shifts. LLM data extraction is one of the highest-value LLM workloads in 2026 — invoices, contracts, forms, claims — and one of the most benchmark-misleading, because the vendor benchmarks are written by the vendors, on documents that flatter their parsers.

This guide covers the extraction pipeline end to end — parse, schema, extract, validate — with the honest benchmark question (LLM-only vs parsing vendors vs open-source, on your documents), the schema design that prevents silent corruption, the PII handling that keeps it legal, and the cost model per 1,000 documents.

What Document Extraction Really Involves

Takeaway: extraction is a five-stage chain — parsing, understanding, schematizing, extracting, validating — and any stage can fail silently.

PDF → 1. Parse (layout, tables, OCR) → 2. Understand (vision or text)
        → 3. Schema (what fields, what types) → 4. Extract (LLM)
        → 5. Validate (types, required, confidence) → JSON

The chain is only as strong as its weakest stage, and the stages fail differently: parsing breaks on scanned and multi-column layouts, understanding breaks on rotated or watermarked content, schema breaks when fields are missing and nobody notices, and validation breaks when it’s not implemented at all. The pipeline’s job is to make every failure loud instead of silent.

Why Extraction Fails in Production

Takeaway: three failure classes — parse errors, schema drift, and missing validation — and only one of them shows up in logs.

  1. Parsing failures. Scanned documents without OCR, tables read as text soup, multi-column layouts flattened into garbage order. The parser determines the ceiling; the LLM can’t extract what the parser destroyed.
  2. Schema drift. Documents change — a new field appears, a vendor changes its template — and the schema stays fixed. Extractions silently return missing fields that pass validation because “missing” wasn’t a rule.
  3. Missing validation. No type checks, no required-field rules, no confidence scoring. The pipeline returns JSON, and JSON that looks right but isn’t is worse than no JSON: it feeds downstream systems with plausible lies.

The industry’s own comparisons — like pdfmux’s 2026 parser face-off — consistently show that parser choice moves accuracy more than model choice. That’s the first lesson: benchmark the parser on your documents before you benchmark the model.

The Neutral Test: LLM-Only vs Parsers vs Open Source

Takeaway: run the three-way test on your own corpus — text-simple documents, scanned mess, and tables — because the ranking inverts by document type.

The 2026 landscape has three families:

  • LLM-only — feed the document (text or image) straight to a multimodal model. Works on clean digital PDFs; degrades on layout complexity.
  • Parsing vendors — LlamaParse, Unstructured, and peers, who normalize layout before the LLM. The benchmark leaders on complex documents, at a per-page cost.
  • Open source — Docling, Marker, and the newer extraction-specialized models like NuExtract3 (an open-weight 4B vision-language model built for structured extraction, self-hostable at NuExtract’s pricing structure). Control and cost ceiling, at the price of ops.

The test that settles it: three document sets — clean digital, scanned, table-heavy — through all three families, measured on field-level accuracy (not “looks similar” matching), per-1,000-docs cost, and failure rate. The honest prediction: LLM-only wins the clean set, vendors win the scanned set, and open source wins the cost column at volume — and your corpus decides which column matters.

How to Assemble the Pipeline: Parse → Schema → Extract → Validate

Takeaway: schema design is the highest-leverage stage — a strict schema with validation turns extraction from a hope into a contract.

The core loop, provider-agnostic — on a unified chat endpoint:

import json
from openai import OpenAI

client = OpenAI()  # unified endpoint

SCHEMA = {  # the contract: required fields fail loudly
    "type": "object",
    "required": ["invoice_number", "vendor", "total"],
    "properties": {
        "invoice_number": {"type": "string"},
        "vendor": {"type": "string"},
        "total": {"type": "number"},
        "currency": {"type": "string", "enum": ["USD", "EUR", "GBP"]},
    },
}

def extract(page_text: str) -> dict:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={"type": "json_schema", "json_schema": {"name": "invoice", "schema": SCHEMA}},
        messages=[{"role": "user", "content": f"Extract the invoice data as JSON:\n{page_text}"}],
    )
    return json.loads(resp.choices[0].message.content)

def validate(doc: dict) -> tuple[bool, list[str]]:
    errors = []
    for field in SCHEMA["required"]:
        if field not in doc or doc[field] in (None, ""):
            errors.append(f"missing required field: {field}")
    if not isinstance(doc.get("total"), (int, float)):
        errors.append("total is not a number")
    return (not errors, errors)

Four rules that make this production-grade:

  1. Schema is a contract. Required fields, enums, and types — enforced by the structured-output mode of your model (the pattern documented in this series) — so “missing” becomes an error instead of an absent key.
  2. Parse before you prompt. Digital PDFs go to the LLM directly; scanned documents go through OCR or a multimodal model. The model catalog tells you which models accept images; the decision rule is “can the parser see the text?”
  3. Confidence and queues. Every extraction gets a confidence score; low-confidence rows go to a human-review queue instead of the database. The queue is what makes “silent failure” loud — and the retryable failures map to the error codes reference.
  4. PII at both ends. Redact before extraction where possible, scan after — personal data handling is a compliance requirement, not a pipeline nicety, and the standard data-privacy checklist applies.

How to Scale and Cut Cost

Takeaway: cost per 1,000 documents is a design number — tiering by document complexity and batching by volume routinely cut it by half or more.

The cost model: per-1,000-docs = parsing cost (if any) + model tokens × model rate. The levers:

  1. Tier by document complexity. Clean digital docs run on budget models; scanned or table-heavy docs run on the expensive path. Most pipelines are 70-80% clean, which means 70-80% of volume pays budget rates — custom routing makes the per-document decision mechanical.
  2. Batch the backlog. Historical document dumps are the perfect batch workload — delay-tolerant, high-volume, and the batch-discount pattern from this series applies the 50% off unchanged.
  3. Cache the template. Same vendor, same template = same prompt prefix. Stable prefixes hit cache pricing, which matters more on extraction than almost anything else, because templates repeat thousands of times.
  4. Build vs buy the parser. Vendors charge per page; open source charges per GPU-hour. The TCO analysis in this series shows the shape: managed wins below volume thresholds, open source wins above them, and the threshold depends on your ops appetite.

Common Mistakes That Silently Corrupt Data

Takeaway: four silent corruption patterns — every one produces plausible-looking wrong data.

  1. Schema without validation. A schema is a description, not a gate. Without type checks and required-field rules, “schema-bound” extraction still returns empty fields that pass as data.
  2. Skipping OCR on scanned docs. Text soup in, garbage JSON out — the parser failure is upstream of the model, and no model prompt fixes it.
  3. No model-version tracking. Model upgrades change extraction behavior; without pinned versions and a regression corpus, “the model got better” silently becomes “the fields shifted.” Version the model string and the schema together.
  4. All-frontier, all the time. Clean documents on the flagship model is the most expensive way to not improve accuracy — tier by complexity, and spend the savings on the review queue.

FAQ

Can I extract from PDFs with just an LLM?

Clean digital PDFs, yes. Scanned documents and complex layouts need OCR or a multimodal model first — the parser determines the ceiling, and the LLM extracts within it.

How do I measure extraction accuracy honestly?

Field-level matching (exact field, exact type, exact value) against a hand-labeled sample, per document type — plus type and required-field validation errors as a separate metric. “Looks similar” matching produces the 3%-silent-wrong problem this guide exists to prevent.

What does extraction cost per 1,000 documents?

Parsing cost (if any) plus model tokens at your tier — budget models on clean documents run far below flagship models on scanned ones. Tier by complexity and the average collapses; batch the backlog and it halves again.

How should I design the schema?

Required fields, enums, and types — enforced through structured output, with missing-required treated as an error. The schema is the contract between the documents and your database, and it deserves the same review as both.

Do I need a parsing vendor, or is open source enough?

Run the three-way test on your corpus. Vendors lead on complex layouts; open source (Docling, Marker, extraction-specialized models like NuExtract3) wins on cost and control at volume. The TCO threshold is real and measurable.

How do I handle PII in extraction?

Redact before extraction where possible, scan the output after, and keep retention minimal — the standard data-privacy requirements apply to extracted data exactly as they apply to the source documents.

Summary

LLM data extraction is a pipeline discipline, not a model property: parse deliberately, define the schema as a contract, extract within it, and validate everything so failures are loud. Benchmark the parser on your documents before the model, tier by document complexity, batch the backlog, and treat the review queue as a feature. Done right, extraction is the most reliable high-volume workload in the LLM stack; done wrong, it’s plausible wrong data feeding your database.

Run 100 pages through the pipeline before you argue with the benchmark. Get your TokSpan API key — $5 in free credits to run the sample (quickstart) — and see per-document costs on your own invoices.