10 document-extraction systems, actually measured

Corpus. 100 pages sampled with stratification (fixed seed, script published) from OmniDocBench (CVPR 2025), the standard benchmark Mistral, Baidu and DeepSeek report against —revision pinned (aa1ee96, v1.6 content, 1,651 pages). Strata: 12 handwritten, 10 degraded scans, 20 dense tables, 15 formula-heavy, 13 hard layouts, 15 figure-rich, 15 dense text; 57 pages contain English, 41 Chinese. The sample deliberately over-weights the *hard* subsets: absolute values run lower than the public leaderboard — what matters is the comparison under identical conditions.

Systems and serving. Cloud: Mistral OCR 4 and 3 (API). Self-hosted on cloud GPUs with each system's official recipe: Unlimited-OCR 3B, DeepSeek-OCR, PaddleOCR-VL 0.9B (official pipeline; the bare mode is reported separately as an ablation), olmOCR-2 7B. CPU: Docling and Marker on a local M-series machine (sequential). naktor-extractor ran in its production configuration.

Quality. Two independent layers. (1) The official OmniDocBench harness (quick_match, GT filtered to the 100 pages): text edit distance, table TEDS, formula distance and reading order. (2) A dual multimodal LLM judge that sees the image and the markdown and scores completeness, fidelity, structure and hallucination, with two judges from different families so no verdict rests on a single model. Layer (1) is strict about structure; layer (2) measures whether the *information* survives even when formatting differs. Divergences get investigated, not averaged.

Cost. Measured, not list: APIs via actual per-page billing; self-hosted via wall-clock of a warm concurrent 100-page batch × the cloud GPU hourly rate, at the maximum concurrency each serving supports. Cold start is measured and reported separately. Local CPU: time × an equivalent cloud container rate. The figure for naktor-extractor is its price, not its cost.

Adversarial verification. Every anomaly was attacked before publication, with the pages and diffs in hand. Raw artifacts exist locally for audit; they are not republished because the corpus copyright forbids it (see Limitations).

1. Mistral OCR 4's price premium doesn't buy quality. At $4/1,000 (2× OCR 3, 4× naktor-extractor), OCR 4 ties on text (0.090 vs 0.089), loses 15 TEDS points on tables and doubles the formula distance. Its real strengths are per-page API latency (2.2s p50) and the highest fidelity score from the second judge (83.6) — a fine product for interactive few-page flows; for bulk digitization its price can't be defended against the on-prem frontier.

2. The Unlimited-OCR case: the hype doesn't survive the harness. Baidu's launch claims state of the art on OmniDocBench. On our hard sample, using its official vLLM recipe, it loses to or ties with naktor-extractor on text, tables and formulas, winning clearly only on handwriting (0.22 vs 0.34) — real merit, far from «best at everything». Its architecture (3B MoE, constant KV) does make it cheap to self-host ($0.74/1,000).

3. Metrics lie too, unless audited: one artifact we almost published. OCR 3 looked better than OCR 4 on English (0.033 vs 0.073); 93% of the gap was ONE page —a table of contents OCR 4 rendered as a table while the GT annotates it as plain text; edit distance punishes the markup, not the reading. Without that page: a tie (0.034 vs 0.037). We publish it because a benchmark that hides its own false positives deserves no trust.

4. Serving is part of the model. PaddleOCR-VL goes from 0.197/TEDS 0.0 (bare VLM, generic prompt) to 0.066/TEDS 84.7 (official pipeline with layout + per-element prompts): the same checkpoint, 3× better text, and from unusable to second place on tables. Any comparison's numbers hinge on serving decisions that almost nobody documents.

5. The quality-price frontier is on-prem. No cloud API dominates naktor-extractor on quality-per-euro. And there is a second axis price doesn't capture: governance — everything sent to Mistral leaves your infrastructure, which for banking, healthcare or government can be a hard disqualifier before price is even discussed. naktor-extractor deploys on your infra, at $0.99/1,000, with top-table quality; and on documents with a native text layer (most of the real world) the cost drops much further — something this 100%-scanned corpus doesn't exercise.

6. (Post-benchmark) The benchmark audited us back. Analyzing page by page where we lost, the whole text gap concentrated in ~5 pages whose table titles, figure captions and sub-questions were missing from our markdown: they were extracted fine, but our rendering dropped them. Noise-free A/B re-render (same outputs, only the rendering changes): text 0.088 → 0.064 (Chinese −36%), reading order 0.176 → 0.152. The biggest return on building a reproducible benchmark was being able to audit our own product with it; the fix is committed with its tests.

• 100 pages. Enough for large separations (15 TEDS points), fragile for fine differences: averages over small subsets (14 formula pages) move ±0.05 on a single page.
• 100% scanned corpus (OmniDocBench ships images): on documents with a native text layer (most of the real world) the cost drops much further — something this corpus doesn't measure —, and classic CPU tools compete on their worst terrain.
• Deliberately hard-skewed sample: do not compare these absolutes with the official leaderboard (sanity check: on the English pages it scores 0.036 vs the published 0.033 — the harness is wired correctly).
• No formula CDM (edit distance only) and no Azure or AWS Textract (no credentials during execution; to be added in a revision).
• Marker was measured but excluded from the main table: our setup (sequential local CPU) yields a cost 100× outside its representative range. Publishing it as «Marker» would mislead; it will be re-evaluated in GPU mode.
• LLM judges are non-deterministic (±2 points between runs) and carry known biases; hence they are the secondary layer, and there are two from different families.
• The authors sell naktor-extractor. The antidote is reproducibility: for the public systems, the harness, page IDs, seeds and configs are published; anyone can re-derive their rows.

Dataset: OmniDocBench revision aa1ee96 (HuggingFace, free download). We publish: the seeded sampling script (corpus/sample.py, seed 20260703), the 100 page IDs, the runners for the public systems with exact configs, the official-harness driver, judge prompts, and this analysis. We do not republish the images or the extracted markdown (corpus copyright) — they regenerate from the scripts in ~1h and ~$15.