AnyDoc and pdf-inspector: Why Firecrawl Open-Sourced Their Rust Document Parsing Stack

Firecrawl published a post on August 6 announcing two open-source Rust libraries: pdf-inspector for PDFs, and AnyDoc for 14 other document formats. They’re separate repos with deliberately separate scopes, but share the same shape — from-scratch Rust, local execution, no API key, no system dependencies, markdown out. Both already ship inside Firecrawl’s hosted /parse and /scrape endpoints, so any team using those APIs is using these libraries in production without realizing it.

What caught my attention wasn’t the libraries themselves. It was the routing decision pdf-inspector makes before any extraction happens, and how that one decision changes the economics of document processing at scale.

The Wrong Bet Most PDF Pipelines Make

Every PDF processing pipeline I’ve seen makes the same default assumption: treat every page as if it might be scanned, so route everything through OCR. That assumption is catastrophic on the long tail of real documents. A 200-page financial report with 150 pages of native text and 50 scanned appendix pages doesn’t need OCR on 75% of its content. But the default routing sends every page through the same GPU-heavy pipeline anyway — slow, expensive, and often less accurate than the native text that was sitting in the PDF the whole time.

pdf-inspector reads a PDF’s internal structure in milliseconds — font encodings, text operators, image coverage — without rendering anything. It classifies each page as text-based or scanned, extracts text directly from the text-based pages with reading order preserved, and hands back page references for the scanned ones with the reason flagged. The Firecrawl team reports their hosted PDF engine, Fire-PDF, runs 3.5x to 5x faster than the previous pipeline because for that 200-page report, 150 pages skip the GPU entirely.

That routing layer is the actual contribution. The text extraction is straightforward — what’s hard is deciding per page whether OCR is even needed. Most libraries don’t bother to ask the question.

AnyDoc and the 14-Format Problem

The non-PDF side of the problem is uglier. Office documents alone span .docx, .doc, .docm, .xlsx, .xls, .xlsm, .pptx, .ppt, .rtf, .odt, .ods, .odp, plus .epub and .csv. No single library covers all of them. Firecrawl’s own benchmark against six other converters — mammoth, LibreOffice, and a handful I hadn’t heard of — shows the next-best alternative, LibreOffice, covering 12 of 14. Everything else is a subset, which is exactly why teams end up stitching multiple libraries together, each with its own dependencies, output shape, and failure modes.

AnyDoc handles all 14 in one binary. The API is genuinely one call:

let markdown = anydoc::to_markdown("contract.docx")?;

Node.js:

import { toMarkdown } from "@firecrawl/anydoc";
const markdown = await toMarkdown("contract.docx");

Python:

import anydoc
markdown = anydoc.to_markdown("contract.docx")

Same shape across all three. No API key. No external service. The Python wheel is pip install firecrawl-anydoc. For text-based PDFs, AnyDoc embeds pdf-inspector so the single call covers PDFs alongside the 14 formats.

The benchmark numbers are aggressive: 4.4ms median per document across all 14 formats, versus 52ms to 1,130ms for the alternatives. Quality was LLM-judged (Claude Sonnet 5) blind against ground truth across completeness, structure, formatting, and cleanliness. AnyDoc scored 81 overall versus 70 for the next-best option, and leads on every individual format.

One caveat worth flagging: the benchmark corpus is Firecrawl’s own, and each tool’s score only averages the formats it actually supports. So mammoth’s 70 covers .docx alone, while AnyDoc’s 81 spans 14. The per-format comparison is the fair one — and on that axis, AnyDoc still leads.

Why This Matters for AI Pipelines

Document parsing is one of those quietly expensive parts of an AI pipeline that nobody budgets for until it’s already eating 30% of the request budget. A RAG system that ingests user uploads — contracts, reports, pitch decks, exported spreadsheets — spends more time on format conversion than embedding in a lot of cases. The conversion layer tends to be a chain of brittle scripts, each handling one format with different output shapes that need to be normalized downstream.

There’s also a privacy angle here that’s easy to miss. Most hosted document parsing services require uploading the document to a third-party API. For legal contracts, financial reports, or anything with PII, that upload is a hard blocker for a lot of teams. pdf-inspector and AnyDoc both run entirely locally. The fact that Firecrawl’s hosted endpoints use these same libraries means the local-first approach isn’t a compromise — it’s the production path, with the hosted API as an optional layer on top.

I’ve been thinking about whether this routing-first pattern shows up elsewhere in AI infra. It does — vector search has the same shape, where naive pipelines do brute-force nearest neighbor on every query when an HNSW index would have answered in 10% of the latency. Inference batching has it too. The pattern is: don’t optimize the heavy compute path until you’ve classified the input correctly. Most pipelines skip the classification step because it feels redundant, then pay for it on every single request.

How the Routing Classifier Actually Works

The interesting engineering in pdf-inspector is the per-page classifier. A PDF page is text-based when its content stream contains real text operators (Tj, TJ, BT/ET) that map glyph IDs to font encodings the parser can decode. It’s scanned when the content stream is essentially “draw this image at this rectangle” with no recoverable text underneath. The classifier reads the content stream, doesn’t render anything, and outputs a binary decision plus the reason.

What makes this fast is that it skips the expensive parts entirely. No PDF-to-image rasterization, no OCR model load, no GPU warm-up. For a 200-page report where 150 pages are pure text, those 150 pages get the native extraction path which reads text operators in microseconds per page. The remaining 50 pages — the scanned appendix, the embedded charts, the photographed signature pages — get handed off to OCR with their page references intact. The caller decides what OCR backend to use.

There’s a subtlety worth noting: text-based PDFs are not all the same. Some have proper Unicode mappings in their font tables; others use custom CIDs or 8-bit encodings that map glyph IDs to scrambled byte sequences. Naive extraction on the second type produces garbage. pdf-inspector’s native extraction handles the encoding mess internally — that’s part of why the library is from-scratch Rust rather than a wrapper around an existing C library. The wrappers inherit the encoding bugs.

For mixed documents, the routing output looks roughly like:

{
  "pages": [
    {"index": 0, "classification": "text", "text": "..."},
    {"index": 1, "classification": "image", "reason": "no_text_operators", "image_ref": "page_1.png"},
    {"index": 2, "classification": "text", "text": "..."}
  ]
}

The caller iterates the array, writes the text pages directly, and runs OCR on the image references. That separation is the actual API contract — and it’s what makes the 3.5x to 5x speedup possible, because the optimization is at the routing layer rather than inside the OCR model itself.

Integration Trade-offs

The from-scratch Rust choice has real consequences. Both libraries compile to a single static binary with no runtime dependencies, which means the Python wheel, the Node.js native module, and the WASM build all share the same code path. There’s no version drift between the language bindings — when pdf-inspector fixes a CID encoding bug, every binding gets the fix.

The trade-off is that compilation is heavier. Building pdf-inspector from source pulls in mupdf-adjacent dependencies for some of the lower-level PDF primitives, and Rust toolchain setup is non-trivial on machines that don’t already have it. The prebuilt wheels solve this for most users, but anyone pinning to an unusual architecture (older ARM, musl libc) will need to compile themselves. Firecrawl publishes a reproducible-results branch specifically for benchmarking — worth using if you’re comparing against your own PDFs and want to control the build environment.

The other trade-off is format coverage depth. AnyDoc covers 14 formats at the conversion-to-markdown level, but markdown is a lossy intermediate. Complex Excel spreadsheets with merged cells, conditional formatting, and pivot tables flatten poorly regardless of the converter. PowerPoint presentations with custom animations and embedded video lose those features entirely. The library gets you to “clean text in markdown structure” reliably, which is what 95% of AI pipelines actually need — but if you’re trying to reconstruct the original document’s visual layout, markdown is the wrong target.

For AI workloads specifically, that’s fine. Embedding models and LLMs want text. Markdown gives them structure (headers, lists, tables) without the noise of font metadata and embedded styles. The AnyDoc output is closer to what a human would type up from the document than what the document actually looks like rendered.

What I’d Want to See Next

The two libraries are deliberately separate. pdf-inspector is the dedicated PDF engine; AnyDoc handles everything else and embeds pdf-inspector for text-based PDFs. That separation is good — a single mega-library would have a worse failure surface — but it leaves a few questions open.

The OCR handoff is one. pdf-inspector flags scanned pages with the reason and page reference, but the actual OCR step is the consumer’s problem. For mixed documents, you still need to wire up a vision pipeline — PaddleOCR, Surya, Tesseract, whatever your stack prefers. A reference integration showing how Firecrawl themselves do this in Fire-PDF would close the loop.

Reproducible benchmarks against non-Firecrawl corpora would help too. The benchmark numbers are compelling, but the corpus selection matters enormously for document parsing. I’d want to see how AnyDoc performs on adversarial inputs — password-protected Office files, embedded macros, scanned-with-headers PDFs that fool the classifier, ancient .doc files from Office 97. The realistic long tail is where document parsing usually breaks, and a single-corpus benchmark won’t surface those failure modes.

The 13k stars on pdf-inspector and rapid adoption of AnyDoc suggest the open-source shape is the right call. Document parsing is plumbing, and plumbing only gets good when it’s a stable, well-maintained dependency rather than a custom integration per project. Firecrawl betting on that — releasing the production-grade libraries that already power their hosted endpoints — is a smarter play than burying them as proprietary moats.

Both repos are on GitHub: pdf-inspector and AnyDoc. If you’re already stitching four or five libraries together to cover the formats your users upload, the integration cost is low enough to be worth a weekend experiment.