The Concurrency Collapse: When llama.cpp Outperforms vLLM at Scale

Every benchmark comparison between llama.cpp and vLLM leads with the same headline: vLLM is 2–3x faster on throughput. That number is real. But it comes from batched inference — running dozens of requests through the model simultaneously to saturate the GPU. It’s the right benchmark for a chat API serving thousands of concurrent users.

It’s the completely wrong mental model for an AI agent.

The Problem With Batch Performance

A standalone coding agent processes one task at a time. It reads a repo, writes a test, runs it, reads the output, writes another test. Each step is sequential. The batch size is 1. And at batch_size=1, the performance landscape looks very different.

The data is hiding in plain sight across several GitHub discussions and benchmark threads. At low concurrency (1–16 parallel requests), the gap between vLLM and llama.cpp narrows dramatically. Some configurations show llama.cpp actually pulling ahead on per-request latency — the metric that matters for an agent waiting for its next thought.

Here’s why: PagedAttention, vLLM’s signature optimization, excels when you can pre-load the KV cache for many concurrent sequences. At batch_size=1, you have one sequence. The KV cache fits in VRAM regardless. The paging overhead — memory fragmentation from variable-length attention spans, the bookkeeping — becomes pure overhead with nothing to offset it.

llama.cpp, meanwhile, has spent years optimizing the single-sequence path. The GGUF format’s memory-mapped tensors, the-KQ normalization, the matrix-vector kernels tuned for consumer GPUs — these shine when you’re running one long-running thought through the model.

The VRAM Equation Changes Too

vLLM requires enough VRAM to hold the full model in FP16 — no quantization, no memory mapping. For a 70B model, that’s roughly 140GB of VRAM. You need a node with multiple A100s or a single monolithic accelerator.

llama.cpp with a Q4_K_M quantized 70B model fits in 40–48GB. A single RTX 4090 or even an RTX 3090 can run it. The per-token cost is lower because the hardware floor is lower.

For an individual developer or a team running an agent stack, this isn’t a minor detail. It’s the difference between running the model on your workstation and renting a cloud GPU node at $3–5/hour.

What This Means for Agent Architecture

The dominant production pattern right now is: vLLM on a GPU cluster for the heavy lifting, with smaller models (7–13B) running locally via llama.cpp for fast tools and retrieval. This is a sensible division of labor. But it has a subtle implication for agent design.

When you split the model stack, you’re making a latency vs. cost tradeoff at the architectural level — not just a model size decision. The local llama.cpp path is low-latency but capability-limited. The remote vLLM path is high-capability but higher-latency and cost-prohibitive for high-frequency calls.

For an agent that needs to make dozens of tool calls per task, the local path wins on round-trip time even if each individual call is weaker. A 200ms local call with a 7B model beats a 2s remote call with a 70B model if the task only needs 7B capability.

This is the “right-sizing” thesis that Jan, LM Studio, and Ollama are all betting on. And it’s why llama.cpp’s single-sequence performance matters more than the batched benchmarks suggest.

The Hidden Concurrency in Multi-Agent Systems

Here’s where it gets interesting for multi-agent pipelines like ACO System. An ACO pipeline has a Planner, Architect, Developer, and QA agent — each running its own model calls, but running them sequentially within each stage. Across the whole pipeline, you have concurrency, but within each agent’s critical path, you’re still serial.

The implication: if you’re building a multi-agent pipeline where individual agents use local inference, you care deeply about single-sequence latency. The pipeline throughput is gated by the slowest serial step. Reducing per-step latency with llama.cpp at every stage compounds across the pipeline.

vLLM’s batched optimization helps when you’re serving many agents simultaneously — a platform concern. llama.cpp’s single-sequence optimization helps when you’re running one agent through a complex task — a developer concern.

The Benchmark That Actually Matters

The thought experiment I keep coming back to: take a real agent task — something like “fix the failing tests in this PR” — and measure time-to-first-action. Not tokens per second. Not requests per second. Time from starting the task to the agent’s first meaningful action.

On an M3 Max MacBook Pro with 128GB unified memory, llama.cpp on a 13B Q4 model handles this in under 5 seconds. On a cloud vLLM instance with an A100, you’re adding network round-trip latency on top of whatever compute time remains.

For interactive agentic workflows, this matters more than batched throughput. The future of AI agent infrastructure might be a fleet of well-tuned llama.cpp instances rather than a handful of vLLM monsters — depending on whose problem you’re solving.

The Tradeoffs Are Genuine

None of this means llama.cpp is universally better. For a startup serving 10,000 concurrent users with long context windows and complex retrieval, vLLM is the right answer. PagedAttention’s KV cache management is genuinely superior for high-utilization scenarios.

But for the individual engineer building, debugging, and iterating on agentic systems — the audience this blog writes for — understanding single-sequence performance is more immediately useful. And the benchmarks that get cited to prove vLLM’s superiority often measure the wrong thing.

When you’re evaluating inference infrastructure for your agent stack, ask specifically: am I measuring batch throughput or per-request latency? Am I optimizing for cost-per-token or time-to-first-token? The answers determine which engine actually wins for your use case.

The concurrency collapse — the point where llama.cpp matches or beats vLLM at single-sequence tasks — is more relevant to agentic AI than the headline batched benchmarks suggest. That’s worth knowing before you rent your first GPU node.