When Ollama switched to MLX as its default Apple Silicon backend in March 2026, the performance numbers were dramatic enough to make people re-run benchmarks they’d already written off. By June 2026, with build 9672 of llama.cpp underpinning the x86 side and a refined MLX pipeline on Apple Silicon, the gap between local inference and cloud-hosted APIs has narrowed enough that the tradeoffs deserve a second look.
The MLX Difference on Apple Silicon
Apple’s MLX framework is a tensor algebra library purpose-built for Apple Silicon’s unified memory architecture. Unlike llama.cpp, which targets broad GPU compatibility, MLX speaks directly to Apple’s neural engine and GPU cores in a way that maximizes memory bandwidth utilization. The key insight is that Apple’s M-series chips have shared memory between CPU and GPU — a 16-inch MacBook Pro with M3 Max can have up to 128GB of unified memory that both the Neural Engine and the GPU cores access without PCIe bandwidth constraints.
In practical terms, running a 7B parameter model like Qwen3-7B through Ollama’s MLX backend produces throughput that was previously only achievable on discrete GPUs. Tokens per second numbers that required an RTX 3090 in 2024 now run on an M3 MacBook Air, with the added benefit of zero fan noise and laptop battery life.
Ollama’s June 11, 2026 MLX engine update pushed this further. The release notes cite higher output quality across all Apple Silicon tiers, with the most dramatic improvements on the base M3 and M3 Pro chips. The reason: earlier MLX implementations were optimized primarily for M3 Max and M4 series chips, leaving the entry-level hardware with sub-optimal memory controller utilization.
llama.cpp Build 9672: The x86 Counterpart
While MLX handles Apple Silicon, the underlying llama.cpp engine that powers Ollama on Linux and Windows received a substantial update. Build 9672 brings improved KV cache quantization, which reduces the memory footprint of long context windows by up to 40% without measurable quality degradation on most benchmarks. For engineers running Ollama on systems with limited VRAM — a common constraint in CI/CD environments or edge deployments — this is a meaningful improvement.
The Command A and North model families also gained native MLX support in this cycle. These are smaller, highly-efficient models designed for code completion and instruction-following tasks where latency matters more than raw capability. Getting them to run at acceptable speeds on an M3 MacBook Air — previously a frustrating experience — now works out of the box.
What This Means for AI Infrastructure
The Ollama team has effectively created a deployment story where a developer’s laptop is a first-class inference target, not an afterthought. This has direct implications for how multi-agent systems are built and tested:
Local development becomes production-representative. When the same model runs on a developer’s MacBook via Ollama and on a Linux server via vLLM, the behavioral differences are minimal. This shrinks the gap between “it works on my machine” and “it works in production.”
Edge inference becomes viable. A 7B model quantized to 4-bit and running through Ollama on an M-series Mac Mini draws under 20 watts under full load. For on-premise deployments where latency to a cloud API is unacceptable — financial trading desks, medical facilities, government offices with air-gapped networks — this changes what’s architecturally possible.
The GGUF ecosystem matures. Ollama 0.30 (June 5, 2026) improved GGUF compatibility through llama.cpp, meaning the vast ecosystem of quantized models from TheBloke and others now loads reliably without the format conversion friction that plagued earlier versions.
The Numbers That Matter
For a concrete picture: a Qwen3-7B model at Q4_K_M quantization through Ollama 0.30 on an M3 Pro (36GB unified memory) sustains approximately 45 tokens per second for prefill and 60 tokens per second for decode. The same model through vLLM 0.8 on an NVIDIA RTX 4090 produces around 55 tokens per second. The gap has effectively closed.
On the M3 Max with 128GB, larger models become feasible. A 14B parameter model at Q5_K_M runs at roughly 30 tokens per second — fast enough for interactive use, quiet enough to run in a library.
The Tradeoffs Are Still Real
It’s worth being precise about what hasn’t changed. Local inference on Apple Silicon still can’t match the throughput of a single H100 GPU for large batch processing. If you’re running async agents that need to process thousands of requests per minute, cloud infrastructure remains the right tool. And Apple’s thermal throttling on sustained workloads means the peak numbers only hold for the first few minutes before the chip manages heat.
But for the engineering workflow of a solo developer or small team — prototyping agent prompts, running regression tests on model outputs, having a model available during travel or in low-connectivity environments — Ollama’s June 2026 releases make local inference a realistic option rather than a compromise.
The question is no longer whether local LLM inference works. It’s whether the workflow you’re building fits within the constraints that make it practical.
Ollama is open source at github.com/ollama/ollama. The MLX backend is included in the standard release for macOS.