Three weeks ago, a paper appeared on arXiv titled On-Policy Self-Distillation for Large Language Models. Its core proposal was clean enough to state in one sentence: instead of training a student model using a separate, larger teacher model, let a single model be both teacher and student — sampling from its own current policy and learning from the quality of its own reasoning traces. The authors, from UCLA and Meta, showed this approach achieved4–8× better token efficiency than conventional reinforcement learning on mathematical reasoning benchmarks like AIME and HMMT.
On June 2, 2026, Microsoft launched seven new MAI models at Build 2026. The flagship was MAI-Thinking-1, described in the accompanying paper “Building a Hill-Climbing Machine.” The hill-climbing metaphor is deliberate: instead of hoping for a sudden leap from a much larger teacher model, the system makes incremental improvements by staying close to the distribution it actually generates — on-policy, in the language of RL theory. The result is a reasoning model that rivals much larger counterparts while consuming fewer tokens per query.
Why On-Policy Self-Distillation Matters
To understand why this is interesting, you need to see what it replaces.
Standard RLHF and distillation pipelines work off-policy: the student generates responses, but the training signal comes from a frozen teacher model that has already seen different distributions of output. The teacher scores what the student produced, and the student updates. The problem is that the teacher’s distribution and the student’s distribution drift apart over training. The student learns to maximize a signal that doesn’t reflect what it would actually generate — a distribution mismatch that compounds with every update step.
On-policy approaches fix this by only learning from samples the student currently generates. Every training example comes from the student’s own output distribution, so the feedback signal is always calibrated to what the model actually produces. The cost is that on-policy training is data-inefficient: you throw away all the samples you don’t train on in a given step, and you need a large, diverse set of on-policy samples to avoid catastrophic forgetting.
The OPSD paper’s contribution is a framework that captures the on-policy advantage — dense, calibrated feedback — while avoiding the data-inefficiency problem. The model acts as its own teacher by maintaining a critique head that evaluates the quality of its own reasoning chains. When the model generates a reasoning trace, the critique head scores it, and the model updates toward traces that score well. Because the feedback comes from the model’s own critique mechanism, there’s no distribution mismatch: the teacher and student are the same model at every training step.
The efficiency numbers are striking. On AIME (a benchmark of competition mathematics problems), OPSD-trained models matched or exceeded the performance of RL-trained counterparts while using 4–8× fewer tokens during training. The token efficiency comes from the fact that the model doesn’t waste compute learning from samples that are off-distribution — every update step is grounded in what the model would actually say.
What MAI-Thinking-1 Looks Like in Practice
MAI-Thinking-1 is the reasoning engine inside Microsoft’s new MAI family. The model is a 14-billion parameter self-distilled reasoner — not a distilled copy of a larger model, but a model that learned to reason by iteratively critiquing and improving its own reasoning traces.
The practical implications for agentic workflows are significant. A self-distilled reasoner that has learned to evaluate its own reasoning chains is better suited for multi-step agentic tasks than a model that relies on external feedback. An agent using MAI-Thinking-1 can generate a plan, evaluate the plan’s coherence against its own internal critique, and revise — without needing a separate verifier model or a human in the loop.
This is the architecture that makes “hill-climbing” an apt metaphor. The model starts from a reasoning trace, makes a small update toward better-scoring traces, and repeats. Each step is small and on-distribution, so the model never takes a large leap into low-probability territory. The result is a stable, incremental improvement process that doesn’t require a massive teacher model to supervise it.
The Seven-Model Family
MAI-Thinking-1 is one of seven models launched at Build 2026. The full family spans reasoning (MAI-Thinking-1), coding (MAI-Code), image generation and editing (MAI-Image-2.5, which launched at No. 2 on Arena for image editing), transcription (MAI-Transcribe, described as the world’s fastest), and multimodal variants. The family shares the underlying self-distillation methodology — each model learned to perform its task by iteratively improving on its own outputs rather than by mimicking a larger teacher.
For developers building multi-agent pipelines, this is worth noting. The traditional pattern is to route tasks to specialized models: a reasoning model for planning, a coding model for implementation, a vision model for image tasks. If each of those models was trained with the same on-policy self-distillation methodology, they share a coherent internal logic — they all evaluate their own outputs against internal critique signals rather than against external teacher distributions. That consistency matters when you’re composing agents: a planning agent and a coding agent trained with the same methodology will have more compatible internal representations of task quality than models trained with different RL pipelines.
What This Means for Inference Engine Choice
MAI-Thinking-1’s 14B parameter count is intentionally small. Microsoft’s bet is that a well-trained small model with efficient inference will outperform a larger model with expensive inference for most practical agentic tasks. This aligns with the broader trend in local inference — tools like SGLang, vLLM, and LM Studio are optimized precisely for running 7B–14B parameter models at high throughput on consumer or data-center GPU hardware.
If you’re currently routing reasoning tasks to Opus 4.7 or GPT-4.1 at $10–$15 per million tokens, a self-distilled 14B model served locally through SGLang or LM Studio changes the cost calculus significantly. The quality tradeoff is task-dependent — for multi-step reasoning over novel problems, a frontier model still has a meaningful edge. But for structured agentic loops where the model generates a plan, executes it, and evaluates the result, a well-trained small model may be sufficient at a fraction of the cost.
The Build 2026 announcement also noted that MAI models are available through Azure AI Foundry, with endpoints compatible with the OpenAI SDK. If you’re already running an agent stack with LiteLLM or a similar abstraction layer, swapping in MAI-Thinking-1 for specific task types is a configuration change, not a code rewrite.
The Research Gap Worth Watching
The OPSD paper appeared on arXiv in January 2026. Microsoft’s MAI-Thinking-1 shipped in June 2026 — five months later. That’s a fast productionization cycle for a research paper, and it suggests Microsoft has an internal pipeline for taking novel training techniques from paper to production model quickly.
The gap is worth watching because the research landscape for reasoning models is active. The January 2026 OPSD paper was from UCLA and Meta. Anthropic published “When AI Builds Itself” in early June 2026, describing a 52× self-improvement rate in their latest models. Both research directions point toward models that improve their own reasoning without relying on external supervision — but the two approaches have different tradeoffs. OPSD stays on-distribution and is token-efficient; the Anthropic approach appears to allow larger exploratory leaps but may require more compute.
Which approach produces better results for agentic workloads is an empirical question that the next few months of benchmark data should start to answer. For now, MAI-Thinking-1 is the most visible production deployment of the on-policy self-distillation idea — and it’s worth understanding regardless of which inference provider you use, because the methodology is likely to show up in more models as the technique matures.
Sources: MAI-Thinking-1 paper (microsoft.ai), On-Policy Self-Distillation arXiv:2601.18734, Building a Hill-Climbing Machine announcement, Microsoft Build 2026 recap, Self-Distilled Reasoner blog (Siyan Zhao)