Recursive Language Models: When 8B Beats GPT-5 by Letting Prompts Live in a REPL

The longer my agent’s conversation history gets, the worse it performs. This is the thing nobody benchmarks well — the same Claude Code session that aced a five-step task at step three starts losing the plot at step thirty. Anthropic calls this context rot: “the number of tokens in the context window increases, the model’s ability to accurately recall information from that context decreases.” The textbook fix is the one you’d reach for in your sleep: compaction, hierarchical summarization, retrieval over the prompt, ReAct-style indexing. Alex L. Zhang’s MIT OASYS team has a different answer, and it’s surprisingly simple.

Recursive Language Models (RLMs) don’t try to fit the prompt in the context window. They park the prompt in a Python REPL as a variable named prompt, and they let the model write code to slice it up, peek at chunks, and recursively call itself — or other LLMs — over those chunks. From the outside, it’s still rlm.completion(prompt, model). Under the hood, the model is writing Python that looks roughly like chunks = [prompt[i:i+8000] for i in range(0, len(prompt), 8000)]; results = [llm_query(c) for c in chunks]. The numbers from the paper are striking enough to be worth quoting directly: on GPT-5, RLMs score a median of 26% above plain compaction, 130% above CodeAct-with-sub-calls, and 13% above Claude Code across four diverse long-context tasks, at comparable cost. Even at the small end, a post-trained RLM-Qwen3-8B beats its base Qwen3-8B by 28.3% on average and approaches vanilla GPT-5 quality on three long-context tasks.

The kicker for anyone shipping agents: the paper includes a BrowseComp-Plus experiment where RLM-with-GPT-5 holds its accuracy flat as the input grows from tens to thousands of documents — over ten million tokens — without using a retriever. The thing we usually reach for to manage long contexts, retrieval, is exactly what the RLM is obviating.

The Mechanism: Prompts as a Variable in a REPL

The core abstraction is a thin wrapper around a language model that exposes three primitives the model can call from a Python REPL: llm_query(prompt) for a one-shot sub-call, llm_query_batched([prompt1, prompt2, ...]) for parallel sub-calls, and rlm_query(...) for recursive RLM calls. The user’s prompt is loaded into a Python variable at the top of the REPL session. The model writes Python code to inspect, chunk, filter, and recursively query over it, then prints the final answer.

The README’s getting-started snippet is the cleanest way to see what an RLM call actually looks like:

from rlm import RLM

rlm = RLM(
    backend="openai",
    backend_kwargs={"model_name": "gpt-5-nano"},
    verbose=True,
)

print(rlm.completion("Print me the first 100 powers of two, each on a newline.").response)

That completion() call, under the hood, runs GPT-5-nano in a loop inside a Python REPL. The model sees its own prompt as prompt = "...", the REPL state, and the function definitions for llm_query / llm_query_batched / rlm_query. It writes code, executes it, reads the output, writes more code, calls sub-LMs as needed, and eventually prints a final answer. The conversation history accumulates across iterations within the same REPL session, so the model can build up intermediate state — a running summary, a deduplication index, a chunk-mapping — that wouldn’t fit in a single context window.

The reason this matters: the prompt is no longer something the model has to “hold in mind.” It’s an object the model can index, slice, grep, and re-query. Context rot happens when the model has to remember a 200K-token document; an RLM never has to remember it, because the document is sitting in a variable that the model can address programmatically.

What RLMs Actually Replace

The paper positions RLMs as a replacement for the standard llm.completion(prompt, model) call — same interface, different substrate. The implementation in alexzhang13/rlm ships with seven REPL environment backends, and this is where the engineering depth of the work becomes visible:

• local (default) — runs exec in the host process. Same venv as the RLM. Fine for benchmarking; unsafe for production with untrusted prompts.
• ipython — real IPython session, optionally in a ipykernel subprocess with cell_timeout enforcement and namespace isolation.
• docker — python:3.11-slim container; host-side proxy bridges LM access back in. Full feature parity with local including recursive sub-RLMs.
• modal — Modal Sandboxes. Fully isolated, host-bridged LM traffic.
• prime — Prime Intellect Sandboxes. Beta; the README flags slow runtimes as an open issue.
• daytona, e2b — additional cloud sandbox providers.

The fact that they had to build six sandbox backends to make this usable is itself the engineering story. The clean abstraction is “park the prompt in a REPL and let the model write Python.” The messy reality is that letting a frontier model run arbitrary Python on your host machine is a security incident waiting to happen, and the only sane deployment path is to put the REPL in a sandbox the model can’t escape. That’s why the README explicitly argues against the JSON tool-calling standard for both sub-agents and generic tool calls: “We want to move away from the JSON tool-calling standard for both sub-agents and generic tool calls.” A CodeAct-style harness with sub-LM calls as functions in Python is the bet. It’s a bet that code-as-tool-interface scales better than JSON-as-tool-interface, and the RLM paper is the most concrete evidence for that bet I’ve seen.

I keep thinking about this in the context of the MCP tunnels post I wrote a while back — that post argued for outbound-only mTLS tunnels and gateway-level policy enforcement because giving every MCP server a credential is a recipe for lateral movement. RLMs are making the same architectural move at a different layer: instead of trusting the model with a long prompt in-context, you give it a sandboxed runtime where it can write code, and the runtime enforces what code is allowed to run. The trust boundary shifts from “the model sees the data” to “the model can call into the data through a controlled API.”

The Post-Training Result Is the Bigger Story

The 8B result is the one I keep coming back to. The paper doesn’t just propose an inference paradigm — they post-train Qwen3-8B into an RLM-native reasoner and report that RLM-Qwen3-8B beats base Qwen3-8B by 28.3% on average across their evaluation suite. They also report that RLM-Qwen3-8B approaches vanilla GPT-5 quality on three long-context tasks. That’s an 8B model with explicit RLM training matching a frontier proprietary model on tasks the frontier model was ostensibly designed for.

The training is integrated with Prime Intellect’s verifiers framework and Prime’s prime-rl RL training library. The RLM repo ships with a training/ directory containing the environment so you can plug your own RLM directly into the inference engine. This isn’t a one-off benchmark — it’s a reproducible training recipe for turning any base model into a recursive reasoner.

The deeper claim is that RLMs are the “next milestone in general-purpose inference-time scaling after CoT-style reasoning models and ReAct-style agent models.” CoT bought us chain-of-thought. ReAct bought us tool use. RLMs buy us the ability to programmatically decompose arbitrarily long contexts and recursively reason over the decomposition. If the post-training recipe scales, this is the architectural template that every serious agent stack is going to adopt over the next year.

What I’d Want to Verify Before Betting on This

The cost story is the one I’m most skeptical about. The paper says RLMs are “comparable cost” to vanilla GPT-5 on long-context tasks. I want to see the per-token cost accounting more carefully, because each recursive sub-call is a separate API roundtrip, and at scale that adds up. The 130% improvement over CodeAct-with-sub-calls is interesting precisely because it’s the same shape of cost profile — the difference must be that RLMs make fewer, smarter sub-calls because the model has visibility into the chunked prompt as code, not just as a string.

The other thing I don’t yet know: how well the post-trained 8B model transfers out-of-distribution. The benchmarks in the paper are long-context tasks. I want to see RLM-Qwen3-8B on something where the recursive decomposition isn’t an obvious win — say, short-context code generation or chat. If it still holds up there, the paradigm is genuinely general. If it only works on long-context, it’s a narrow tool.

There’s also a sandbox escape surface that nobody has audited yet. A model that runs Python in a REPL inside a Docker container is still a model that runs Python. The Prime Sandboxes path is flagged “slow” in the README; the Modal path works but adds a network roundtrip per code execution; the local path is explicitly “not for production.” The defensive posture I want to see before deploying this in a multi-tenant agent is: what tools does the REPL expose, can the model import arbitrary Python packages, and is there a way to constrain which llm_query calls the model can make. The README hints at custom_tools and custom_sub_tools but doesn’t ship a hardened reference config.

The thing I’m most curious about, though, is whether RLMs compose with the inference-time scaling techniques people are already using. If I’m running an agent that uses RLM for long-context reasoning and self-consistency for hard decisions, does the self-consistency failure mode I wrote about yesterday still apply? My guess is yes — the RLM gives you better long-context handling but the recursive sub-calls are still samples from the same weights, and correlated errors compound at every level of recursion. The interesting empirical question is whether the recursive decomposition breaks the error correlation enough to recover the theoretical self-consistency ceiling. I haven’t seen that experiment yet.