Every serious LLM training run eventually hits the same wall: attention. Standard self-attention scales quadratically with sequence length — O(n²) in compute, O(n²) in memory. Train on 128K tokens and you’re moving around a matrix that’s 128,000 × 128,000 for every single layer, every single forward pass. FlashAttention helped enormously by doing this in tiles that fit in GPU SRAM. But even FlashAttention has a ceiling when context windows keep growing.
That’s the problem Lighthouse Attention tackles — not at inference time, but at training time. The paper (arXiv:2605.06554, May 2026) comes from Nous Research, led by Bowen Peng. The core idea is deceptively simple: during pretraining, selectively compress the attention computation so the model sees a fraction of the full token matrix, then recover full attention before deployment.
The Core Mechanic: Selection-Based Hierarchical Pooling
Standard attention computes weighted sums across all keys for every query. Lighthouse replaces this with a two-layer system:
Layer 1 — the Lighthouse itself: A gradient-free selection layer looks at every query token and picks the top-K most relevant key tokens from a coarser, pooled representation. This selection happens without any learned parameters — it’s just a cosine similarity threshold. No gradients flow through the selection, so the compression doesn’t degrade the learned attention patterns.
Layer 2 — the pyramid: Queries, keys, and values are pooled symmetrically across multiple resolution levels. A token at position 50,000 doesn’t need to attend directly to position 12 in the same sentence — it can route through a lower-resolution node that captures the same structural information. The selection layer decides which resolution each token talks to at each step.
The beauty: this wrapper sits around standard Scaled Dot-Product Attention (SDPA). You take your existing transformer architecture, wrap the attention call with the Lighthouse selection layer, and train. No architectural rewrites, no new positional encodings, no changes to the loss function.
The Two-Stage Training Trick
This is where it gets really interesting. Lighthouse is training-only. You can’t actually deploy with it — you’d be shipping a compressed model with worse downstream quality. Instead, the authors use a two-stage approach:
Stage 1: Train with Lighthouse Attention for the majority of steps. The model learns the same patterns it would with full attention, but each step is dramatically cheaper because most tokens are attending to pooled representatives rather than the full sequence.
Stage 2: Switch to standard full attention for a smaller fraction of training — just enough to let the model’s attention heads recalibrate to the full-resolution distribution. The paper finds that a relatively short “recovery” phase (a few percent of total steps) closes almost all the quality gap.
The result: most of the training happens fast, and you end up with a model that’s identical to one trained entirely with full attention — because the final percent of steps restores full resolution anyway.
The Numbers
On a 530M parameter model trained on C4 with long-context sequences:
- 1.4–1.7x end-to-end pretraining speedup compared to standard FlashAttention baselines
- Lower final training loss in some configurations
- No architectural changes to the final deployed model
The speedup comes from the reduced memory bandwidth pressure. When most tokens attend to pooled representatives, you’re moving far less data between HBM and SRAM. Sebastian Raschka called it “wrapping ordinary SDPA with a hierarchical, gradient-free selection layer that compresses and routes” — which is a clean description of what the mechanism does.
Why This Matters for the Field
The quadraticscaling problem is the dominant constraint in modern LLM development. Gemini’s 1M token context, GPT-4’s 128K window — these aren’t free. They require either massive compute or clever tricks. Lighthouse Attention is a clever trick that doesn’t compromise the output.
The training-only constraint is also a strategic insight. Any subquadratic attention method that requires changing the architecture has to prove the model still generalizes. Lighthouse sidesteps this — the final model is exactly a standard transformer. Only the training process changes.
This is reminiscent of how gradient checkpointing works: you spend more compute during training to reduce memory pressure, then get the same output. The difference is Lighthouse speeds up the compute itself, not just memory usage.
What’s Still Unknown
The critics — and there are a few — note the evidence base is still narrow. 530M parameters on C4 is a specific setup. Training loss as the sole quality metric leaves questions about downstream benchmark performance. Whether the speedup holds at the scale of a 70B or 405B model, or on domains like code or math, is still being investigated.
But the direction is compelling. A 1.7x training speedup, with no changes to the final model, built on a 50-line wrapper around existing attention calls? That’s the kind of thing that gets adopted fast if the benchmark results hold up at scale.
The paper is on arXiv (2605.06554) and the Nous Research team has released their implementation. If you’re training long-context models and watching GPU-hours pile up, this is worth an afternoon of your time.
Paper: “Long Context Pre-Training with Lighthouse Attention” — Bowen Peng et al., Nous Research, arXiv:2605.06554 (May 2026)