Activity Frames: The Missing Compiler Between Screen Capture and Agent Memory

The number that pulled me into the Activity Frames paper was not 86× compression. It was 135.7%.

On one evaluated workday, the measured record put Google Chrome first at 161.6 minutes and Cursor second at 143.9 minutes. A Claude Sonnet 4.5 summary of the same capture called Cursor the primary application at “about seven hours,” inflated its measured time by 2.9×, and invented a session running from 6:40 PM to 5:26 AM. The prose was readable. The arithmetic was wrecked.

That is a useful failure because it is not the usual cartoon of hallucination. The model did not invent Photoshop or claim the user visited Netflix. Across the paper’s absent-fact probes, every tested representation had a 0% hallucination rate. The summary failed on quantities: duration, ordering, and boundaries. It converted an event stream into fluent text, then made the text sound more reliable than it was.

Activity Frames, a preprint by independent researcher Nossa Iyamu, takes the opposite approach. It does not ask a language model to decide what happened on a screen. It compiles local capture into typed, bounded records using ordinary deterministic code. Same capture database, same query window, same output. Every record links back to the raw rows that produced it.

This is less glamorous than another “memory agent.” That is the point.

Conversation Memory Cannot See the Work

Most agent memory systems begin after the agent enters the room. They store messages, tool calls, execution traces, summaries, and selected facts. MemGPT pages conversation history through a bounded context. Mem0 extracts durable memories from dialogue. Zep organizes events into a temporal knowledge graph. Workflow-memory systems learn from an agent’s own successful runs.

All of those are useful, but they share a blind spot: they know what flowed through the agent.

A personal agent can remember that I was asked to review a pull request. Unless it observed the desktop, it cannot know that the user spent 46 minutes in the PR, switched to a dashboard, returned twice, searched for a related issue, and abandoned an email draft at 16:40. Asking the user to narrate that activity defeats much of the reason for having an ambient assistant.

Screen capture is no longer the difficult part. ActivityWatch has tracked application focus for years. Microsoft Recall stores snapshots on Copilot+ PCs. OpenAI’s Chronicle research preview builds memories from recent screen content for Codex on macOS. The unresolved engineering problem is consumption: what representation can an agent afford to read, audit, cache, and trust?

Raw capture is a bad answer. One representative day in the paper contained 2,066 snapshot rows and serialized to 126,812 tokens. Giving that to a model means paying the model to reconstruct sessions, merge duplicate observations, account for dwell time, identify sites, and notice gaps. Worse, it must perform that bookkeeping every time the memory is read.

An LLM summary is compact, but now memory construction inherits model cost, non-determinism, context-window failure, and quantitative drift. Activity Frames inserts a compiler between those two bad choices.

What an Activity Frame Contains

The compiler groups capture rows into episodes keyed by application and site. A frame records:

• application and browser host;
• start, end, wall time, and credited active time;
• typed page references such as pull_request, issue, email, profile, or doc;
• counts of keystrokes, clicks, and copies, without typed content by default;
• interruptions and coverage gaps;
• pointers to the source capture rows.

The distinction between wall time and active time matters. A snapshot is an instant, not a duration. Activity Frames assigns dwell by looking at the gap to the next snapshot, capped at 90 seconds. A gap above 300 seconds ends the session. A short A → B → A detour lasting at most 20 seconds is folded back into A, but B remains recorded as an interruption and its seconds are not credited to A.

Those constants are exposed, simple, and reviewable. There is no latent model deciding whether twelve seconds in Slack was “important.”

The schema also makes an unusually good design decision: measured data and inferred meaning occupy separate tiers. Tier 1 contains only fields derivable from capture. Tier 2 may add labels such as a project name or intent, but inferred fields must be namespaced, confidence-tagged, and linked to evidence. Strip Tier 2 and the measured document remains valid.

That boundary answers a problem from my July post on agent memory. I argued there that the hard part was calibration, not storage: an agent needs judgment to decide what is useful, yet its judgment is exactly what can be stale or wrong. Activity Frames does not solve calibration. It removes calibration from the factual base. The compiler records two GitHub issue views; another layer may call them “release triage,” but it cannot quietly rewrite the observation.

The Compiler Is Intentionally Boring

The reference implementation is an MIT-licensed Python package and MCP server. The basic path is short:

pip install activity-frames
aframes record
aframes context --hours 2
aframes mcp

aframes record provisions a local recorder on supported macOS systems. The compiler itself has no runtime dependencies, opens its SQLite source read-only, and emits JSON, YAML, Markdown, or a compact context block. Six MCP tools expose recent context, full activity, day summaries, communication surfaces, patterns, and step-level replay data.

The segmentation pass is essentially:

for each monitor stream:
  group consecutive snapshots by (application, site)
  credit min(gap_to_next_snapshot, 90 seconds)
  end a session when the gap exceeds 300 seconds
  merge A → B → A when B is at most 20 seconds
  preserve B as an interruption

URL enrichment is deterministic too. The implementation ships parsers for more than twenty services, then falls back through generic search and infrastructure patterns before producing a generic domain reference. Unknown URLs still map to something; the parser never drops them because it lacks a bespoke category.

There is a nice, slightly unfashionable systems idea here. In agent architecture, we often use an LLM because the input is messy and interpretation will eventually be required. Activity Frames asks which work can be completed before interpretation begins. Sessionization, interval arithmetic, URL parsing, gap detection, click containment, and evidence linking do not benefit from creative inference. Turning them into a prompt is an expensive way to avoid writing a compiler.

The 86× Result, Without the Marketing Shortcut

For the paper’s representative full day, the three representations were:

Representation	Tokens	Reduction vs. raw
Raw snapshot rows	126,812	1×
Full compiled JSON	34,815	3.6×
Compact context block	1,469	86×

The 86× figure applies to the compact prompt-ready block, not the complete schema document. That distinction matters. The full document preserves much more detail and reduces the day by 3.6×. The compact block spends a strict token budget on coverage, an application ledger, and a chronology appropriate for routine context injection.

Compilation took a median 68 ms over five runs on an Apple Silicon laptop, with a 65–72 ms range. Across the longer certification run, median full-day compilation was 220.9 ms, with days ranging from 0.1 to 930 ms. The cost followed the size of the new daily delta rather than total history.

For question answering, the evaluation used eight selected days and 64 questions generated across application dominance, duration, app counts, rankings, domain visits, start times, and absent facts. Ground truth came from an independent SQL oracle rather than the compiler itself.

Input to agent	Sonnet 4.5	Opus 4.5
Raw rows	82.1%	91.1%
LLM summary	66.1%	80.4%
Activity Frames block	98.4%	98.4%

The compiled block’s mean error on the dominant application’s duration was 7.3% at both tiers. The summaries scored 135.7% error with Sonnet and 25.2% with Opus. The block also fit on all eight days; the busiest day reached 257,000 raw tokens, so the raw and summary baselines could not run there.

The result supports a practical claim: deterministic preprocessing lets a mid-tier model read the record as accurately as a frontier model. It does not prove that Activity Frames has solved general episodic memory. The questions are tightly aligned with fields the compiler has already calculated, and the paper says so. Asking which app dominated is partly a test of whether the agent can read a ledger whose arithmetic is complete.

Still, that is not a weakness to wave away. Precomputing exact quantities is the product. We do not ask a database chatbot to reconstruct a revenue ledger from individual payment events on every question. We materialize the view.

Replay Is the More Ambitious Half

The paper then moves from memory to procedural reuse. Repeated UI action n-grams become guarded plans: ordered actions with expected element, role, and application. On a matching interface, a local executor can fill variable slots and replay the routine without asking a model to rediscover every click.

This produces the paper’s Routine Overhead Ratio, or R: modeled tokens required for a screenshot-driven agent to re-derive a workflow divided by tokens in the compiled replay representation.

The conservative operational median is 60× for a guarded plan of roughly 248 tokens. A minimal script produces a 343× median ceiling at action granularity. The latter is deliberately labeled an information-content ceiling, not a realized deployment saving. The numerator assumes roughly 2,500 tokens per screenshot-driven step and is modeled, not billed.

The frequency side is more sobering and more useful. Raw repeated action structure covers 83.1% of steps, but most of that is keyboard texture and generic loops. Requiring a routine to contain at least two named targets cuts the delegable recurrence to 9.0%. Training on the first 40 active days and testing on the final 11 yields 7.7% out of sample.

That changes the economics. A covered routine might avoid almost all re-derivation, but only around 8% of real steps in this single-user corpus fall into previously identified, specific routines. The realistic fleet-wide token ceiling is therefore near 8%, not 99% and certainly not 343×.

One live proof-of-concept replayed a two-step compose-and-type routine in an authenticated browser at zero model tokens during execution. It grounded elements by accessibility role and name, adapted when element references changed, and performed no action on the wrong page. Good. But it was one seeded two-step routine; the full mined-routine, live-billed, three-arm comparison remains future work. The paper is admirably direct about this.

Where the Evidence Stops

The corpus belongs to one professional using one machine. The systems analysis spans 61 calendar days with 46 active days and 109,735 snapshot rows; the later overhead freeze contains 128,756 frames over 51 active days. Those are substantial traces, but still one person’s work pattern. The reported recurrence rate cannot be generalized to developers, designers, analysts, or support teams as populations.

Screen presence is also not attention. Heartbeats keep reading and watching visible, but they also credit time when a user steps away from an awake display. Uninterrupted heartbeat runs reached 42 minutes. Multi-monitor accounting goes the other direction: each monitor receives tenure, so 27% of captured minutes in this corpus may be double-credited in per-application ledgers. Paper, phone, and spoken conversation disappear as gaps.

The privacy model is local, not magical. Typed text is excluded from compiled output by default, audio is off, and the compiler uses a read-only database. But the underlying capture database contains sensitive screen-derived material and measured 9.5 GB, about 0.19 GB per active day. Device encryption and access control are requirements, not optional hardening.

Finally, “zero model” starts after capture. The recorder uses on-device OCR. Determinism holds from stored OCR text to compiled document, not from pixels to text; changing the OCR model can change the input. This is still a valuable contract, just a narrower one.

The open repository, nossa-y/activity-frames, currently exposes the schema, compiler, MCP surface, evaluation harness, and replay research. The next test I want is not another compression number. It is a multi-user temporal holdout with live billing: mine routines from several roles, replay only guard-matched hits for a month, and publish the deopt rate, incorrect-match rate, capture overhead, and actual provider usage JSON. Until then, Activity Frames is a convincing memory compiler with a promising replay path—not yet evidence that one person’s 7.7% recurrence is the economics of everyone’s computer work.