Context Windows Aren't Memory: Architecting State for Long-Running AI Agents
Why a bigger context window won't fix a long-running AI agent, and how we architect external state for AI systems that run operations for weeks, not minutes.
Give a model a bigger context window and it looks like you have given it a memory. You have not. You have given it a larger desk. Everything on that desk is still swept clean the moment the session ends, and a desk that holds a million tokens is still a desk, not a filing system. The distinction sounds pedantic until you try to run an agent for longer than one conversation. Then it becomes the whole problem.
Most AI demos run for a single turn. Most useful AI runs for weeks. An agent that manages operations, triages inbound work, or keeps a financial picture current is not answering a question, it is holding a job. The question that drives our R&D is not "how much can the model read at once" but "what does this agent need next week that it learned today, and where does that live once the window is gone."
Working memory versus storage
A context window is working memory. It is fast, it is expensive, and it disappears. Treating it as the agent's permanent record is the single most common architecture mistake we see, and it is seductive precisely because vendors keep enlarging the window. Larger windows raise the ceiling on what an agent can consider in one pass. They do nothing for what an agent should remember between passes.
State is the thing that persists. It is the decisions an agent has made, the facts it has confirmed, the tasks still open, the corrections a human gave it last Tuesday. None of that belongs in the prompt by default. It belongs in a store the agent reads from and writes to deliberately, the same way an application reads from a database rather than keeping every record in RAM and hoping the process never restarts.
Once you draw that line, the architecture gets simpler, not harder. The context window becomes a scratchpad you assemble fresh for each task. Durable state becomes a system you design, version, and inspect on its own terms.
What breaks when you conflate the two
The failure mode is quiet. An agent that stuffs its entire history into the window does not crash. It degrades. Three things go wrong at once, and all three are expensive.
Cost scales with everything you carry. If every turn re-reads the full history, you pay for the whole transcript on every single call. For a long-running agent that is not a rounding error, it is the dominant line item, and it grows without bound as the job continues.
Signal drowns in volume. Models attend unevenly across a long window, and the thing that actually matters for this turn, a constraint a human set three weeks ago, gets the same weight as yesterday's small talk. More context is not more intelligence. Past a point it is noise with a token bill attached.
Behaviour stops being reproducible. When state lives implicitly in an ever-growing transcript, you cannot say why the agent did what it did, because the input was different every time and you never wrote it down. For anything touching finance or a regulated process, "we are not sure what it saw" is not an acceptable answer.
How we architect state
We keep context assembly explicit. Before each task the agent retrieves only the state relevant to that task, composes a window for it, and discards the window afterwards. Nothing accumulates by accident. If a fact is in the prompt, a line of code put it there on purpose.
We store state as structure, not prose. A rolling English summary of "what has happened so far" is lossy and drifts every time it is rewritten. Typed records, open tasks, confirmed facts, and decisions with timestamps do not drift, they can be queried, and they can be shown to a human without the model in the loop. Prose is for the model to read; structure is for the system to trust.
We compact deliberately, and we keep the source. Older detail gets summarised to stay within budget, but the summary is a derived view, not a replacement. The underlying events stay in the store so an agent, or a person, can go back to what was actually said rather than to the model's paraphrase of it. Compaction is a caching strategy, not a memory strategy.
We treat the handoff as a first-class event. In professional services the work moves constantly, between one agent and another, between a session today and a session next month, between the machine and the person who owns the outcome. State is what survives that handoff. If the next actor cannot reconstruct where things stand from the store alone, without replaying a conversation they were never part of, the architecture has failed regardless of how well any single turn performed.
The limits worth being honest about
External state is not free and it is not magic. You now own a schema, and a bad schema is harder to fix than a bad prompt because the agent has been writing into it for weeks. We have changed our minds about what deserves to be durable more than once, and each change meant a migration.
Retrieval can fetch the wrong state as easily as a window can bury the right fact. Moving the problem out of the context window does not dissolve it, it relocates it to the quality of what you store and how you decide what is relevant now. That is a genuine engineering surface, with its own tests and its own failure modes, and it needs the same scrutiny you would give a query planner.
And some work genuinely fits in one window. If a task starts and finishes inside a single session and never needs to be referred to again, an elaborate state layer is over-engineering. The discipline is knowing which kind of problem you have before you build for the other one.
The takeaway
The frontier that matters for applied AI is not the size of the context window. It is the quality of the state you keep outside it. Bigger windows make better demos. Better state architecture makes agents you can run in production, hand between people, and still explain six weeks later.
If you are building an AI system that has to hold a job rather than answer a question, design the memory before you design the prompt. The prompt is what the agent is thinking about right now. The state is what makes it the same agent tomorrow.
Small-Model Economics: Why the Frontier Model Isn't Always the Right Call
When a smaller model beats the frontier model in production. Cost, latency and failure-mode trade-offs from applied AI R&D in Australia, finance to voice.
Designing for the AI-Native Web: Schema, llms.txt, and Being Citable
How to make a site citable by AI assistants in 2026: what structured data and llms.txt actually do for the AI-native web, and where the advice goes wrong.
CPS 230 When Your Service Provider Is an AI System
APRA's operational risk standard never mentions artificial intelligence. It doesn't need to. If your AI sits inside a bank's critical operation, the obligations arrive through the contract — and the transition period has already expired.