Agent Memory Architectures: Working Set, Episodic, Semantic, and Beyond

Agent "memory" is not a single mechanism. It is a family of state surfaces — context windows, server-side conversation handles, scratchpad files, retrieval indices, skill libraries — each with its own write authority, retrieval policy, lifetime, and trust boundary. This article maps the working/episodic/semantic/procedural taxonomy onto the implementations it describes, treats the taxonomy as a useful metaphor rather than an ontology, and argues that the load-bearing engineering questions are governance ones: who writes durable state, how it is promoted, how it is invalidated, and whether it is treated as trusted instruction or untrusted evidence.

Coverage note: verified through May 2026.

What "memory" is being used to mean

The cognitive labels practitioners borrow from psychology — working memory, episodic memory, semantic memory, procedural memory — were never load-bearing terms. They were a metaphor for thinking about which information should persist across turns, sessions, and agents. In production systems, those labels collapse into mundane implementation questions:

Who writes the state? The user, the model, an automated extractor, or a tool returning untrusted data?
Who can read it? The current thread, the same user across threads, the user's organization, every agent in a swarm?
When is it loaded? Always-in-context, retrieved by tool call, embedded into the prompt by the harness, or surfaced only when matched by a retriever?
How is it invalidated? TTL, manual delete, supersession by a newer fact, contradiction detection, garbage collection?
Does the model treat it as instruction or as evidence? A CLAUDE.md line is read as instruction; a chunk retrieved from a vector index is read as data — and if the chunk contains an instruction, the boundary collapses.

Most of the failures discussed later in this article — bloat, contradiction, retrieval drift, prompt-injection-via-memory, identity drift — stem from one of those five questions being answered badly, not from the absence of any particular mechanism.

A working definition: agent memory is information that changes future agent behavior by being preserved, retrieved, or reintroduced into a model's input. This excludes inference-time accelerators (KV caches that do not affect output) but includes any state that is later loaded into a prompt or bound to a tool call — whether it lives in a file, a database, a vector store, or a provider-managed conversation handle.

The four-part taxonomy as metaphor

The conventional taxonomy maps roughly as follows:

Cognitive label	What it means in agents	Typical surface
Working set	The active prompt and conversation visible to the model on a given turn	Context window
Episodic	What happened in this task or session — scratchpads, run logs, observations	Files, conversation transcripts
Semantic / declarative	Curated facts, knowledge entries, embeddings	Vector index, wiki pages, knowledge graph
Procedural	Reusable behaviors — skills, tool definitions, prompts, policies	Skill files, tool schemas, system prompts

The taxonomy is useful as an on-ramp. It maps cleanly onto Letta's memory hierarchy (MemGPT, Letta memory blocks, archival memory) and onto the implicit shape of Claude Code's memory pattern (Claude Code memory docs). It also breaks down at most boundaries.

A CLAUDE.md file is "procedural" if you read it as a behavior specification, but it is "semantic" if it contains project facts and "episodic" if it accretes session learnings — in practice, most CLAUDE.md files contain all three. The Anthropic memory tool (Anthropic memory tool docs) is "episodic" when it stores a per-task scratchpad and "semantic" when it stores reusable facts; the same file API serves both. A vector store of past conversations is "semantic" if you query it by topic and "episodic" if you query it by timestamp.

The taxonomy is therefore a vocabulary for talking about what role a piece of state plays in a given agent. It is not a partition of the implementation space. The implementation space is better described by orthogonal axes:

Axis	Values
Persistence / scope	Turn, thread, project, user, organization, cross-agent, global
Write authority	User, model, tool output, automated extractor, human reviewer
Read policy	Always-in-context, model-triggered tool call, embedding retrieval, manual import
Trust treatment	Trusted instruction, trusted reference, untrusted evidence
Mutability	Append-only, mutable in place, versioned, tombstoned on delete

Real frameworks position themselves as combinations of these axes. They do not implement "episodic memory" or "semantic memory"; they implement particular write authorities, scopes, and retrieval policies, which can be cast as one of the cognitive labels after the fact. Treating the cognitive labels as the primary architecture risks category errors that flatten meaningfully different mechanisms — particularly when reasoning about security and lifecycle.

State surfaces, surface by surface

Working set: the context window

The "working memory" of an LLM agent is the input it receives on a given turn. Long context is necessary but not sufficient: even when relevant information sits inside the window, models do not reliably attend to it. Liu et al. 2023 showed that performance on retrieval-from-context tasks degrades substantially when the relevant passage is buried in the middle of the prompt — best performance is typically near the beginning or end. The implication is structural: stuffing more state into context creates intra-prompt retrieval failures, not just cost.

Practical agents therefore treat the context window as a managed surface rather than an unbounded sink. Anthropic's context engineering cookbook frames this directly: "memory is only as good as what the agent chose to save," memory tool calls add overhead, and persistent memory does not solve in-session context growth (Anthropic context engineering cookbook). Compaction, summarization, sliding-window strategies, and explicit "load this file when relevant" hooks are all answers to the same question — what should be in the window right now? — and they are mostly disjoint from the question of what should be in long-term memory.

The working set has one property the other surfaces lack: everything in it is presented to the model as authoritative. There is no metadata channel that says "this paragraph is untrusted." Anything that ends up in context — whether it came from the user, a tool result, or a retrieved memory — is candidate instruction. This is the source of the AgentDojo result: agents lack a formal way to distinguish instruction from data, and tool-returned content can hijack agent behavior. The taxonomy of memory does not change this; it amplifies it, because every other memory surface eventually flows through context.

Server-side conversation state and KV-cache reuse

Several recent provider features get casually described as "memory" but operate at the runtime layer rather than the epistemic one. They deserve to be separated cleanly.

OpenAI Responses and Conversations. The Responses API supports previous_response_id to chain turns, and the Conversations resource persists messages, tool calls, and outputs across requests. Responses are stored for 30 days by default unless store=false is set (OpenAI conversation state guide, OpenAI data controls). This is server-side conversation continuity: the client does not need to resend the full transcript, but previous inputs are still billed as input tokens, and the model still sees them as part of the prompt on each turn. It is ergonomic state management, not a system that decides what to remember.

Prompt caching. OpenAI's prompt caching documentation describes exact-prefix matching of repeated prompts, with in-memory retention typically measured in minutes and an extended retention option of up to 24 hours for supported models. KV tensors may be persisted in local provider storage. Critically: caching does not affect final generation (OpenAI prompt caching). Anthropic's prompt caching is shaped similarly. KV cache hits reduce latency and cost but produce the same outputs as a cold prompt would.

This is closer to memoized compute than to memory. KV cache is not addressable by the agent, not editable by the user as knowledge, not queryable, and not an epistemic store. It is invisible to the agent's reasoning and exists only as long as the provider's eviction policy allows. Treating it as part of the agent memory taxonomy creates more confusion than insight: the engineering controls (cache key shaping, prefix stability, batch routing) have no overlap with the controls for episodic or semantic memory (write authority, retrieval policy, expiry).

The clean line is: server-side conversation state is durable input plumbing; KV cache is durable inference plumbing; neither is a substitute for explicit memory mechanisms. They reduce the cost of carrying state forward; they do not decide what state to carry.

Episodic memory: scratchpads, run logs, per-task files

Episodic memory in agents is what happened during a particular task or session. Implementations vary widely:

A single scratchpad file written during one task and discarded after.
A run log directory keyed by task ID.
A conversation transcript stored verbatim with metadata.
A per-thread vector index of the model's own observations.

Anthropic's client-side memory tool (Anthropic memory tool docs) is the cleanest example of episodic memory exposed as a primitive. The model receives a memory tool with view, create, str_replace, insert, delete, and rename operations over a memory directory. The implementer is responsible for path constraints, size limits, expiration, and sensitive data handling. The tool deliberately does not impose a schema: the model writes Markdown notes, and the harness decides how to bound the filesystem.

Episodic memory has three intrinsic problems:

Promotion ambiguity. A note written during a task may or may not be useful in the next task. Naive systems either keep everything (bloat) or keep nothing (forgetfulness). Mem0's approach is to extract structured facts from conversation history and merge them into a maintained store with conflict resolution rules (Mem0 paper). This is plausible but still under-evaluated: the paper reports a 26% relative improvement over OpenAI memory on an LLM-as-judge metric and significant latency and token-cost savings, but it is a vendor-authored arXiv paper and the benchmark methodology has not been independently replicated.
Provenance loss. A summarized scratchpad is the model's interpretation of what happened, not the source record. Aggressive summarization to control bloat trades fidelity for compactness. Once summarized, contradictions between the summary and the underlying transcript are hard to detect — and the summary is what gets retrieved next time.
Trust collapse. A scratchpad written by the model in response to a tool result inherits the trust of the tool result. If the tool returned attacker-controlled content (a poisoned web page, a malicious email), the scratchpad now contains attacker-controlled instruction-shaped text. When the scratchpad is reloaded in the next session, it is read as if it were the agent's own prior reasoning. This is the central concern in Anthropic's managed-agents memory documentation, which warns explicitly that if an agent processes untrusted input, prompt injection can write malicious content into read-write memory, and future sessions may read that content as trusted memory (Anthropic Managed Agents memory).

Semantic / declarative memory: knowledge stores and retrieval

Semantic memory in the agent context typically means a curated knowledge store that is queried at retrieval time and injected into context. The dominant implementation is vector-RAG over a corpus of documents, but the term covers a range:

Pure RAG: chunk a corpus, embed, retrieve top-k, inject. The retrieval is opaque to the model; the model sees only the retrieved chunks.
GraphRAG / structured retrieval: retrieve via a graph or schema rather than nearest-neighbor in embedding space.
Curated wiki: maintain Markdown pages synthesized from sources, hand-edited or model-edited, with explicit cross-references.
Profile / preference memory: a small store of user-specific facts that gets included in every prompt.

The Karpathy LLM Wiki gist (Karpathy gist) is a useful counterpoint to plain RAG. The pattern proposes a persistent, interlinked Markdown wiki maintained by an LLM, with raw sources kept separately, a schema file describing page structure, indexing for retrieval, periodic linting for contradictions and stale claims, and explicit cross-references. The argument is not that retrieval is unnecessary but that re-deriving synthesis from raw documents on every query is wasteful and inconsistent — better to maintain a curated synthesis layer above the raw sources. The pattern is best described as curated semantic consolidation; it is plausible for slow-moving corpora where periodic human review is feasible, and unproven for fast-moving corpora where wiki maintenance becomes the bottleneck.

The empirical case for any particular semantic-memory architecture is thinner than vendor materials suggest. Long-term memory benchmarks show substantial gaps:

LongMemEval reports a roughly 30% accuracy drop for commercial chat assistants and long-context LLMs across sustained interactions, evaluating extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.
LoCoMo finds that long-context LLMs and RAG strategies help but still substantially lag human performance on long-range temporal and causal dialogue understanding.

Both benchmarks were run on systems with retrieval and substantial context — the gap is not that retrieval is missing but that retrieval-plus-context is still insufficient for sustained-interaction memory tasks. The Karpathy LLM Wiki versus standard RAG comparison has not been settled by published benchmarks; there is no public evaluation comparing it to hybrid RAG, GraphRAG, long-context prompting, or human-maintained wikis on freshness, factuality, latency, and maintenance cost.

Semantic memory inherits a category of attacks that working-set and episodic memory do not face at the same intensity: corpus poisoning. PoisonedRAG showed that injecting a small number of malicious texts into a knowledge database can steer downstream outputs, with high attack success rates in their setup. The attack surface is the ingestion pipeline: any source the system indexes becomes a potential injection vector. RAG systems that index web content, support tickets, email, or shared drives are routinely connected to surfaces an attacker can write to. Provenance, source classification, and ingestion review are not RAG hygiene niceties — they are load-bearing security controls.

Procedural memory: skills, tools, and policies

Procedural memory is reusable behavior. The clearest examples are:

Skills: bundles of instructions, scripts, and references that the harness can load when relevant. Claude Code's skills are progressively-loaded — the skill metadata is always present, and the body is loaded when the skill is invoked.
Tool definitions: schemas the model can call, often paired with descriptions that effectively constitute behavior specifications.
System prompts and policy files: instructions that bind the agent's behavior across all turns.
Memory hierarchy files (Claude's CLAUDE.md, Cursor's rules, etc.): project- or directory-scoped instructions loaded into context.

The Claude Code memory documentation notes that CLAUDE.md files are loaded as context, not enforced configuration, and that contradictory rules may be picked arbitrarily (Claude Code memory docs). This is the central caveat for procedural-memory-as-prompt: the model is reading the rules, not executing them. There is no harness-enforced policy layer; there is a strong-prior layer that is sometimes overridden by other content in the prompt or by user instruction. The rules/ directory pattern with paths: frontmatter is closer to enforced procedural memory because the harness decides which rules load based on the file being edited, but even there the model still has to follow them.

Procedural memory's risks are different from episodic and semantic. The dominant problems:

Stale skills. A skill written six months ago refers to a deprecated API. The harness still loads it; the agent still tries to use it.
Overbroad tool permissions. A skill grants the agent shell access for one workflow; the permission persists for unrelated tasks.
Hidden behavior. A CLAUDE.md rule from an obscure subdirectory shapes the agent's behavior in ways the user has not seen.
Procedural drift via memory writes. If procedural files are themselves writable by the agent (as CLAUDE.md is, in some configurations), they become a target for the same prompt-injection-into-memory attacks that affect episodic memory — except that injected content is read as instruction, not data.

The line between procedural memory and the other categories is the trust treatment. Procedural memory is read as instruction. That makes write authority over procedural memory the most security-sensitive write authority in the system. Adding a fact to a knowledge store and adding a rule to a system prompt look superficially similar; they are not the same operation.

Framework mapping

The major frameworks instantiate these surfaces in different combinations.

Framework	Working set management	Server-side state	Episodic	Semantic	Procedural	Notable boundary properties
Claude Code (Anthropic)	Compaction, `/compact`, context indicator	Provider-managed in Claude Code app	`~/.claude/projects/<project>/memory/` files; `MEMORY.md` index	Optional via MCP servers, RAG plugins	`CLAUDE.md` hierarchy, skills, agents, hooks	Memory files are user-visible Markdown; promotion is model-decided with user-typed `# <note>` shortcut
Anthropic Memory tool (API)	Caller's responsibility	Caller's responsibility	File-backed memory directory via `memory` tool	Caller-implemented atop file API	Caller-implemented atop file API	Path traversal, size, sensitive-data filtering all caller's responsibility (Anthropic memory tool docs)
Anthropic Managed Agents memory	Managed	Managed	Persistent memory store (read-write)	Same store, different access patterns	Same store	Explicit warning on prompt injection into memory (Anthropic Managed Agents memory)
OpenAI ChatGPT Memory	UI-managed	N/A (consumer product)	"Saved memories" + reference to chat history	Same store	None directly	User-visible memory list; user can edit/delete (OpenAI Memory FAQ)
OpenAI Responses API	Caller's responsibility	`previous_response_id`, Conversations	Not directly; caller stores	Not directly; caller implements	Not directly	30-day default retention unless `store=false` (OpenAI conversation state, data controls)
MCP memory servers	N/A (tool layer)	Per server	Common pattern: knowledge graph or key-value store	Common pattern: structured store with read/write tools	None	Tool-mediated; trust treatment depends on harness (MCP servers)
Karpathy LLM Wiki	N/A (pattern)	N/A	Raw sources kept separately	Markdown wiki, schema-driven, LLM-maintained	Schema file is procedural	Curated synthesis layer; depends on review and linting (Karpathy gist)
Letta / MemGPT	Virtual context paging	Provider state	"Core memory" blocks always in context; recall memory	Archival memory (vector store)	Memory blocks can encode persona/system	Shared memory blocks across agents; memory tool calls are model-driven (Letta memory, archival memory)
mem0	N/A (memory layer)	N/A	Extracted memories from conversation history	Same store, with merge/conflict resolution	None directly	Vendor-reported gains over baselines; methodology should be read carefully (Mem0 paper)

A few observations the table compresses:

Claude Code's memory layer is not one mechanism. It is at least three: the CLAUDE.md hierarchy (procedural, always loaded, project- or user-scoped), the auto-memory directory (~/.claude/projects/<project>/memory/ with MEMORY.md as an always-loaded index, episodic/semantic, model-managed with user-typed shortcuts), and the optional MCP memory servers (whatever the server implements) (Claude Code memory docs). The user-facing label "memory" elides the differences. The differences matter because each surface has a different write authority, scope, and trust treatment.

OpenAI ChatGPT Memory and the OpenAI Responses API are different products. ChatGPT Memory is a consumer feature where the assistant maintains a list of saved memories users can edit (OpenAI Memory FAQ). The Responses API supports conversation state via previous_response_id and the Conversations resource, but it does not provide an autonomous "remember this" mechanism — that is a caller's responsibility (OpenAI conversation state). Conflating the two is a frequent source of confusion in framework comparisons.

MCP memory servers are an integration pattern, not a single architecture. The MCP servers reference repository (MCP servers) ships a memory server that maintains a knowledge graph, but any MCP server can expose memory-like tools. The relevant questions — scope, trust, write authority — are answered by the specific server, not by MCP itself.

Letta and MemGPT center the hierarchy explicitly. MemGPT proposes virtual context management with memory tiers and pageable memory; Letta's docs describe core memory blocks (always in context), recall memory, and archival memory (vector store), with shared memory blocks across agents (Letta memory, memory blocks, archival memory). This is the closest production system to a literal implementation of the working/episodic/semantic taxonomy.

Mem0 is an extracted-memory layer. It treats the conversation history as the source and maintains a derived memory store with conflict resolution and selective retrieval. Its claimed numbers — 26% relative improvement over OpenAI memory on LLM-as-judge, 91% lower p95 latency, >90% token-cost savings versus full-context approaches (Mem0 paper) — are promising but vendor-authored and not independently replicated at the time of writing.

Failure modes

The taxonomy by itself does not predict where systems fail. The empirical record is concentrated in a small number of recurring patterns.

Memory bloat

Append-only memory grows. Even with reasonable per-item size limits, a year of session memories outgrows useful context budgets. The mitigations create their own problems:

Aggressive summarization preserves the model's interpretation of events and discards the source. Provenance is lost; later correction is harder.
TTL-based expiry drops things that may still be relevant. A user preference set six months ago may still apply; a workaround logged six months ago may still be load-bearing.
Capacity-based eviction pushes out memories ranked lowest by some heuristic — usually recency or access frequency. Stable, rarely-mentioned facts (an allergy, a long-term project constraint) score poorly and can be evicted before transient noise.

The honest engineering position: there is no automatic eviction policy that is right by default. Useful systems combine TTL, capacity limits, explicit user pinning, and periodic review. Anthropic's context engineering cookbook makes the related point that memory is only as useful as the agent's choice of what to save, and that memory tool calls add per-turn overhead (Anthropic context engineering cookbook).

Contradictory updates

Two writes can disagree. Common patterns:

A user expresses a preference once (likes terse responses), then later expresses the opposite (asks for thorough explanations on a complex topic). Without conflict-resolution rules, both end up in memory and the agent sees inconsistent instruction.
A project fact changes (the database moves from MySQL to Postgres). The new fact is added; the old fact is not removed. Retrieval returns both.
Two agents write to the same shared memory block with different summaries of the same task.

Mem0 reports explicit merge and conflict-resolution behavior (Mem0 paper). Letta's shared memory blocks rely on the caller to decide write semantics (Letta memory blocks). Most file-backed memory systems (Anthropic's memory tool, Claude Code's memory directory) leave conflict resolution to the model: the model is supposed to notice contradictions and resolve them, which it sometimes does and often does not.

The architectural lesson is that mutability without versioning is a contradiction-generator. Append-only stores with explicit supersession (this fact replaces that fact, with both kept for audit) are more robust than mutable stores without history.

Retrieval drift

Embedding-based retrieval can rank a stale or thematically similar memory above a current one. The failure mode:

A user's old preference scores high on cosine similarity to the current query and overrides a more recent contradicting preference.
A previous-task scratchpad scores high because it shares vocabulary, and the agent treats it as relevant context for a new task.
Two memories with similar embeddings but contradictory content both retrieve, and the model picks one essentially at random.

LongMemEval and LoCoMo both quantify variants of this: memory systems built on retrieval-plus-context still degrade significantly on temporal reasoning, knowledge updates, and abstention tasks (LongMemEval, LoCoMo). The benchmarks have their own framing decisions, but the directional finding — retrieval is not enough — is robust across them.

Mitigations are well-known and underused: filter by recency before semantic ranking, weight retrieval by source authority, run a small re-ranker over candidates, and explicitly include "no relevant memory" as a valid retrieval outcome.

Prompt injection through memory

This is the failure mode that most clearly distinguishes durable memory from transient context. Because memory writes flow into future prompts, an attacker who can write to memory can inject instructions into future agent runs. Concrete paths:

Tool-output to scratchpad to next session. An agent fetches a web page; the page contains hidden instructions; the agent summarizes the page to its scratchpad; the scratchpad is loaded next session. Anthropic's managed-agents memory documentation calls this out by name (Anthropic Managed Agents memory).
Corpus poisoning. An attacker writes content to a shared corpus the agent indexes. PoisonedRAG demonstrates effective attacks with very small numbers of poisoned entries.
User-message-to-procedural-memory. A user (or anyone with chat access) instructs the agent to "remember" a directive. If the harness has any mechanism that turns chat content into procedural memory — # <note> in Claude Code, the auto-extraction in Mem0, ChatGPT's memory writes — this becomes a privilege-escalation surface.
Cross-tenant bleed. In multi-tenant systems with shared memory, a memory written in one tenant's context surfaces in another's.

AgentDojo provides the most useful general framing: agents have no formal mechanism to distinguish instruction from data inside the context window, and tool-returned content can hijack agent behavior. Memory systems that funnel tool output into future prompts are extending this attack surface across time.

The mitigations that actually help:

Treat memory loaded from any source as untrusted evidence by default, not as instruction. The prompt structure should mark memory regions explicitly and discourage the model from following instructions inside them.
Require human review for procedural-memory writes. The line between "store this fact" and "instruct the agent to behave this way next time" is the line between a knowledge update and a privilege change.
Maintain provenance for every memory entry: which session, which tool output, which user, which timestamp. Provenance enables targeted invalidation when a source is later identified as compromised.
For shared memory, enforce scope boundaries at the storage layer, not at the prompt layer. A model instructed not to reveal cross-tenant memory is not a control; storage that does not return cross-tenant memory is.

Identity and authority drift

The subtler failure mode is that memory silently shifts whose voice the agent is reading. A scratchpad written in response to a user instruction sounds like the agent's own reasoning when reloaded next session. A summary of a tool result reads as the agent's prior knowledge. Over many sessions, the agent's "memory" becomes a layered palimpsest of model interpretations of user instructions, tool outputs, and prior summaries — none of it cleanly attributable.

This is not a security failure in the prompt-injection sense, but it is a correctness and trust failure. When the user asks "why did you do that?", the agent reads its own scratchpad and produces a justification that may have nothing to do with the original reasoning. Provenance is a partial mitigation; treating long-running memory as a derived artifact rather than a primary record is a better one.

The cross-session, cross-agent question

The framing "should memory be cross-session/cross-agent or strictly per-thread?" is too binary. The defensible answer is scoped, opt-in, and depends on the deployment.

The case for cross-session memory is real. Without it, the agent forgets project context every session, loses preferences, and repeats the same setup work. Letta's archival memory and shared memory blocks (Letta archival memory, memory blocks) and Anthropic's memory tool (Anthropic memory tool docs) both exist because per-thread isolation has obvious productivity costs. Mem0's claimed gains depend on cross-session retention.

The case against cross-session memory is also real. Every failure mode in the previous section is amplified by persistence: prompt injection becomes durable, contradictions accumulate, identity drift compounds, and stale facts survive. AgentDojo and PoisonedRAG show that the attack paths exist; the question is only whether deployment scope makes them load-bearing. There is also a measurement gap — there is no published evidence base demonstrating that automatic cross-session memory improves real task outcomes on deployments that handle untrusted input enough to justify the security and consistency costs.

The cross-agent variant is harder. Sharing memory across agents lets a swarm avoid redundant work, but it creates a high-bandwidth channel for one compromised agent to influence others. Letta's shared memory blocks can be useful for coordination; they also turn one agent's poisoned memory into every connected agent's poisoned memory. Multi-agent systems that share memory should answer the same question they would for any high-trust shared resource: who can write, who can read, how is provenance preserved, and what is the blast radius of a compromised writer.

The defensible default position:

Per-thread isolation by default. Episodic memory should not leak across threads unless the user opts in.
User-scoped memory with explicit promotion. Cross-session memory should require either user action (# <note> in Claude Code, "remember this" in ChatGPT) or a vetted automatic extractor with conservative defaults.
Project-scoped procedural memory under version control. CLAUDE.md and equivalent files should live in the project, be reviewed by humans, and be diffable. They are code, not chat.
Cross-agent memory only with explicit governance. Shared memory blocks should have a designated owner, a review process for writes that change behavior, and provenance metadata strong enough to support targeted revocation.
Organization-scoped semantic memory under ingestion review. Knowledge stores that index internal documents need source classification, ingestion review, and the same access controls as the underlying documents — not weaker ones.

The position implied by some vendor framings — that memory is by default cross-session and the user can turn it off — gets the burden of proof backwards. Persistence is a capability with security implications; opt-out persistence is a poor default.

Design principles for choosing a memory architecture

The frameworks differ enough that no single architecture is right for all agents. The following principles are derivable from the failure modes above.

Define the trust treatment per surface. Decide for each surface whether the model should read its contents as instruction, as trusted reference, or as untrusted evidence. The decision affects prompt structure, write authority, and review process. Do not let surfaces drift between treatments — a scratchpad that becomes a project rule has changed trust class.

Separate write authority from read access. A user can read everything in their own memory; that does not mean every tool can write to it. Write authority should be more restrictive than read access, and procedural-memory writes should require the highest authority.

Maintain provenance. Every memory entry should carry source, timestamp, and (where applicable) the originating session, tool, or user message. Provenance is what makes targeted invalidation possible after a source is found to be compromised. It is also what makes audits feasible.

Prefer append-with-supersession over mutate-in-place. Versioning is the cheapest defense against contradiction and the cheapest enabler of audit. Tombstones for deletes preserve the fact that something was deleted, which matters for security review.

Bound everything. Size limits, count limits, TTL, and per-scope quotas. The defaults in production systems are routinely too generous. The cost of an aggressive default is small (some entries get evicted that were marginally useful); the cost of an unbounded store is unbounded.

Treat retrieval as ranked candidates, not as truth. Retrieved memories should be presented to the model as candidates with metadata, not as ambient context. Include "no memory was relevant" as a first-class outcome.

Make memory inspectable. A user should be able to enumerate what the agent remembers, in human-readable form, and edit or delete individual entries. Mechanisms that hide memory from the user create both trust and safety problems.

Test memory under contradiction. A memory system's behavior on the happy path tells you almost nothing. The interesting cases are: an old fact contradicts a new fact, an attacker-controlled source contradicts a user instruction, two agents disagree about the same task. Evaluate these cases explicitly.

Do not conflate continuation with memory. Server-side conversation state and KV cache reduce the cost of carrying forward what is already in the prompt. They are not substitutes for explicit decisions about what to remember and what to forget.

What is and is not settled

What the evidence base supports:

The cognitive taxonomy (working/episodic/semantic/procedural) is a useful vocabulary but does not partition the implementation space cleanly. Real systems mix surfaces.
Long context is necessary but not sufficient: in-prompt retrieval failures are well-documented (Liu et al.).
Long-term-memory benchmarks consistently show large gaps between current systems and human performance (LongMemEval, LoCoMo).
Memory is a documented prompt-injection surface (Anthropic Managed Agents memory, PoisonedRAG, AgentDojo).
Specific vendor systems instantiate specific combinations of scope, write authority, and trust treatment, and those combinations matter more than the cognitive labels.

What remains contested or under-evaluated:

Whether extracted-memory layers like mem0 generally outperform alternatives, or whether the reported gains are concentrated in specific evaluation regimes.
Whether the LLM Wiki pattern's curation overhead pays off compared with hybrid RAG and long-context prompting on real workloads.
Whether cross-session memory improves real task outcomes enough to justify the security and consistency costs in deployments where the agent processes any untrusted input.
Whether shared memory across agents is net beneficial outside narrow coordination tasks.

The honest article position is that agent memory is a design space, not a solved ladder of progress. The right architecture is the one whose write authority, scope, retrieval policy, and trust treatment match the deployment's actual requirements — and those requirements differ between a personal coding assistant, a customer-service agent, an enterprise multi-agent system, and a research prototype.

Companion entries

Core theory: - Context Window Economics - Lost in the Middle: Position Effects in Long Context - Working Memory vs Long-Term Memory in Agent Systems - State Surfaces and Trust Boundaries

Practice: - Designing CLAUDE.md and Procedural Memory Files - The Anthropic Memory Tool: File-Backed Episodic Memory - OpenAI Responses API: Conversation State Patterns - MCP Memory Servers: Tool-Mediated External State - Karpathy's LLM Wiki Pattern - Mem0 and Letta: Production Memory Layers - Compaction and Summarization Strategies - Retrieval Augmented Generation: Ingestion and Provenance

Counterarguments and risks: - Memory Bloat and the Eviction Problem - Prompt Injection Through Persistent Memory - Memory Poisoning and Corpus Integrity - Identity Drift in Long-Running Agents - Cross-Session Memory: Continuity vs Contamination - Cross-Agent Memory and Shared State Hazards

Adjacent infrastructure: - KV Cache and Prompt Caching: Inference State vs Memory - Skills as Procedural Memory - Agent Governance and Audit