Stop Building Opinionated Memory

Every agent memory system picks one retrieval strategy and optimizes for one benchmark. But in production, the same agent needs to recall differently for every query. The right answer is to stop deciding for the agent and let it tune retrieval itself.

engineering agents memory architecture
Stop Building Opinionated Memory

Every agent memory paper published in the last year follows the same pattern: pick a benchmark, optimize retrieval for it, claim state of the art. MAGMA gets 70% on LoCoMo. Zep gets 94.8% on DMR. Mem0 gets 26% improvement over the OpenAI baseline. Each one is impressive. None of them solve the actual problem.

The actual problem is that there is no single right answer for memory.

The Benchmark Trap

Benchmarks measure one thing well. LoCoMo tests multi-session conversational recall. LongMemEval tests long-term interactive memory at scale. DMR tests dialogue memory retrieval. Each benchmark rewards a specific retrieval configuration: a specific embedding strategy, a specific ranking formula, a specific top-k.

So every team optimizes for their chosen benchmark. They hardcode the retrieval strategy that scores highest, publish the paper, and ship the system. The result is opinionated infrastructure: memory systems that are excellent at one kind of recall and mediocre at everything else.

But in production, the same agent needs to recall differently depending on the situation. Within a single conversation, the agent might need:

  • Semantic similarity: “What do we know about this customer’s deployment?”
  • Temporal ordering: “What happened in the last three interactions?”
  • Causal tracing: “What caused this escalation?”
  • Cross-domain pattern matching: “Have we seen this failure mode in another project?”

Four questions. Four retrieval strategies. Four different optimal weight configurations. A system optimized for LoCoMo handles the first case. It fumbles the other three.

Opinionated memory uses one fixed configuration for every query and gets 1 out of 4 right. Agent-tuned memory adapts strategy and weights per query and gets 4 out of 4.

What Your Brain Actually Does

Think about how you remember things. When someone asks “what happened with that deal?”, you think chronologically. When they ask “why did we lose?”, you trace causes. When they ask “have we seen this pattern before?”, you search across domains for structural parallels.

Same brain. Same memories. Completely different retrieval modes. And the weights shift constantly: sometimes recency matters most, sometimes the importance of the memory matters more, sometimes pure relevance is all you need.

Your brain does not have one retrieval configuration. It adapts retrieval to the question being asked. Every benchmark in the world optimizes for one configuration. Your brain optimizes for all of them, on the fly.

The Wrong Split

Most memory systems put the intelligence in the infrastructure. The system decides how to retrieve, how to rank, how to filter. The agent gets back a list of results and works with whatever it gets.

This is backwards. The agent is already the smartest thing in the stack. It understands the query intent. It knows whether the user is asking about recent events or historical patterns. It knows whether recency matters or importance matters. It knows the domain.

Why would you hardcode decisions that the agent is better equipped to make?

Let the Agent Decide

We built HEBBS the other way around. Instead of making the infrastructure smart and the agent dumb, we made the infrastructure tunable and let the agent be smart.

Four recall strategies the agent picks per query:

# Semantic lookup
hebbs recall $VAULT --query "deployment architecture" --strategy similarity

# Timeline reconstruction
hebbs recall $VAULT --query "customer interactions" --strategy temporal --entity-id acme

# Root cause analysis
hebbs recall $VAULT --query "escalation" --strategy causal

# Cross-domain patterns
hebbs recall $VAULT --query "compliance gaps" --strategy analogical

Scoring weights the agent adjusts per call:

# Pure semantic (factual lookup)
--w-relevance 1.0 --w-recency 0.0 --w-importance 0.0 --w-reinforcement 0.0

# Recency-heavy (what changed?)
--w-relevance 0.2 --w-recency 0.7 --w-importance 0.1 --w-reinforcement 0.0

# Importance-heavy (critical decisions)
--w-relevance 0.3 --w-recency 0.1 --w-importance 0.5 --w-reinforcement 0.1

Strategy-specific parameters the agent tunes:

# Wider beam for thorough search
--ef-search 200

# Deeper causal chain traversal
--max-depth 8

# More structural weight for analogical matching
--alpha 0.3

One engine. Same memories. The agent controls how it retrieves on every single call.

The Evidence

We tested this on a real vault: 52 legal documents, 949 indexed memories across contracts, board minutes, vendor assessments, compliance policies, risk registers, and incident reports.

Baseline: All 20 evaluation queries run with default settings (similarity, k=5). Strict keyword matching against top-5 results, no fuzzy matching, no partial credit. Keyword recall: 75%.

After agent-driven tuning: The agent analyzed which queries failed and why, then adjusted strategy, weights, and k per query. Keyword recall: 94%.

The 19 percentage point jump came from four fixes the agent discovered on its own:

  1. Insufficient k (13 of 16 failures): Default k=5 missed supporting propositions. Fix: k=10.
  2. Missing entity names in cues (5 of 16): Generic queries missed entity-associated results. Fix: include entity names.
  3. Wrong strategy for cross-entity queries (2 of 16): Similarity returns single-entity results. Fix: analogical strategy with alpha=0.3.
  4. Missing recency weighting (3 of 16): Default weights ignore temporal ordering. Fix: weights 0.3:0.5:0.2:0.

The agent stored five generalized retrieval strategies as high-importance memories. In subsequent conversations, it recalled those strategies before executing queries, applying domain-tuned parameters automatically.

The whole tuning loop took under 60 seconds.

Why This Matters for Every Use Case

The legal vault agent learned one set of strategies. A sales agent operating on CRM transcripts would learn a completely different set. A coding agent operating on a repository’s documentation would learn yet another.

That is the point. There is no universal memory configuration. Every domain, every corpus, every agent workflow has different retrieval needs. The infrastructure should not pretend to know the right answer. It should expose the controls and let the agent figure it out.

This is why we publish playbooks instead of shipping a black box. Here is how to configure memory for sales agents. Here is how for support. Here is how for coding assistants. Starting points that the agent then tunes further as it learns what works.

The Infrastructure’s Job

If the agent controls retrieval, what does the infrastructure do? It keeps the memory store healthy.

Memories decay over time following the Ebbinghaus forgetting curve. Frequently recalled memories get reinforced. Contradictions are detected and flagged for resolution. Episodes consolidate into insights. Stale facts fade while important patterns strengthen.

These are the maintenance mechanisms that make agent-driven tuning possible. You cannot tune retrieval over a degraded store. If your memory is full of contradictions and stale data, it does not matter how good your query rewriting is. The results will be noise.

Six mechanisms keep the store clean:

  1. Fast encoding: Sub-millisecond ingestion, no blocking
  2. Hebbian association: Embeddings evolve as relationships form
  3. Contradiction detection: Conflicting facts are flagged automatically
  4. Conflict resolution: The agent (or a human) decides which version wins
  5. Consolidation: Raw observations cluster into generalized insights
  6. Adaptive decay: Unreinforced memories fade; recalled memories strengthen

The infrastructure maintains the store. The agent controls the retrieval. That is the split.

The Playbook Pattern

When an agent tunes its retrieval and stores the learned strategies as memories, something interesting happens. Those strategies become part of the memory store. They are subject to the same reinforcement and decay as any other memory.

Strategies that get recalled frequently (because they work) strengthen. Strategies that stop being useful fade. The memory system learns how to be used, and that learning follows the same lifecycle as every other memory in the system.

This creates a self-reinforcing loop:

Agent encounters a query type
  → Recalls stored retrieval strategies
  → Picks the best strategy for this query
  → Executes recall with tuned parameters
  → Gets good results → strategy is reinforced
  → Gets bad results → agent tunes further, stores updated strategy

Over time, each agent builds up a domain-specific retrieval policy that is tailored to its corpus, its users, and its query patterns. No two agents end up with the same policy, because no two agents operate in the same context.

Stop Chasing Benchmarks

The next agent memory paper will claim SOTA on some benchmark by optimizing one more knob in their retrieval pipeline. It will be impressive on that benchmark and mediocre in production.

The alternative is to stop pretending there is one right answer. Build infrastructure that stays healthy. Expose the retrieval controls. Let the agent, the thing that actually understands the query, decide how to use them.

That is what we built. That is how memory should work.