Twenty-four seats: giving an LLM a memory

A language model remembers nothing. It rereads.

Everything it "knows" about you sits in the text it is handed again on every turn. While the conversation is short, you hand it everything. After three weeks it no longer fits, and you have to choose.

Écho is a conversational character engine that I host myself. One character, one conversation that lasts, and one requirement: that it recalls what I told it last month without my repeating it. My largest corpus holds 299 memories. On each turn, 24 of them go into the context.

This article takes apart the system that picks those 24. There are two paths: the one that writes a memory, the one that reads it back. And just about every threshold, on both, was wrong before it was measured.

Overview#

WRITE
  message → reply → reflection (JSON) → filters in code → deduplication → SQLite

READ
  message → candidates → semantic scores → selection of 24 → context
                ↑
        vector index (derived, rebuilt in the background)

Three choices hold everything else up.

SQLite is the only truth. Vectors are just an index, recomputable at any time. If the vector service goes down, memory keeps working, less finely.

The model proposes, the code decides. The model extracts candidates. What becomes a memory, what merges, what enters the context: that's deterministic Python, testable without a network.

A turn must never wait for memory. If it is slow or missing, we answer anyway.

Writing#

Reflection comes after the reply#

When a message arrives, the character replies first. Only then does a second model call — the reflection — reread the exchange and return JSON: candidate memories, a summary update, a few bounded state deltas.

The order matters. A reflection that fails cannot fail a reply that has already been delivered.

Each candidate comes with a type (fact, preference, promise, episode), a scope (global, or specific to the current story) and a confidence score.

The confidence filter filtered nothing#

Only candidates at 0.85 and above were kept. The next idea seemed obvious: recover the 0.60–0.85 band, the "maybes", and submit them to me for confirmation.

Before writing the interface, I measured the band. Twenty-four invented exchanges with a deliberate gradient: 6 clear facts, 6 hesitations ("I think", "maybe"), 6 facts only hinted at, 6 bits of small talk. Two draws, capturing all candidates, before the filter.

Empty. Clear facts come out 12 times out of 12, small talk 0 out of 12, hesitations 4 out of 12.

The model doesn't say "I'm 70% sure". When it doubts, it stays silent. And the one hesitation it did emit carried its reservation in the sentence itself, with a confidence of 0.90. Uncertainty is encoded in words, never in a number.

So the 0.85 filter is decorative, and the confirmation interface would have had no customers. Ticket closed, zero lines written.

A point of method, because I was called out on it. My first conclusion rested on a single draw: 0 candidates out of 11 in the band. That refutes nothing. If the true rate were 15%, you would still see zero out of eleven one time in six (0.85¹¹ ≈ 17%). It took the second draw for the bound to drop below the threshold I had set myself.

Asking the model to abstain doesn't work#

A memory must be about the user. That has been in the prompt since day one. Measured: 10% of extracted memories were about the character or the setting.

I hardened the instruction, with an explicit clause: "if nothing concerns the user, return nothing". The rate doubled.

It was the third time I had measured the same thing: an instruction asking the model not to produce doesn't hold. Pushed to fill a field, it fills it.

So the filter moved into code. Same for two other write-time defects, also measured:

• about one reflection in three flipped a negation while copying ("doesn't like X" became "likes X"), and both versions then lived side by side;
• the reflection sometimes filed an attitude of the character as a lasting fact about the user.

Three small lexical detectors, each calibrated on the real cases collected, each with its documented false positives. It isn't elegant. It is decidable, and it can be tested.

Deduplicating: a threshold right up against the true variants#

Without deduplication, a repeated fact gets written every time. I found six wordings of the same fact in a real database. Six seats taken out of twenty-four.

An exact hash doesn't see variants. You need a proximity measure, and it has to be lexical and synchronous: the write path cannot depend on a vector model being available.

The score is the maximum of three measures on normalized text:

• the sequence ratio (difflib), computed in both directions because it isn't symmetric — 0.817 one way, 0.783 the other on a real pair, straddling the threshold;
• the Jaccard of content words, which sees permutations;
• the word-for-word inclusion of the shorter text in the longer.

The threshold, 0.80, wasn't chosen. I read it off 1,326 pairs, every combination of 52 real memories: at 0.80 and above, only true variants; the closest pair of distinct facts sits at 0.72 — same sentence shape, two different facts.

Between 0.72 and 0.80, where do you put the cursor? Not in the middle. The two errors don't cost the same:

• missing a merge costs a repetition;
• merging wrongly erases a nuance, silently.

So the threshold hugs the true variants, not the middle of the interval.

A different number is a different fact#

Then an adversarial bench found this. Two memories of the form:

The contract provides for 10 sessions at €200 each. The contract provides for 3 sessions at €50 each.

Score: 0.963. The second overwrote the first. The correction I had just given was lost, silently.

Digits weren't even words as far as tokenizing went. Adding them wasn't enough: beyond 40 characters, the sequence ratio took over again. The final rule is blunter:

def _score(na, ta, nb, tb):
    ca, cb = _jetons_porteurs(na), _jetons_porteurs(nb)
    plafond = 1.0 if ca == cb else SIMILARITY_THRESHOLD - 0.01
    ratio = 0.0
    if min(len(na), len(nb)) >= MIN_RATIO_CHARS:
        ratio = max(
            difflib.SequenceMatcher(None, na, nb).ratio(),
            difflib.SequenceMatcher(None, nb, na).ratio(),
        )
    jaccard = len(ta & tb) / len(ta | tb) if ta | tb else 0.0
    brut = max(ratio, jaccard, 1.0 if _verbatim_contained(na, nb, ta, tb) else 0.0)
    return min(brut, plafond)

Carrier tokens (jetons porteurs) are the ones that often are the fact itself: digits, days, months, "tomorrow", "tonight". If they differ between the two texts, the score is capped just under the threshold, whatever the other measures say.

That list was built over five rounds of review, each with its measured counter-example (the corpus is French):

• "next Tuesday" merged with "next Thursday" at 0.938;
• "tonight" was overwritten by "tomorrow night" at exactly 0.80;
• splitting "après-demain" (the day after tomorrow) to find "demain" (tomorrow) inside made the two equal, and tomorrow's appointment was destroyed by the day after's;
• "mais" (but) folded into "mai" (May): the most ordinary conjunction in French became a phantom month.

The cost is accepted, on the right side: "14h30" and "14 h 30" make two entries instead of one. One more repetition, never an erased fact.

I had also tried inclusion of words rather than word-for-word. Removed: "A likes B" is a word subset of "A likes discussing his problems with B". A major fact would have been absorbed by a minor one.

Which way the merge goes#

When two memories merge, the most recent wording wins — unless it is poorer. A short variant arriving after a rich one is absorbed without rewriting anything.

And absorption doesn't refresh the memory's date. The date is a ranking key on the read side: a fact I keep repeating would have ended up monopolizing the top of the ranking, ahead of fresh memories.

Reading#

Twenty-four seats, six passes#

On each turn, all active memories are candidates. Selection is a pure function: a list in, 24 memories out, and for each one the reason it is there. That reason is shown on a diagnostics page. Without it, I couldn't have tuned anything.

The passes, in order:

1. The reserve. A quarter of the seats is set aside for global and semantic memories, before anything else.
2. Pinned memories, the ones I marked by hand, up to the limit minus the reserve.
3. Global ones, up to their quarter.
4. Semantic ones, those the last message brings back, up to 16.
5. A quota per type: 6 facts, 5 preferences, 4 promises, 5 episodes.
6. The catch-up: whatever is left, in rank order (pinned, story-specific, type, confidence, recency).

The reserve exists because of a measurement. A story with 26 pinned memories and 10 relevant global memories gave: 24 pinned, 0 global, 0 semantic. Everything lasting the character knew about me had vanished behind the current story.

reserve_candidates = sum(
    1
    for memory in ordered
    if not memory.pinned
    and (memory.scope == "global" or memory.id in semantic_ids)
)
reserve = min(
    max(1, round(limit * GLOBAL_CONTEXT_SHARE)),
    reserve_candidates,
)
for memory in ordered:
    if memory.pinned:
        if len(selected) >= limit - reserve:
            break
        add(memory, "Épinglé")

The reserve is bounded by the real candidates: if there are only two global memories, six seats aren't frozen for nothing.

Second measurement, same mechanism one floor up. There was a priority type, the rules the character must always respect, served before everyone else. One profile had 19 of them. They took 19 of the 24 seats. Its recall had become insensitive to every setting I had.

They left the budget: still all in the context, but no longer competing for a seat. The four freed seats weren't redistributed among the types: they go to the catch-up pass. Nothing told me who should get them, and getting it wrong cost more than waiting for a measurement.

Semantic recall, measured#

Each memory has a vector. The last message is vectorized, compared by cosine to all the others, and those above 0.32 are eligible for the semantic pass.

Four questions came up. I built a bench: 18 reference memories, 6 queries, 10 declared relevant, drowned in distractors up to the target size, over five different distractor sets. The metric is recall@k: are the relevant ones in the top k? That is exactly what the product does.

Task prefixes. The vector model recommends prefixing queries and documents. On 234 memories:

None beats no prefix. Rejected.

The delicious part: the original ticket quoted another metric, the mean gap between relevant and non-relevant, which concluded the opposite. It doesn't measure what the system does.

The backdrop. I was appending the story's title and the current situation to every query, to help elliptical messages like "shall we go on?".

The backdrop is several times the length of the message. It dominates the vector, it is identical on every turn, and it pulls all queries toward the same point. It didn't help where it was meant to: it took recall down to zero. Removed.

The number of semantic seats.

Degradation starts around 200 memories, and it only affects depth: relevant memories never leave the top 24, they leave the semantic seats. Raised from 8 to 16.

Capping the number of candidates. Scoring is pure Python, no vector library: 1.5 ms for 100 memories, 13.4 ms for 1,000, 40.4 ms for 3,000. Against a turn that takes seconds, capping would buy nothing and would make old memories unreachable. Rejected.

Two lessons of method, learned the hard way on this bench:

• My first labeling declared relevance by theme. It concluded that the model didn't discriminate. The labeling was what was wrong: recall@3 went from 10/10 to 9/15 without a single ranking changing.
• On 18 memories, no variant degraded anything. At that size nothing degrades. The effects show up at 234.

Never make a turn wait#

On August 19, between 5:27 and 6:40 a.m., fifteen turns were served with no semantic memory at all. The machine serving the vectors was asleep. And the first turn after each retry window paid two 120-second attempts before giving up.

Since then, the turn only vectorizes its own query, with five seconds and a single attempt. Healthy latency is 0.2 to 0.8 seconds; if the machine is asleep, retrying doesn't wake it.

query_vector = (
    await asyncio.wait_for(
        self.ollama.embed(
            [query.strip()[:2000]],
            model=self.settings.ollama_embedding_model,
            timeout=TURN_EMBEDDING_TIMEOUT,
            attempts=TURN_EMBEDDING_ATTEMPTS,
        ),
        timeout=TURN_EMBEDDING_TIMEOUT,
    )
)[0]

The timeout is written twice, and it isn't a repetition. The one passed to the client only bounds the HTTP request. Before that, the call waits for a semaphore and a lock that background indexing can hold for several minutes. wait_for is what bounds the whole operation.

Indexing itself left the turn. A background loop runs every 45 seconds, compares content fingerprints, and only vectorizes what changed. So a memory created just now isn't scored yet on the next turn: it is still served by the deterministic passes, which sort by recency.

On failure, selection runs without scores, and a degraded-turn counter is incremented. On August 19, only a log line knew what was happening.

Making forgetting measurable#

"Which memory exists but never comes up?" I had no way to answer.

Each memory now carries three counters: how many times it was a candidate, how many times served, and how many times served by the semantic pass. The distinction matters. A memory admitted because it is recent wasn't retrieved, it was just sorted.

Cost: 0.36 ms per turn on a corpus of 300. And a turn during an outage counts nothing: a sleeping machine must not read as an ignored memory.

The loops#

The position that quotes itself#

The character also has positions: what it thinks about a subject, with the sentence that justifies it.

One evening it comes up with a line that fits perfectly. The reflection files it, word for word, as a lasting position. Five hours and 31 messages later, on a trigger with no connection at all, the line comes back out identically.

And a minute later, the reflection re-validates the position by citing as evidence… the line the position had just prompted.

The stored text dictates the line, the line justifies the stored text. The update date is pushed back on every turn: the position can no longer age or give up its place.

Two fixes. On the write side, the field's label literally invited copying ("the position, in its own words"): reworded, copying goes from 4 cases out of 12 to 0 out of 12. On validation, a justification that reuses 30 characters or more of the position it claims to support no longer re-anchors it. Threshold read off the 21 real justifications in the log: the self-citation reused 53, the closest legitimate justification 13.

The summary that forgets#

The story summary was rewritten in full at every reflection. A fact the model stopped repeating disappeared without a trace.

My first idea: anchors. Identified facts, held by the code, that the model can only remove by giving a reason. The free summary would keep only the recent part.

Measured over 39 real reflections: worse than before. One key fact out of four kept at the end of the conversation, against three out of four with the old system. The model promoted almost nothing to an anchor, three times in 39 turns, and anything that lived only in the recent part evaporated on the next turn. The guard worked perfectly. What fed it was too narrow.

The shipped version keeps the full summary and puts the anchors on top, as a safety net. A repetition between the two is accepted.

What I'm keeping#

A threshold is read, not chosen. 0.80 to merge, 30 characters for self-citation, 16 semantic seats: each comes from a real distribution, not from intuition. And the 0.85 filter, chosen by intuition, filtered nothing.

Two errors never cost the same. A repetition is visible and fixable. An overwritten memory is only recovered by luck. The whole write path leans the same way.

You don't ask a model to stay silent. Three measurements, the same result three times. What must be excluded is excluded in code.

Memory degrades, it doesn't block. A slightly less informed reply beats a reply that never arrives. But the degradation has to be counted.

Measure at scale. At 18 memories everything works. At 200 the system starts having an opinion.

The limits, to finish. The lexical detectors are written for French and don't carry over. Several benches run on invented corpora. And all of this serves a single user: me. I don't know what this selection does at ten thousand memories, and neither does the system.

— Pato