Research
We publish how we measure ourselves, what the numbers are, and where we still lose. Internal results are updated as they clear review.
Benchmarks
All runs use the same model for answering (Claude Sonnet 5) and differ only in what goes into the context: the full conversation history, or the top facts Esfis recalls under an 800-token budget.
LongMemEval
500 questions · 5 memory abilitiesLoCoMo
up to 35 sessions · ~300 turns eachBEAM
conversations up to 10M tokensConvoMem
75k QA pairs · 6 memory categories| Benchmark | Esfis | Raw context | Tokens / question | Δ tokens |
|---|---|---|---|---|
| LongMemEval | 84.2% | 71.0% | 790 vs 112k | −99.3% |
| LoCoMo | 74.1% | 68.8% | 760 vs 26k | −97.1% |
| BEAM | 90.6% | 86.3% | 800 vs 1M (truncated) | −99.9% |
| ConvoMem | 95.0% | 72.4% | 640 vs 41k | −98.4% |
Methodology
Ingest as a user would
Each session is fed to the observer in order, with its original timestamps. No benchmark-specific prompts.
Recall under a fixed budget
800 tokens per question, no exceptions. The point is what fits in every request, not what fits in a demo.
Same judge, blind
Answers from both conditions are graded by the benchmark's own judge where one exists, else by a model that never sees which condition produced them.
Publish the failures
Every run ships with the questions we got wrong, by category. Temporal reasoning over superseded facts remains our weakest area.
Technical notes
Open problems
- Knowing when not to recall
- Injecting memory into a question that doesn't need it costs tokens and occasionally misleads. We score abstention and are not yet happy with it.
- Conflicting sources
- Two assistants can be told different things on the same day. Today the newest wins; a better model of trust per source is in progress.
- Forgetting on purpose
- Decay is a heuristic. We want a defensible model of what a person would expect a good assistant to have forgotten.