When a rare word beats a vector

When an agent calls recall, esfis has to pick, out of everything it knows, the few items that answer the question. And it has to do it offline and without a large model. To manage that it combines two signals, which do not carry the same weight.

Vectors on their own say everything looks alike

macOS ships sentence-embedding models (NLEmbedding). They are free, they run locally and they capture meaning well. The problem is that the similarities bunch up in a very narrow band: in a store of personal items, everything resembles everything else a little, and something always comes out on top.

So similarity is not used as an absolute value. What gets measured is how far each item stands out from the rest for this particular question: how many standard deviations above the mean. Whatever does not stand out clearly adds nothing.

The words supply the precision

The other half is BM25, the classic ranking method of search engines. A rare word counts for more than a common one: “esfis” or “Vercel” say far more than “is” or “use”. And a word from the question that appears nowhere in the store counts half, so that filler words cannot drown out the one that matters.

The final score is three parts words to one part vectors. On top of that, an item sharing no word at all with the question only counts if the vectors consider it a very clear case; otherwise its score is zero.

Not answering is an answer too

Then comes the selection:

• If the best result does not clear a floor, nothing is returned: the question has nothing to do with what you know.
• Of the rest, only what comes close to the best survives — roughly half its score or more.
• Eight results at most.
• On a tie, the item that has been used more often wins.

An agent handed twenty weak items works worse than one handed three good ones, or none.

Catalan has no model of its own

macOS ships no sentence-embedding model for Catalan. esfis uses the Spanish one, which is the closest, and the lexical half makes up the difference.

The language is determined by the content, not by the Mac's settings: esfis reads the first items in the store and detects their language. The model used is recorded, and if it changes — because the store has become mostly English, say — the vectors are recomputed automatically. Vectors from two different models cannot be compared, and without that check nobody would notice.

One more detail: only the first paragraph of a long text is vectorised. A fact imported from an .md file can run to twenty lines, and a vector of the whole thing would say nothing. The full body already counts on the lexical side.

What we still get wrong

A question about somebody you have never mentioned — “where does Alex live”, say — can come back with two or three weak items. Because the distinctive word appears nowhere, a common word from the question pulls too much weight. Those items score half what a real hit scores, but the floor does not rule them out.

You can see it straight away in Settings › Test, which shows the score of every item and what stayed out. It is the next thing we want to fix, and we will announce it in the changelog when it is done.