You asked whether insights could run against a local model, and the answer
changes the calculus. I had assessed the Anthropic API, which needs a
credential and sends data off the machine, and embedding a model in the
binary, which costs ~90MB and a dependency. An already-running local server
has none of those: no credential, no per-call cost, nothing leaving the
machine, and nothing beyond the standard library talking to localhost. My
"no LLM" conclusion was right for the options I had in view and stated too
broadly.
Discovers Ollama, LM Studio, or llama.cpp by probing OLLAMA_HOST,
FATHOM_LOCAL_MODEL_URL, then the three default ports, with a short timeout.
Finding nothing is the ordinary case, not an error: there is no warning, no
configuration required, and the tab behaves exactly as it did.
Used for two things:
- Grouping prompts by meaning rather than shared vocabulary. MinHash has a
provable blind spot -- "diagnose the intermittent failure" and "fix the
flaky test" mean the same thing and share no words -- and embeddings close
it. This does affect what gets found, so the method used is shown on every
clustered finding and MinHash is the fallback on any failure.
- Naming a group, so a finding reads "Flaky test triage - asked 10 times"
rather than "You asked for much the same thing 10 times". This was the one
gap I identified as needing semantics and dismissed as not worth a
dependency; at zero cost it is worth it.
The line the model does not cross: it may write a phrase, never a number.
Detection stays entirely heuristic, so occurrence counts, costs, and the
evidence behind them are identical with or without it. A model that could
invent an occurrence count would poison the only thing making this tab worth
trusting. Tests assert the numbers survive labelling, that a failed or
rambling label leaves the heuristic title in place, and that sequence
findings are never sent for naming since their steps already name them.
cleanLabel absorbs what small models actually do -- preambles, reasoning
blocks, quotes, "Common theme:" prefixes -- and rejects anything
sentence-shaped, because a missing label costs polish while a rambling one
would be shown as fact.
Verified end to end against a stand-in server: findings went from "You asked
for much the same thing 10 times" to "Endpoint pagination work - asked 10
times", with the counts unchanged. Verified the toggle reverts detection to
lexical rather than only changing the wording, and that killing the model
server leaves the tab working with no warning.
zzg
bf955e5ca4