fathom/README.md

# Fathom

A self-hosted trace store for AI agents. One Go binary, one data directory,
one config file.

```sh
go build -o fathom ./cmd/fathom
./fathom serve
```

That is the whole deployment. No Docker, no Postgres, no ClickHouse, no Node
toolchain, no build step for the frontend. Stop the process, tar the data
directory, and you have moved the installation. The mental model is Gitea's,
deliberately — including the default port, 3000.

Status: **working.** Storage, OTLP ingest, the query engine, the trace detail
view, the web UI, and retention are all built and tested, including end to end
against a real Claude Code session. FreeBSD packaging is not done yet.

---

## Quickstart

You need Go. On a Mac, `brew install go` if you do not have it.

### 1. Build

```sh
git clone https://git.z.gd/zzg/fathom.git
cd fathom
go build -o fathom ./cmd/fathom
```

One binary, no dependencies to install.

### 2. Run

```sh
./fathom serve
```

Fathom listens on **:3000** and keeps everything in `./data`. Leave it running.

### 3. Point Claude Code at it

In a **new terminal**, paste this block, then start Claude Code from that same
terminal:

```sh
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1
export OTEL_TRACES_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/json
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:3000
export OTEL_LOG_USER_PROMPTS=1
export OTEL_LOG_TOOL_CONTENT=1
export OTEL_LOG_ASSISTANT_RESPONSES=1

claude
```

Anything you do in that session gets traced. The settings only apply to that
terminal; close it and Claude Code goes back to normal.

Two of those lines matter more than they look:

- `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1` is what turns spans on. Without it
  you get metrics and logs but no trace tree at all.
- The `OTEL_LOG_*` lines are what send the prompts and tool payloads. Without
  them the traces arrive with timings but no content, and the detail view is a
  bar chart instead of a transcript.

### 4. Open it

<http://localhost:3000>

Traces appear a few seconds after each turn. Click one to see the span tree,
and click any span to read what actually happened in it.

### 5. Get told when there is something to see (optional)

Findings you have to remember to check are findings you do not see. Install the
`/insights` skill and a Stop hook that nudges you when something new turns up:

```sh
./claude/install.sh
```

It prints a snippet to add to `~/.claude/settings.json`. After that, the end of
a turn occasionally says:

> You've got insights on your Claude usage — Flaky test triage, asked 10 times
> ($1.25). Run /insights to learn more.

and otherwise says nothing. It costs about 20ms, makes no API calls, and never
blocks. See [claude/README.md](claude/README.md) for what it takes to interrupt
you and how to tune it.

### Just want to look at the UI?

Generate sample traces without running an agent:

```sh
./fathom gen -traces 40
```

### Insights

The **Insights** tab looks for cost patterns in recent traces: sequences you
have repeated often enough that a small script or a throwaway skill would pay
for itself, context you are paying for more than once, and spend that produced
nothing. Every finding leads with what it cost and ends with the traces it came
from, so you can check it rather than take it on faith.

It runs in the background and reads the store through the same API the rest of
the UI uses. Turning it off (`[insights] enabled = false`) removes a tab and
changes nothing else.

If a model server is already running on this machine — Ollama, LM Studio,
llama.cpp — Fathom finds it and uses it to group prompts by meaning rather than
by shared wording, and to name the groups. No credential, no per-call cost,
nothing leaving the machine. **It is never asked to produce a number or to
decide what counts as a finding**; every count, cost, and trace is computed the
same way with or without it. With nothing installed, the tab works exactly as
described above.

---

## The problem

Debugging an agent is not the same problem as debugging a web service, and
general-purpose APM tools solve the second one. When an agent run goes wrong,
the questions are: which tool call produced that nonsense, what was actually in
the prompt at that point, how many tokens did the retry burn, and which subagent
went off the rails. Those questions are about *content and causality*, not p99
latency.

Fathom is built for that shape of question. It ingests OpenTelemetry from any
standard instrumentation, reassembles each run into a span tree, and renders the
tree as something an engineer can read top to bottom.

## Design

### Ingest is OTLP, and only OTLP

`POST /v1/traces` and `POST /v1/logs` accept OTLP/HTTP JSON. There is no native
format and no Fathom SDK, which is the point: anything that already speaks
OpenTelemetry — Claude Code, OpenLLMetry, the LangChain OTel exporters, any OTel
SDK — works with an endpoint change and no code change. A bespoke format would
be marginally simpler to parse and would strand every existing instrumentation.

**Logs are ingested alongside traces, not instead of them.** This matters more
than it sounds. In practice the spans carry timing and metadata while the actual
substance — the user's prompt, what a tool was passed, what it returned, what
the model said back — arrives as OTLP log records. Spans alone give you a bar
chart; the logs are what make it a transcript.

**Attribute names are read as aliases, not as one vocabulary.** Emitters
disagree about what to call the same fact: Claude Code reports `cost_usd`,
`input_tokens`, and `tool_name`, while an emitter following OpenTelemetry's
GenAI conventions reports `gen_ai.usage.cost`, `gen_ai.usage.input_tokens`,
and `gen_ai.tool.name`. Ingest and correlation already knew this; the insights
analyzer originally did not, so a trace from any other emitter rendered
correctly in the trace view and reported no cost, no tokens, and every tool
named "tool" — a wrong answer that presents as "no findings" rather than as a
bug. The analyzer now resolves each fact through an ordered list of names,
Claude Code's first. It is a read-time alias: what an emitter sent stays on
disk verbatim, the same way `classify()` keeps the original span name whatever
kind it decides on.

Correlating the two is not a `span_id` join, and assuming it is produces a
broken UI. Emitters routinely stamp every log record in a run with the *root*
span's id, so joining on `span_id` collapses the entire transcript onto the root
and tells you nothing about which step did what. Fathom instead correlates on
typed identity keys carried by both sides — a tool call's id links a tool-result
record to the tool span that ran it, and a request id links a model response to
the LLM span that produced it. The keys are configurable per emitter, because
the names differ between instrumentations.

Records that carry no such key are a different case, and get different
treatment: a user prompt or a session-level event is about the run as a whole,
so for those the root span id they were stamped with is exactly right. Only when
that id names nothing in the trace does Fathom fall back to the innermost span
whose time window contains the record.

Spans are buffered and flushed on a timer or a count threshold, so ingest never
costs an fsync per span.

### Storage is columnar files on disk

```
data/spans/2026-09-15.fathom   columnar chunks
data/spans/2026-09-15.blob     large text, referenced by offset+length
data/logs/2026-09-15.fathom
data/logs/2026-09-15.blob
```

One shard per table per day. A shard file is a sequence of length-prefixed,
CRC'd chunks, each independently decodable and carrying its own local string
dictionary. Appending a chunk never rewrites a footer, and a chunk left torn by
a crash is detected on open and truncated. Within a chunk, timestamps and enums
are fixed-width columns and strings are dictionary-coded `uint32`.

Large text — prompts, tool payloads, model output — lives in a separate
append-only `.blob` file per shard, referenced by offset and length. The
columnar chunks stay small and scan-fast, and a blob is read only when someone
actually opens that trace in the UI.

**The only index is the date shard.** A query scans every shard whose range
overlaps its time range, one goroutine per shard, fanning into a single
aggregator. This is Scuba's philosophy, and it is *more* defensible here than in
the paper: Scuba needed a cluster to brute-force its volume, and a single
machine can scan a day of one team's agent traffic in milliseconds.

Retention is age- and size-based eviction of whole shards by a background
goroutine — the unit of deletion is the same as the unit of partitioning, so
eviction is an `unlink`.

### Query is a deliberately small grammar

```sql
SELECT column, aggregate(column), ...
FROM table
WHERE time >= min_ts AND time <= max_ts [AND condition ...]
GROUP BY column, ...
ORDER BY aggregate(column)
LIMIT n
```

A mandatory time range, no joins, no subqueries, no window functions.
Aggregates are count, sum, avg, min, max, and percentiles. There is nothing to
join *against* — a trace is self-contained, and `parent_span_id` reconstructs
the tree within a single table.

**The trace detail view deliberately bypasses this engine.** Given a `trace_id`,
a separate code path rebuilds the span tree and lazy-loads blobs. That view is
the day-to-day reason the project exists; the aggregate query engine is for the
occasional "which tool is burning all my tokens" question. Optimizing the
grammar for the second at the cost of the first would be backwards.

### The UI ships inside the binary

A single HTML/CSS/JS file embedded with `go:embed`. No bundler, no `npm
install`, no separate frontend deploy, no version skew between API and UI.
Three views, in order of how much they matter: the trace list, the trace detail
waterfall, and an aggregate query builder.

---

## Alternatives considered

**Langfuse** — the closest thing to what we want, and genuinely good. Ruled out
on operations: it is Docker-first, and running it without containers means
standing up Postgres and a Next.js app by hand. That is the exact cost this
project exists to avoid.

**Arize Phoenix** — the nearest existing analog, embedded and Docker-free. Ruled
out for being Python. Nothing wrong with Python; it just isn't a single static
binary you can drop on a box with no runtime.

**Jaeger / Tempo / a general APM** — these are the wrong abstraction rather than
a bad implementation. They model RPC fan-out between services. An agent run is a
deeply nested tree of prompts, tool calls, and subagents where the payloads *are*
the debugging information, and every one of these tools treats large payloads as
something to sample away.

**SQLite (`modernc.org/sqlite`) as a metadata layer** — tempting for the trace
list: one row per trace with root name, duration, status, token count, cost, and
the list view becomes an indexed query instead of a scan. Rejected. It buys
speed the brute-force scan does not need at this scale, and costs a second
source of truth that must be kept consistent with the shards — exactly the class
of bug that is miserable to find. If the list view ever gets slow, the fix is to
maintain a small per-shard summary, not to introduce a second database.

**Git refs as the storage substrate** — from a prior spec, storing traces as
commits under `refs/traces/*` with eval annotations as notes and packfile
compaction as the LSM analog. Content-addressing would dedupe repeated system
prompts nearly for free, which is a real win. Deferred: it adds a `git` plumbing
dependency and buys nothing at single-node scale, where flat files are simpler
to implement and faster to reason about. **Revisit if** multi-writer sync shows
up — several people running local agents who want traces to converge without a
central server is precisely the problem content-addressed storage is good at.

**Distributed / multi-node** — Scuba's entire reason for existing, and the one
mechanism from that paper deliberately not copied. Fathom borrows Scuba's
storage and query ideas at single-machine scale; adopting its topology would
reintroduce the operational weight that ruling out Docker was meant to shed.

**A native ingest format alongside OTLP** — considered and dropped. It is maybe
thirty lines to sniff and accept, but it is a second contract to document, test,
and keep working, in service of instrumentation nobody has written yet. OTLP is
the format that already has clients.

---

## Lineage

- **Scuba** (Wiener et al., VLDB 2013) — per-shard dictionary encoding, no index
  beyond the time partition, age and space based eviction, a narrow SQL grammar,
  and a sample-rate column that aggregates compensate for. Adopted; the
  leaf/aggregator tree is collapsed into goroutines, since there is no network
  to fan out across.
- **Gitea** — the deployment story. One binary, one data directory, one config
  file, port 3000.

## Other commands

```sh
# What has Fathom found? Human-readable, or JSON / JSONL for other programs.
./fathom insights
./fathom insights --json
./fathom insights --jsonl

# Only what has not been reported before (and mark it reported).
./fathom insights --new

# Stop-hook output for Claude Code. Always exits 0, prints at most one line.
./fathom insights --hook

# Forget what has been reported, so everything is new again.
./fathom insights --reset

# Change the port or the data directory.
./fathom serve -addr :8080 -data /srv/fathom

# Query from the shell. Timestamps are unix microseconds.
./fathom query 'SELECT kind, count(*), p95(duration) FROM spans
                WHERE time >= 1789448033881000 AND time <= 1789534433881000
                GROUP BY kind ORDER BY count(*) DESC'

./fathom version
```

Configuration is optional; see `fathom.ini.example` for every setting.
`FATHOM_DATA_DIR` and `FATHOM_ADDR` override the file, and flags override both.

Everything Fathom owns lives under the data directory. Stop the process, tar
the directory, and you have moved the installation.

## A note on what the data actually looks like

Two things about real agent telemetry were not obvious from the specification
and shaped the implementation:

**Content does not arrive on the spans.** Spans carry timing and metadata; the
prompts, tool payloads, and model responses arrive as separate OTLP log
records. A trace store that ingests only `/v1/traces` can tell you a tool call
took 111ms but not what it ran or what came back.

**Those records are stamped with the root span's id.** Correlating them by
`span_id` therefore attaches the entire transcript to the root node and leaves
every interesting span empty — a failure that looks like success until you open
a trace. Fathom joins on `tool_use_id` and `request_id` instead. See
`internal/tracetree` for the details.

## Build order

1. Storage format and write path — done
2. OTLP ingest for traces and logs — done
3. Query engine — done
4. Trace tree reconstruction and blob fetch — done
5. Web UI — done
6. Retention and eviction — done
7. FreeBSD packaging — not started

FreeBSD is the eventual deployment target. No CGO is used anywhere and there
are no third-party dependencies, so `GOOS=freebsd go build` is expected to work
as-is, but that is a separate problem and is neither tested nor packaged yet.

## License

MIT.