Fathom

A self-hosted trace store for AI agents. One Go binary, one data directory, one config file.

go build -o fathom ./cmd/fathom
./fathom serve

That is the whole deployment. No Docker, no Postgres, no ClickHouse, no Node toolchain, no build step for the frontend. Stop the process, tar the data directory, and you have moved the installation. The mental model is Gitea's, deliberately — including the default port, 3000.

Status: working. Storage, OTLP ingest, the query engine, the trace detail view, the web UI, and retention are all built and tested, including end to end against a real Claude Code session. FreeBSD packaging is not done yet.

---

Quickstart

You need Go. On a Mac, brew install go if you do not have it.

1. Build

git clone https://git.z.gd/zzg/fathom.git
cd fathom
go build -o fathom ./cmd/fathom

One binary, no dependencies to install.

2. Run

./fathom serve

Fathom listens on :3000 and keeps everything in ./data. Leave it running.

3. Point Claude Code at it

In a new terminal, paste this block, then start Claude Code from that same terminal:

export CLAUDE_CODE_ENABLE_TELEMETRY=1
export CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1
export OTEL_TRACES_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/json
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:3000
export OTEL_LOG_USER_PROMPTS=1
export OTEL_LOG_TOOL_CONTENT=1
export OTEL_LOG_ASSISTANT_RESPONSES=1

claude

Anything you do in that session gets traced. The settings only apply to that terminal; close it and Claude Code goes back to normal.

Two of those lines matter more than they look:

• CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1 is what turns spans on. Without it you get metrics and logs but no trace tree at all.
• The OTEL_LOG_* lines are what send the prompts and tool payloads. Without them the traces arrive with timings but no content, and the detail view is a bar chart instead of a transcript.

4. Open it

http://localhost:3000

Traces appear a few seconds after each turn. Click one to see the span tree, and click any span to read what actually happened in it.

5. Get told when there is something to see (optional)

Findings you have to remember to check are findings you do not see. Install the /insights skill and a Stop hook that nudges you when something new turns up:

./claude/install.sh

It prints a snippet to add to ~/.claude/settings.json. After that, the end of a turn occasionally says:

You've got insights on your Claude usage — Flaky test triage, asked 10 times ($1.25). Run /insights to learn more.

and otherwise says nothing. It costs about 20ms, makes no API calls, and never blocks. See claude/README.md for what it takes to interrupt you and how to tune it.

Just want to look at the UI?

Generate sample traces without running an agent:

./fathom gen -traces 40

Insights

The Insights tab looks for cost patterns in recent traces: sequences you have repeated often enough that a small script or a throwaway skill would pay for itself, context you are paying for more than once, and spend that produced nothing. Every finding leads with what it cost and ends with the traces it came from, so you can check it rather than take it on faith.

It runs in the background and reads the store through the same API the rest of the UI uses. Turning it off ([insights] enabled = false) removes a tab and changes nothing else.

If a model server is already running on this machine — Ollama, LM Studio, llama.cpp — Fathom finds it and uses it to group prompts by meaning rather than by shared wording, and to name the groups. No credential, no per-call cost, nothing leaving the machine. It is never asked to produce a number or to decide what counts as a finding; every count, cost, and trace is computed the same way with or without it. With nothing installed, the tab works exactly as described above.

---

The problem

Debugging an agent is not the same problem as debugging a web service, and general-purpose APM tools solve the second one. When an agent run goes wrong, the questions are: which tool call produced that nonsense, what was actually in the prompt at that point, how many tokens did the retry burn, and which subagent went off the rails. Those questions are about content and causality, not p99 latency.

Fathom is built for that shape of question. It ingests OpenTelemetry from any standard instrumentation, reassembles each run into a span tree, and renders the tree as something an engineer can read top to bottom.

Design

Ingest is OTLP, and only OTLP

POST /v1/traces and POST /v1/logs accept OTLP/HTTP JSON. There is no native format and no Fathom SDK, which is the point: anything that already speaks OpenTelemetry — Claude Code, OpenLLMetry, the LangChain OTel exporters, any OTel SDK — works with an endpoint change and no code change. A bespoke format would be marginally simpler to parse and would strand every existing instrumentation.

Logs are ingested alongside traces, not instead of them. This matters more than it sounds. In practice the spans carry timing and metadata while the actual substance — the user's prompt, what a tool was passed, what it returned, what the model said back — arrives as OTLP log records. Spans alone give you a bar chart; the logs are what make it a transcript.

Attribute names are read as aliases, not as one vocabulary. Emitters disagree about what to call the same fact: Claude Code reports cost_usd, input_tokens, and tool_name, while an emitter following OpenTelemetry's GenAI conventions reports gen_ai.usage.cost, gen_ai.usage.input_tokens, and gen_ai.tool.name. Ingest and correlation already knew this; the insights analyzer originally did not, so a trace from any other emitter rendered correctly in the trace view and reported no cost, no tokens, and every tool named "tool" — a wrong answer that presents as "no findings" rather than as a bug. The analyzer now resolves each fact through an ordered list of names, Claude Code's first. It is a read-time alias: what an emitter sent stays on disk verbatim, the same way classify() keeps the original span name whatever kind it decides on.

Correlating the two is not a span_id join, and assuming it is produces a broken UI. Emitters routinely stamp every log record in a run with the root span's id, so joining on span_id collapses the entire transcript onto the root and tells you nothing about which step did what. Fathom instead correlates on typed identity keys carried by both sides — a tool call's id links a tool-result record to the tool span that ran it, and a request id links a model response to the LLM span that produced it. The keys are configurable per emitter, because the names differ between instrumentations.

Records that carry no such key are a different case, and get different treatment: a user prompt or a session-level event is about the run as a whole, so for those the root span id they were stamped with is exactly right. Only when that id names nothing in the trace does Fathom fall back to the innermost span whose time window contains the record.

Spans are buffered and flushed on a timer or a count threshold, so ingest never costs an fsync per span.

Storage is columnar files on disk

data/spans/2026-09-15.fathom   columnar chunks
data/spans/2026-09-15.blob     large text, referenced by offset+length
data/logs/2026-09-15.fathom
data/logs/2026-09-15.blob

One shard per table per day. A shard file is a sequence of length-prefixed, CRC'd chunks, each independently decodable and carrying its own local string dictionary. Appending a chunk never rewrites a footer, and a chunk left torn by a crash is detected on open and truncated. Within a chunk, timestamps and enums are fixed-width columns and strings are dictionary-coded uint32.

Large text — prompts, tool payloads, model output — lives in a separate append-only .blob file per shard, referenced by offset and length. The columnar chunks stay small and scan-fast, and a blob is read only when someone actually opens that trace in the UI.

The only index is the date shard. A query scans every shard whose range overlaps its time range, one goroutine per shard, fanning into a single aggregator. This is Scuba's philosophy, and it is more defensible here than in the paper: Scuba needed a cluster to brute-force its volume, and a single machine can scan a day of one team's agent traffic in milliseconds.

Retention is age- and size-based eviction of whole shards by a background goroutine — the unit of deletion is the same as the unit of partitioning, so eviction is an unlink.

Query is a deliberately small grammar

SELECT column, aggregate(column), ...
FROM table
WHERE time >= min_ts AND time <= max_ts [AND condition ...]
GROUP BY column, ...
ORDER BY aggregate(column)
LIMIT n

A mandatory time range, no joins, no subqueries, no window functions. Aggregates are count, sum, avg, min, max, and percentiles. There is nothing to join against — a trace is self-contained, and parent_span_id reconstructs the tree within a single table.

The trace detail view deliberately bypasses this engine. Given a trace_id, a separate code path rebuilds the span tree and lazy-loads blobs. That view is the day-to-day reason the project exists; the aggregate query engine is for the occasional "which tool is burning all my tokens" question. Optimizing the grammar for the second at the cost of the first would be backwards.

The UI ships inside the binary

A single HTML/CSS/JS file embedded with go:embed. No bundler, no npm install, no separate frontend deploy, no version skew between API and UI. Three views, in order of how much they matter: the trace list, the trace detail waterfall, and an aggregate query builder.

---

Alternatives considered

Langfuse — the closest thing to what we want, and genuinely good. Ruled out on operations: it is Docker-first, and running it without containers means standing up Postgres and a Next.js app by hand. That is the exact cost this project exists to avoid.

Arize Phoenix — the nearest existing analog, embedded and Docker-free. Ruled out for being Python. Nothing wrong with Python; it just isn't a single static binary you can drop on a box with no runtime.

Jaeger / Tempo / a general APM — these are the wrong abstraction rather than a bad implementation. They model RPC fan-out between services. An agent run is a deeply nested tree of prompts, tool calls, and subagents where the payloads are the debugging information, and every one of these tools treats large payloads as something to sample away.

SQLite (modernc.org/sqlite) as a metadata layer — tempting for the trace list: one row per trace with root name, duration, status, token count, cost, and the list view becomes an indexed query instead of a scan. Rejected. It buys speed the brute-force scan does not need at this scale, and costs a second source of truth that must be kept consistent with the shards — exactly the class of bug that is miserable to find. If the list view ever gets slow, the fix is to maintain a small per-shard summary, not to introduce a second database.

Git refs as the storage substrate — from a prior spec, storing traces as commits under refs/traces/* with eval annotations as notes and packfile compaction as the LSM analog. Content-addressing would dedupe repeated system prompts nearly for free, which is a real win. Deferred: it adds a git plumbing dependency and buys nothing at single-node scale, where flat files are simpler to implement and faster to reason about. Revisit if multi-writer sync shows up — several people running local agents who want traces to converge without a central server is precisely the problem content-addressed storage is good at.

Distributed / multi-node — Scuba's entire reason for existing, and the one mechanism from that paper deliberately not copied. Fathom borrows Scuba's storage and query ideas at single-machine scale; adopting its topology would reintroduce the operational weight that ruling out Docker was meant to shed.

A native ingest format alongside OTLP — considered and dropped. It is maybe thirty lines to sniff and accept, but it is a second contract to document, test, and keep working, in service of instrumentation nobody has written yet. OTLP is the format that already has clients.

---

Lineage

• Scuba (Wiener et al., VLDB 2013) — per-shard dictionary encoding, no index beyond the time partition, age and space based eviction, a narrow SQL grammar, and a sample-rate column that aggregates compensate for. Adopted; the leaf/aggregator tree is collapsed into goroutines, since there is no network to fan out across.
• Gitea — the deployment story. One binary, one data directory, one config file, port 3000.

Other commands

# What has Fathom found? Human-readable, or JSON / JSONL for other programs.
./fathom insights
./fathom insights --json
./fathom insights --jsonl

# Only what has not been reported before (and mark it reported).
./fathom insights --new

# Stop-hook output for Claude Code. Always exits 0, prints at most one line.
./fathom insights --hook

# Forget what has been reported, so everything is new again.
./fathom insights --reset

# Change the port or the data directory.
./fathom serve -addr :8080 -data /srv/fathom

# Query from the shell. Timestamps are unix microseconds.
./fathom query 'SELECT kind, count(*), p95(duration) FROM spans
                WHERE time >= 1789448033881000 AND time <= 1789534433881000
                GROUP BY kind ORDER BY count(*) DESC'

./fathom version

Configuration is optional; see fathom.ini.example for every setting. FATHOM_DATA_DIR and FATHOM_ADDR override the file, and flags override both.

Everything Fathom owns lives under the data directory. Stop the process, tar the directory, and you have moved the installation.

A note on what the data actually looks like

Two things about real agent telemetry were not obvious from the specification and shaped the implementation:

Content does not arrive on the spans. Spans carry timing and metadata; the prompts, tool payloads, and model responses arrive as separate OTLP log records. A trace store that ingests only /v1/traces can tell you a tool call took 111ms but not what it ran or what came back.

Those records are stamped with the root span's id. Correlating them by span_id therefore attaches the entire transcript to the root node and leaves every interesting span empty — a failure that looks like success until you open a trace. Fathom joins on tool_use_id and request_id instead. See internal/tracetree for the details.

Build order

1. Storage format and write path — done
2. OTLP ingest for traces and logs — done
3. Query engine — done
4. Trace tree reconstruction and blob fetch — done
5. Web UI — done
6. Retention and eviction — done
7. FreeBSD packaging — not started

FreeBSD is the eventual deployment target. No CGO is used anywhere and there are no third-party dependencies, so GOOS=freebsd go build is expected to work as-is, but that is a separate problem and is neither tested nor packaged yet.

License

MIT.