Promptster vs a DIY OpenTelemetry dashboard: build vs buy for AI-coding TokenOps

TL;DR

• OpenTelemetry's GenAI semantic conventions are the emerging standard for LLM telemetry: gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.request.model, and friends. With a Collector plus Tempo/Grafana you can build a board showing tokens per model and daily cost.
• It's real and it's cheap in license terms — you own the data and the stack. For an observability team that already runs Grafana, this is legitimate infrastructure.
• Two honest catches. First, as of 2026 the GenAI conventions are still experimental, and coding-agent CLIs (Claude Code, Codex, Cursor, Copilot) don't emit them out of the box — you instrument, or they're dark. Second, a token/cost span has no developer identity, no workflow, and no judgment: it tells you a request cost $0.004, not who ran it, what work it served, or whether it was waste.
• Promptster gives you developer-and-workflow attribution, session reconstruction, waste-vs-leverage signal, and per-engineer coaching — without instrumenting anything or running a collector.
• Pick the DIY board if you have an observability team, want everything in one Grafana, and the goal is raw per-request telemetry. Pick Promptster if the goal is TokenOps: knowing which developer and workflow burns tokens, and cutting the waste.

What the DIY OTel approach is, briefly

The OpenTelemetry GenAI SIG has, since 2024, standardized attribute names for LLM calls: model, input/output token counts, finish reasons, and more. The build looks like this: instrument each LLM call (directly or via an instrumentation library) so it emits a span with the GenAI attributes, run an OpenTelemetry Collector, store spans in Tempo (or metrics in Prometheus), and build Grafana panels. Cost is computed as (input_tokens * input_rate + output_tokens * output_rate) / 1e6 and written onto the span as a custom attribute. Vendors are catching up — Datadog added native support around OTel v1.37, Grafana collects LLM traces in Loki — but the self-hosted Collector-plus-Grafana path is the canonical build-it-yourself route.

What a DIY OTel dashboard does well

• You own the data and the stack. No third party, no vendor lock-in, everything in your own Grafana next to the rest of your telemetry.
• Unification. LLM spans sit beside your app traces, infra metrics, and logs. For an SRE team, one pane for everything is genuinely valuable.
• Per-request granularity. Every instrumented call becomes a span with exact token counts and a computed cost. If you want the raw event stream, this is it.
• Cheap in licensing. Running Tempo + Grafana is a fraction of a managed platform's price. If you already operate this stack, the marginal cost is low.

Where a DIY board stops short

You have to instrument everything, and then maintain it. The GenAI conventions are experimental in 2026, and coding agents don't emit them natively. Getting Claude Code, Codex, Cursor, and Copilot to produce clean, comparable spans is real engineering — proxies, wrappers, instrumentation libraries — and then it's your team's job to keep it working as every one of those tools ships breaking changes. A dashboard is not a one-time build; it's a standing maintenance commitment. Promptster's capture is the product's problem, not yours.

A span has no developer and no workflow. This is the structural gap. gen_ai.usage.output_tokens = 1840 tells you a request was expensive. It does not tell you which engineer ran it, which repo and task it served, or how it fit into the session around it. You can bolt on custom attributes to approximate identity, but you're rebuilding, badly, the attribution model Promptster treats as native — because Promptster reads the session, where the developer and workflow are already there.

Raw metrics aren't a discipline. Tokens-per-model on a Grafana panel is a number. TokenOps is a loop: attribute → surface waste → optimize → operate. A DIY board gives you the first quarter of the first stage and leaves the judgment to you — was that burn a re-read loop or real leverage? Should this workflow use a smaller model? Which engineer needs coaching? Those questions need session reconstruction and an opinion, which a metrics panel does not have and was never meant to have.

No coaching, no privacy model. A dashboard reports to whoever can see it. Promptster gives each engineer a private replay of their own session plus specific coaching — a fundamentally different surface than a shared Grafana board, and one you'd have to design from scratch (including the privacy boundary) if you built it yourself.

Side-by-side comparison

Dimension	DIY OpenTelemetry dashboard	Promptster
Setup	Instrument every agent, run a Collector, build boards	Install the CLI; capture is handled
Maintenance	Yours, ongoing, as tools change	The product's
GenAI conventions	Experimental (2026); not emitted by coding agents natively	N/A — reads the session directly
Developer attribution	Not present; approximate via custom attributes	Native to the session
Workflow reconstruction	None — isolated spans	Prompts + workflow moments, replayable
Waste signal	You interpret raw metrics	Re-read loops, bloated context, redundant runs, oversized models — named
Coaching	Build it yourself	Private per-engineer coaching, included
Data ownership	Fully yours, in your stack	Promptster's capture model
Cost	Low license, high engineering time	Product subscription, near-zero engineering time

When to pick the DIY OTel dashboard

• You have an observability team that already runs Grafana and wants LLM telemetry unified with everything else.
• Data residency or ownership is non-negotiable and it must live entirely in your own stack.
• You want the raw per-request event stream and are comfortable interpreting it yourself.
• Your need is monitoring, not a program. If you just want a daily-cost panel and don't need developer attribution or coaching, a board is enough — and you avoid a subscription.

When to pick Promptster

• You want the TokenOps loop, not just a metric. Attribution to developer and workflow, waste surfaced and named, and a path to cut it — that's the discipline, and a Grafana panel is the first sliver of it.
• You don't want to instrument and babysit collectors across four coding agents that each break their output monthly.
• You need developer-level attribution that actually holds — native to the session, not reverse-engineered from custom span attributes.
• You want engineers coached privately, not just a shared board of numbers. That surface, and its privacy model, is built.

Common questions

Isn't building it ourselves cheaper? On licensing, yes. On total cost, usually not — instrumenting four coding agents, running a Collector, and maintaining boards against tools that ship breaking changes is a standing engineering commitment. And at the end you have raw metrics, not developer attribution or coaching. The honest test: if you'd never build your own APM, you probably don't want to build your own TokenOps stack either.

Can I use both — OTel for infra and Promptster for the coding agents? Yes, and it's sensible. Keep OTel/Grafana for app and infra telemetry you already own; use Promptster for the AI coding session, where developer-and-workflow attribution and coaching are the point. Different jobs, no conflict.

Does Promptster use OpenTelemetry under the hood? Promptster captures the coding session directly through read-only session capture rather than depending on each agent emitting experimental GenAI spans. The GenAI conventions are promising, but as of 2026 the coding agents don't emit them cleanly, which is exactly the instrumentation gap a DIY board has to close and Promptster sidesteps.

Will the GenAI conventions eventually make this trivial? Maybe — if the coding agents adopt them natively and they stabilize. Even then, a standardized token span still won't carry developer identity, workflow, or judgment. The conventions solve format; TokenOps needs attribution and opinion on top, which is the layer Promptster is.

See also

• What is TokenOps? — the discipline a raw dashboard doesn't give you
• What Is TokenOps? Why AI coding spend just became a discipline — the category argument
• TokenOps vs FinOps — where AI-spend attribution sits next to cloud FinOps
• See a Promptster Teams demo — developer-level AI-session telemetry, nothing to instrument