---
title: "Telemetry Should Close the Improvement Loop"
description: "Telemetry should not end at observability. As software teams add coding agents and agentic workflows, telemetry needs to become the evidence layer that helps products get better."
pubDate: "2026-06-18"
heroImage: "/blog/telemetry-improvement-loop-hero.png"
tags: ["AI Agents", "Telemetry", "Product Engineering"]
---

Telemetry is supposed to make the product better.

That sounds obvious, but it is easy to forget.

Over time, observability became its own industry. That was necessary. Modern software is distributed, asynchronous, and full of partial failures. Teams needed logs, traces, metrics, dashboards, alerts, and ways to understand what happened in production.

But somewhere along the way, the instrument panel became the product.

We made it easier to observe systems, but not always easier to improve them.

The next shift is not another dashboard.

It is turning telemetry into an improvement loop.

LoopKit is the product I am building around that shift: not observability for humans first, but improvement infrastructure for agents first.

The category is not "more telemetry." It is feedback-to-fix infrastructure: production evidence turned into bounded work packets for coding agents, with humans at the approval boundaries.

## Why This Is Urgent Now

AI products are no longer improved only by changing code.

They are improved by changing prompts, tool contracts, retrieval paths, evals, policies, product copy, retries, schemas, and the surrounding application logic.

That creates a new kind of operational gap.

A system can be technically up while still taking the wrong path, calling the wrong tool, giving an unhelpful answer, frustrating the user, or quietly regressing after a prompt or model change.

The teams adopting coding agents and evals already have enough raw material. They have traces, spans, logs, support notes, user feedback, eval runs, CI failures, and GitHub issues.

The bottleneck is not collection.

The bottleneck is converting all of that into a standard improvement packet that an agent can act on and a human can review.

## Observability Is Not the End State

Good observability answers important questions:

- What happened?
- Where did it happen?
- How often is it happening?
- Is the system healthy right now?
- What changed around the time it broke?

Those questions matter. I do not want less visibility.

But visibility is only useful if it changes what the team does next.

If a trace shows that a tool call failed, the product does not improve because someone looked at a span. It improves when that failure becomes a clearer tool contract, a regression eval, a better error message, a safer retry path, or a code change that removes the defect.

If feedback shows that users are dissatisfied, the product does not improve because the feedback was stored. It improves when the feedback is clustered with related traces, turned into a concrete issue, fixed, and verified in the next production window.

Telemetry should not stop at "look at this."

It should help the system say, "fix this, prove it, and watch whether it stays fixed."

<section class="tool-blueprint tool-blueprint--bad" aria-labelledby="observability-dead-end-title">
	<div class="tool-blueprint__intro">
		<p>Current Default</p>
		<h3 id="observability-dead-end-title">Most Telemetry Still Ends in Human Triage</h3>
		<span>The data exists, but the next useful product change is still assembled by hand.</span>
	</div>
	<div class="tool-blueprint__flow" aria-label="Current observability handoff">
		<div class="tool-blueprint__step">
			<b>App</b>
			<span>Errors, traces, feedback, latency, weak outcomes.</span>
		</div>
		<div class="tool-blueprint__arrow" aria-hidden="true">→</div>
		<div class="tool-blueprint__step">
			<b>Telemetry</b>
			<span>Logs, spans, dashboards, eval runs, support notes.</span>
		</div>
		<div class="tool-blueprint__arrow" aria-hidden="true">→</div>
		<div class="tool-blueprint__step tool-blueprint__step--danger">
			<b>Dashboard</b>
			<span>Useful for inspection, weak as an action interface.</span>
		</div>
		<div class="tool-blueprint__arrow" aria-hidden="true">→</div>
		<div class="tool-blueprint__step">
			<b>Human Triage</b>
			<span>Manual clustering, interpretation, and handoff.</span>
		</div>
		<div class="tool-blueprint__arrow" aria-hidden="true">→</div>
		<div class="tool-blueprint__step">
			<b>Maybe Backlog</b>
			<span>Issue quality depends on the person and the moment.</span>
		</div>
	</div>
	<div class="tool-blueprint__contract">
		<div>
			<b>Inspection</b>
			<span>The product tells you what happened, not what to improve.</span>
		</div>
		<div>
			<b>Loose Handoff</b>
			<span>Evidence, issue, eval, and fix plan drift apart.</span>
		</div>
		<div>
			<b>No Proof</b>
			<span>The next telemetry window rarely closes the loop.</span>
		</div>
	</div>
</section>

## Humans Should Not Be the First Readers of Raw Telemetry

Most telemetry products still assume a human is the primary consumer.

That assumption made sense when the only way to interpret production behavior was to put an engineer in front of a dashboard.

But agentic software changes the shape of the workflow.

Coding agents can read structured evidence. They can inspect related traces. They can compare failures across windows. They can draft issue bodies, generate eval candidates, propose patch plans, and check whether the next run improved or regressed.

Humans still matter. More than ever.

But humans should increasingly sit at the review, policy, and approval boundaries, not in the middle of raw trace archaeology.

The product surface should be different:

- agents consume bounded, normalized telemetry windows,
- agents produce human-readable and agent-readable reports,
- humans approve external writes and risky changes,
- agents implement bounded fixes,
- telemetry verifies whether the fix worked.

That is a different product than an observability dashboard.

## The Missing Artifact Is the Improvement Brief

The artifact I want from telemetry is not a chart.

It is an improvement brief.

An improvement brief should tell a coding agent what product pain exists, why it matters, what evidence supports it, what files or systems are likely involved, what eval should be added, and what boundary requires human approval.

In a repo, this could literally be `agent-issue-brief.md`. In an API, it could be the same object shape returned to a coding agent.

It should be specific enough that an agent can start useful work without rediscovering the entire incident.

It should also be readable by a human who wants to understand the state of the product without opening five observability tools.

That brief might include:

- a ranked product pain portfolio,
- repeated failures and affected workflows,
- slow or unreliable tool calls,
- customer feedback and unsatisfied requests,
- regressions after previous fixes,
- telemetry gaps that prevent verification,
- the approval state for GitHub issues, evals, PRs, or deploys.

This is where telemetry becomes operational memory.

Not memory as vague context.

Memory as evidence that makes the next improvement easier.

<section class="tool-blueprint" aria-labelledby="brief-anatomy-title">
	<div class="tool-blueprint__intro">
		<p>Improvement Brief Anatomy</p>
		<h3 id="brief-anatomy-title">The Unit of Work Is the Agent-Readable Brief</h3>
		<span>The brief packages production pain, supporting evidence, and the next verification step into one reviewable artifact.</span>
	</div>
	<div class="tool-blueprint__contract">
		<div>
			<b>Evidence Window</b>
			<span>Project, tenant, source, time range, event count, and raw links.</span>
		</div>
		<div>
			<b>Product Pain</b>
			<span>User-facing failure, affected workflow, frequency, and severity.</span>
		</div>
		<div>
			<b>Diagnosis</b>
			<span>Likely root cause, related traces, and files or systems to inspect.</span>
		</div>
		<div>
			<b>Eval Candidate</b>
			<span>The regression test or measurement that should exist after the fix.</span>
		</div>
		<div>
			<b>Approval Boundary</b>
			<span>What an agent may draft locally and what requires human approval.</span>
		</div>
		<div>
			<b>Verification Signal</b>
			<span>The future telemetry window that proves fixed, regressed, or open.</span>
		</div>
	</div>
</section>

<section class="tool-blueprint" aria-labelledby="telemetry-loop-title">
	<div class="tool-blueprint__intro">
		<p>Improvement Loop</p>
		<h3 id="telemetry-loop-title">Telemetry Should Produce Work, Not Just Visibility</h3>
		<span>The useful loop starts with production signals and ends with a verified improvement window.</span>
	</div>
	<div class="tool-blueprint__flow" aria-label="Telemetry improvement loop">
		<div class="tool-blueprint__step">
			<b>App</b>
			<span>Errors, traces, feedback, latency, weak outcomes.</span>
		</div>
		<div class="tool-blueprint__arrow" aria-hidden="true">→</div>
		<div class="tool-blueprint__step">
			<b>Telemetry</b>
			<span>Bounded evidence window, scoped by project and time.</span>
		</div>
		<div class="tool-blueprint__arrow" aria-hidden="true">→</div>
		<div class="tool-blueprint__step tool-blueprint__step--tool">
			<b>Brief</b>
			<span>Product pain, issue draft, eval candidate, patch direction.</span>
		</div>
		<div class="tool-blueprint__arrow" aria-hidden="true">→</div>
		<div class="tool-blueprint__step">
			<b>Agent Work</b>
			<span>Human-approved issue, eval, PR, or local patch plan.</span>
		</div>
		<div class="tool-blueprint__arrow" aria-hidden="true">→</div>
		<div class="tool-blueprint__step">
			<b>Verification</b>
			<span>Next telemetry window proves fixed, regressed, or still open.</span>
		</div>
	</div>
	<div class="tool-blueprint__contract">
		<div>
			<b>Agent First</b>
			<span>Raw telemetry is normalized for coding agents and automations.</span>
		</div>
		<div>
			<b>Human Control</b>
			<span>Humans review policy, approval, external writes, and merge.</span>
		</div>
		<div>
			<b>Closed Loop</b>
			<span>Every fix should have a later measurement window.</span>
		</div>
	</div>
</section>

## This Is Especially Important for Agentic Products

Agentic products fail differently from traditional software.

They can call the wrong tool with valid JSON. They can take the long path through a workflow. They can recover poorly from a partial failure. They can satisfy a schema while disappointing the user. They can regress because a prompt, tool contract, retrieval path, or policy changed.

Those failures are hard to understand from infrastructure metrics alone.

You need product telemetry:

- What was the user trying to do?
- Which tool did the agent choose?
- What evidence did it have?
- What boundary did it hit?
- Did the outcome satisfy the user?
- Did a previous improvement actually reduce the failure?

This is why I do not think the future observability consumer is mainly a human staring at dashboards.

The first-class consumer is a coding agent or automation that can turn production evidence into the next safe product change.

The human still owns judgment.

The agent should own the repetitive investigation and packaging work.

There are three different records here, and they should not be confused.

The source system remains the operating truth: the trace store, feedback inbox, data lake, CI run, issue tracker, or audit ledger.

The improvement telemetry is a projection for diagnosis and measurement.

The agent artifact is the brief, eval candidate, issue draft, patch plan, or verification report that moves the product forward.

## Bring Your Own Telemetry Is Not Enough

Adapters matter.

Many teams already have traces in Sentry, OpenTelemetry, LangSmith, Langfuse, Phoenix, Braintrust, Cloudflare logs, CI systems, or their own data lakes. A useful improvement system should read those sources.

But "bring your own telemetry" cannot be the whole product.

If a project does not already have a good telemetry store, the improvement loop needs an ingestion point. It needs something as easy to adopt as a Sentry-style event API, but designed around agent consumption instead of dashboard browsing.

That means the product needs both paths:

- adapter mode for teams that already have useful telemetry,
- native ingest for teams that need a telemetry store.

Both paths should produce the same normalized evidence and the same improvement artifacts.

The store is necessary, but it is not the value by itself.

The value is the loop.

That is also why LoopKit can start as an additive layer above existing telemetry.

A team with Sentry, LangSmith, Langfuse, Phoenix, Braintrust, OpenTelemetry, Cloudflare logs, or a data lake should not have to migrate before seeing value. They should connect a source, generate the first improvement packet from a recent window, and decide whether the resulting issue or eval is better than what they would have written by hand.

## Why I Am Building LoopKit

LoopKit is my attempt to make this loop explicit.

The idea is simple:

```text
app -> telemetry -> human/agent-readable report -> improvement work -> verification
```

LoopKit should not be another place where humans go to admire traces.

It should be the place where telemetry, reports, issue briefs, eval candidates, approval state, and verification windows come together so coding agents can improve the product.

The first consumer is Ops Room because it already has the shape of this problem: production telemetry, agent workflows, evals, approval boundaries, GitHub handoff, and a real need to know whether a fix actually improved the product.

That is the right proving ground.

If this works there, the product becomes easier to explain:

add LoopKit to a repo, send or connect telemetry, let agents pull bounded evidence, and get reviewed issues, evals, PR plans, and verification reports that make the product better.

## What LoopKit Is Not

LoopKit is not a replacement for Datadog, New Relic, Sentry, LangSmith, Langfuse, Phoenix, Braintrust, OpenTelemetry, or a warehouse.

Those systems are useful sources of truth.

LoopKit sits above or beside them and asks a narrower question: what should a coding agent do next, what evidence supports it, who must approve it, and how will we verify that the product improved?

It is also not an auto-merge bot.

The goal is not to remove judgment. The goal is to remove repetitive investigation, weak issue handoff, missing evals, and unverified fixes.

## The Product Opportunity

The first wave of observability helped teams inspect software.

The next wave has to help teams improve AI behavior.

The wedge is production AI teams that already have agents, evals, telemetry, GitHub, and customer feedback, but do not have a repeatable way to convert failures into reviewed fixes and measured improvements.

The first product should be simple: generate the first improvement packet from the last production window.

Over time, the product can become the improvement memory for a repo: what failed, what was tried, what was approved, what shipped, and whether the next telemetry window proved it worked.

## The Bar

The bar is not "we collected telemetry."

The bar is:

- Did we find real product pain?
- Did we turn it into a clear improvement artifact?
- Did a coding agent have enough evidence to act?
- Did a human keep control over risky boundaries?
- Did the next telemetry window prove the product improved?

That is the reason to build this kind of product.

Not because observability needs another category.

Because telemetry should close the improvement loop.