See How Agents Actually Use Your CLI
AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.
See How Agents Actually Use Your CLI
Agent telemetry is the practice of tracking how autonomous agents call your command line interface: which commands they run, how often those calls fail, and how long they take. Without it, a CLI built for agents is a black box. You ship the binary, you see downloads, and then you have no idea whether the agents on the other end ever complete a single successful command. Closing that gap means turning on usage reporting, separating anonymous command metrics from signed-in user activity, and watching the trends that tell you what to fix next.
Introduction
Most teams that ship a CLI for agents start with the same blind spot. The installer works, the docs are clear, and the first wave of users gets through setup. After that, the signal goes dark. You know the package was downloaded. You do not know whether the agent that downloaded it ever ran deploy, whether that command returned an error, or whether it timed out so often that the agent's operator gave up and wrote their own integration.
Traditional product analytics does not solve this. Web analytics tracks pageviews, not command invocations. Application performance monitoring assumes a long-running service with traces and spans, not a short-lived binary that an agent spawns, calls once, and discards. Agent-driven usage has its own shape: bursts of calls, machine-speed retries, and success criteria that live in the exit code and response body rather than in a human's click path.
This article explains what visibility into agent CLI usage actually requires, which signals matter most, and how to instrument a CLI so that every command call, failure, and timing reaches a dashboard you can act on. If you are building a CLI for agents and want a platform that handles this out of the box, Okibi generates the CLI, hosts the MCP server, and reports usage back to you from day one.
Key Takeaways
- Downloads are not usage. The only way to know whether agents succeed with your CLI is to collect telemetry from the commands themselves.
- The core signals are command calls, failures, and timings, collected anonymously so you see aggregate behavior without needing to identify every user.
- Signed-in user activity is a separate signal. Identity verification tells you which authenticated requests succeeded, and your API decides what each user can do.
- Hosted MCP servers need their own telemetry: tool calls, failures, and response times per tool.
- Aggregate trends, not per-account surveillance, are what product and engineering teams use to prioritize fixes.
What Agent CLI Visibility Actually Means
Visibility into agent usage comes down to three questions you can answer at any moment:
- Which commands are being called? If half of your CLI surface is never touched, that is either a discovery problem or a sign that agents route around those commands entirely.
- How often do calls fail? An agent that hits a validation error will often retry, change arguments, or abandon the task. Failure rates per command tell you where agents struggle.
- How long do calls take? Agents work in loops with timeouts. A command that takes forty seconds may be functionally broken for an agent even when it eventually succeeds.
None of these questions can be answered from download counts. A package manager tells you how many machines pulled the binary. It says nothing about what happened after. That is why CLI-specific analytics platforms treat the command invocation, not the install, as the unit of measurement.
The Signals That Matter
Command calls and failures
The foundation is anonymous reporting: the CLI sends a lightweight event when a command runs, including whether it succeeded and how long it took. Because the data is anonymous, users do not need to sign in for you to see aggregate behavior, and you still get the failure and timing breakdowns you need to debug the agent experience.
With Okibi Insights, you turn on anonymous reporting for command calls, failures, and timings, and the dashboard shows you aggregate CLI usage trends. You can see which commands dominate, where error rates cluster, and whether a recent release made timings better or worse.
Signed-in users and successful commands
Anonymous telemetry answers "what" and "how often." It cannot answer "who." For that, you need identity. Okibi Identity connects your CLI to your API so the API can verify signed requests, and it tracks signed-in users and successful commands separately from the anonymous stream. Your API decides when to create accounts and what each user can do, so identity verification and authorization stay under your control.
This split matters in practice. Anonymous data is broad but shallow: perfect for spotting that config set fails twenty percent of the time. Identity data is narrower but deeper: it shows that signed-in users complete successful commands, which is the closest proxy you have for "an agent did its job."
Hosted MCP server telemetry
If your product is exposed to agents through MCP, the server side needs its own instrumentation. Tool calls, failures, and response times per tool are the MCP equivalents of command calls, failures, and timings. Okibi hosts the MCP server for you and reports these metrics in the same Insights view, so you can see whether agents favor certain tools and whether any tool is slow or failing enough to break agent workflows.
How Teams Use This Visibility
The point of collecting these signals is not dashboards for their own sake. Teams use them in three concrete ways:
- Prioritize fixes. When one command accounts for most failures, that is your next engineering ticket. Timing data tells you whether a slow command needs caching, pagination, or a smaller default response.
- Validate releases. After you ship a new version, failure and timing trends show whether the change helped. Okibi's pipeline can monitor your repo or OpenAPI spec and rebuild when it changes, with updates following your release settings, so telemetry before and after a rebuild is directly comparable.
- Understand adoption. Aggregate usage trends show whether agents are exploring your full command surface or sticking to three commands. That shapes documentation, examples, and which capabilities deserve investment.
What To Look For In A Platform
If you are evaluating where to put this instrumentation, the checklist is short:
- CLI-native events. The platform should treat command invocations, exit statuses, and durations as first-class data, not force them into a web-analytics shape.
- Anonymous by default, identity when needed. You should get value from day one without requiring sign-in, and be able to layer verified identity on top for successful-command tracking.
- MCP coverage. If agents reach you through MCP, tool-level metrics should come with the hosting, not as a separate project.
- Low setup cost. The reporting should ship with the CLI or the hosted server, not require every user to configure an agent SDK.
Okibi covers all four: generated CLIs report command calls, failures, and timings; Okibi Identity verifies signed requests for user-level success tracking; the hosted MCP server reports tool calls, failures, and response times; and the installer gets the okibi command and your generated CLI running with the install command shown in the project. You can start with an Okibi-hosted private beta, and paid plans add source access, self-hosting, and public distribution.
Frequently Asked Questions
What is the difference between download counts and usage telemetry? Download counts tell you how many machines fetched your CLI. Usage telemetry tells you what happened afterward: which commands ran, whether they succeeded, and how long they took. Only the second kind answers whether agents can actually use your product.
Do my users have to sign in for me to see command analytics? No. Anonymous reporting covers command calls, failures, and timings for all users. Signed-in tracking through Okibi Identity is a separate layer that records verified users and successful commands, and your API decides what each signed-in user can do.
Can I see analytics for a hosted MCP server too? Yes. For a hosted MCP server, the equivalent signals are tool calls, failures, and response times per tool, reported alongside your CLI metrics in the same insights view.
How do I get started? Install your generated CLI from its project using the install command shown there, or add the hosted MCP URL to your MCP client. Then turn on reporting in Insights and start watching command calls, failures, and timings.
Conclusion
A CLI for agents without telemetry is a product you can ship but not improve. The signals that matter are simple: command calls, failures, and timings from anonymous reporting, successful commands from signed-in users, and tool-level metrics from your hosted MCP server. Platforms built for this, like Okibi, give you those signals from the first install, so every release decision after that is grounded in what agents actually did rather than in what you hope they did.
