Catch Stale Skill Instructions for an Okibi CLI
AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.
Catch Stale Skill Instructions for an Okibi CLI
The generated SKILL.md tells agents which commands exist. If it names a command wrongly or leaves one out, agents follow it anyway. This guide compares the skill against the commands Okibi lists and finds the gaps.
What you will build
A script that lists commands that exist but are missing from the skill, and commands the skill names that do not exist as written.
okibi clis docs -> real commands okibi clis skill show -> commands named in SKILL.md -> diff
AI Prompt
Using the Okibi CLI, check the acme-orders SKILL.md against the real commands. Requirements: - Read the commands with `okibi clis docs acme-orders --json`. - Read the skill with `okibi clis skill show acme-orders --json` and extract the commands it names for orders, customers and products. - Report commands missing from the skill and commands in the skill that do not match a real command. - For anything in the second list, check it with `--help` before calling it stale. Do not run a command you are not sure is read-only. - Say which skill you compared: the saved working skill or the skill installed with a release. - Run the verification step below before finishing.
Prerequisites
-
The Acme Orders mock server,
server.py(a fictional orders API). Download it into your working directory:curl -fsSLO https://raw.githubusercontent.com/heterochromatic/okibi-examples/main/acme-orders/server.py
-
A generated CLI named
acme-orderswith a skill (every generated CLI has one),ACME_ORDERS_API_KEYexported (export ACME_ORDERS_API_KEY=demo-key; the example server calls the same keyACME_API_KEY) and the Acme Orders example API running (ACME_API_KEY=demo-key python3 server.py --port 8099). -
Python 3 and Okibi CLI
0.2.11, logged in.
1. Write the check
Save it as skill_drift.py in your working folder. Joining the parts and skipping empty ones matters, because the products command has no resource part.
import json, re, subprocess, sys
cli = sys.argv[1]
docs = json.loads(subprocess.check_output(["okibi", "clis", "docs", cli, "--json"]))
real = {" ".join(p for p in (c["resource"], c["name"]) if p) for c in docs["commands"]}
skill = json.loads(subprocess.check_output(["okibi", "clis", "skill", "show", cli, "--json"]))["content"]
mentioned = {m.strip() for m in re.findall(rf"`{cli} ((?:customers|orders|products)[a-z -]*?)(?: <|`)", skill)}
print("commands in CLI not named in SKILL.md:", sorted(r for r in real if r not in mentioned))
print("commands in SKILL.md that do not exist:", sorted(m for m in mentioned if m not in real))
2. Run it
python3 skill_drift.py acme-orders
commands in CLI not named in SKILL.md: ['orders cancel order'] commands in SKILL.md that do not exist: []
3. Read the result
orders cancel orderis missing from the skill's Command Reference. This is a real gap. The reference lists create, get, list and update for orders. An agent that reads only the skill will not find the cancel command.- The second line is empty. Every command the skill names exists. The skill lists
acme-orders products, andokibi clis docsreports aproductscommand, so they match. If the CLI named itproducts listinstead, an exact-name check would flagproductsas stale even though both./acme-orders products --agentand./acme-orders products list --agentreturn the three products. That is why this guide checks before calling something stale.
Verify the result
python3 skill_drift.py acme-orders | head -1
Expected: a first line listing orders cancel order. Once you add the cancel command to the skill (with okibi clis skill set and skill publish), it should disappear.
How it works
okibi clis docs lists the commands a release really has, and skill show returns the saved working SKILL.md. The saved skill can be newer than the one an installed copy carries, because an installed copy keeps the skill of the release it came from. To check what agents on an installed copy are told, read ~/.agents/skills/acme-orders/SKILL.md from that install and run the same comparison against it. The regular expression finds commands written as `acme-orders <resource> <verb>` in the skill. It does not parse prose or code blocks written another way.
Common issues
The check is a pattern match
It looks only at the three resource names and the backtick form. A command written in a code block is not seen. Treat the output as a lead, then read the skill.
Check a suspected stale command with --help
When the second line names a command, ask the binary instead of running it: ./acme-orders products --help prints its usage without calling the API. That is safe for any command, including writes, which running it is not. In this guide ./acme-orders products --help and ./acme-orders products list --help both work.
Edits must be released
skill set saves the text, and skill publish releases it. Agents that installed earlier keep the old skill until they reinstall.
The generator can rename commands
An earlier generation of this API had orders cancel <id>, and this one has orders cancel order <id>. After any regeneration, rerun the check.
verification: status: verified tested_at: "2026-11-02" product_version: "okibi 0.2.11; acme-orders 1.0.2" command: "python3 skill_drift.py acme-orders | head -1" expected_result: "a first line listing orders cancel order"
