okibi.ai

Command Palette

Search for a command to run...

Smoke Test the API Commands of an Okibi CLI Before You Release

Last updated: 11/18/2026

AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.

Smoke Test the API Commands of an Okibi CLI Before You Release

Okibi's build checks that each command answers --help and that doctor runs. It does not check that commands return the right data from your API. This guide adds that check: a short script that runs the CLI's nine API operations against a real or staging API with live, uncached reads, checks the values it gets back, and exits nonzero if any check fails.

What you will build

A shell script of 14 checks: five reads whose returned values are asserted, five error cases with their exit codes, three dry-run writes, and a check that no order was created. It prints ok or FAIL for each and exits 1 if any check failed.

smoke.sh  ->  live, uncached reads + exit codes + dry-run writes  ->  ok / FAIL per check  ->  exit 0 or 1

AI Prompt

Using the generated acme-orders CLI, write and run a smoke test that can fail a CI job.

Requirements:
- Read with `--data-source live --no-cache --agent` so a cached answer cannot pass for a live one.
- Assert returned values, not only exit codes: 3 orders on the first page, `ord_1` is `paid`, 2 customers, `cus_1` exists, 3 products.
- Check the error exit codes: invalid enum 1, missing argument 2, not found 3, missing key 4, API conflict 5.
- Cover the write commands with `--dry-run` only: orders create, orders update, customers create. Count the orders before and after and require the count to be unchanged.
- Run the checks once, print one line per check, and exit nonzero if any check failed.
- Prove the script can fail: add one check that must fail and confirm the exit code is 1.
- Run the verification step below before finishing.

Prerequisites

  • The Acme Orders example files, openapi.json (a fictional orders API spec) and server.py (a mock server for it). Download them into your working directory:

    curl -fsSLO https://raw.githubusercontent.com/heterochromatic/okibi-examples/main/acme-orders/openapi.json
    curl -fsSLO https://raw.githubusercontent.com/heterochromatic/okibi-examples/main/acme-orders/server.py
    
  • acme-orders generated from the Acme Orders example spec and built with Go (okibi generate --spec ./openapi.json --name acme-orders --agent; okibi generate --agent does not download the CLI, so wait until okibi status <run-id> --json (<run-id> is build.runId from its output) shows "status": "succeeded", then run okibi pull <run-id> and go build -o ./acme-orders ./cmd/acme-orders inside the project), with the binary at ./acme-orders, ACME_ORDERS_API_KEY exported (export ACME_ORDERS_API_KEY=demo-key; the example server calls the same key ACME_API_KEY) and the Acme Orders example API running (ACME_API_KEY=demo-key python3 server.py --port 8099) with its untouched data.

  • Okibi CLI 0.2.11.

1. Write the script

Save it as smoke.sh in the folder that contains ./acme-orders (the script calls it by that relative path), and export the key first with export ACME_ORDERS_API_KEY=demo-key.

#!/bin/sh
# Smoke test for the acme-orders CLI. Run from the folder that contains ./acme-orders.
# Needs ACME_ORDERS_API_KEY exported and the example API running with its untouched data.
B=./acme-orders
LIVE="--data-source live --no-cache --agent"
fail=0
ok()  { echo "ok    $1"; }
bad() { echo "FAIL  $1"; fail=1; }

# expect_exit WANT LABEL COMMAND...  : the command must exit with WANT
expect_exit() { want=$1; label=$2; shift 2; "$@" >/dev/null 2>&1; got=$?
  [ "$got" = "$want" ] && ok "$label (exit $got)" || bad "$label (exit $got, wanted $want)"; }

# expect_json LABEL PYTHON_EXPRESSION COMMAND... : stdout must be JSON `d` and the expression must be true
expect_json() { label=$1; expr=$2; shift 2; out=$("$@" 2>/dev/null)
  echo "$out" | python3 -c "import json,sys; d=json.load(sys.stdin); sys.exit(0 if ($expr) else 1)" 2>/dev/null \
    && ok "$label" || bad "$label"; }

count() { $B orders list --all $LIVE 2>/dev/null | python3 -c "import json,sys; print(len(json.load(sys.stdin)['results']))"; }
before=$(count)

# reads: live, uncached, with the returned values checked
expect_json "orders list returns 3 live orders"  "d['meta']['source']=='live' and len(d['results'])==3" $B orders list $LIVE
expect_json "orders get ord_1 is paid"           "d['results']['id']=='ord_1' and d['results']['status']=='paid'" $B orders get ord_1 $LIVE
expect_json "customers list returns 2"           "len(d['results'])==2" $B customers list $LIVE
expect_json "customers get cus_1"                "d['results']['id']=='cus_1'" $B customers get cus_1 $LIVE
expect_json "products returns 3"                 "len(d['results'])==3" $B products $LIVE

# errors: exit codes
expect_exit 1 "invalid status is rejected"       $B orders list --status bogus $LIVE
expect_exit 2 "missing order id is a usage error" $B orders get $LIVE
expect_exit 3 "unknown order is not found"       $B orders get ord_999 $LIVE
expect_exit 4 "missing key is an auth error"     env -u ACME_ORDERS_API_KEY $B orders list $LIVE
expect_exit 5 "cancelling a shipped order is an API error" $B orders cancel order ord_2 $LIVE

# writes: dry run only, then prove nothing was created
expect_exit 0 "orders create (dry run)"   $B orders create --customer-id cus_1 --items '[{"product_id":"prd_1","quantity":1}]' --dry-run $LIVE
expect_exit 0 "orders update (dry run)"   $B orders update ord_1 --status paid --dry-run $LIVE
expect_exit 0 "customers create (dry run)" $B customers create --email [email protected] --name Test --dry-run $LIVE
after=$(count)
[ "$before" = "$after" ] && ok "order count unchanged ($before)" || bad "order count changed ($before to $after)"

# set SMOKE_BREAK=1 to add a check that must fail, to prove the script exits nonzero
[ -n "$SMOKE_BREAK" ] && expect_exit 0 "deliberate failure" $B orders get ord_999 $LIVE

[ "$fail" = 0 ] && echo "ALL CHECKS PASSED" || echo "SOME CHECKS FAILED"
exit $fail

2. Run it

sh smoke.sh; echo "exit=$?"
ok    orders list returns 3 live orders
ok    orders get ord_1 is paid
ok    customers list returns 2
ok    customers get cus_1
ok    products returns 3
ok    invalid status is rejected (exit 1)
ok    missing order id is a usage error (exit 2)
ok    unknown order is not found (exit 3)
ok    missing key is an auth error (exit 4)
ok    cancelling a shipped order is an API error (exit 5)
ok    orders create (dry run) (exit 0)
ok    orders update (dry run) (exit 0)
ok    customers create (dry run) (exit 0)
ok    order count unchanged (7)
ALL CHECKS PASSED
exit=0

The orders cancel order ord_2 check expects 5, because ord_2 is shipped and the API answers 409. It changes nothing.

3. Prove it fails when it should

SMOKE_BREAK=1 sh smoke.sh | tail -3; SMOKE_BREAK=1 sh smoke.sh >/dev/null; echo "exit=$?"
ok    order count unchanged (7)
FAIL  deliberate failure (exit 3, wanted 0)
SOME CHECKS FAILED
exit=1

SMOKE_BREAK=1 adds a check that cannot pass. The script prints FAIL and exits 1, so a CI job that runs it fails.

Verify the result

sh smoke.sh >/dev/null; echo "exit=$?"
SMOKE_BREAK=1 sh smoke.sh >/dev/null; echo "exit=$?"

Expected: exit=0 for the first command and exit=1 for the second. In CI, run sh smoke.sh and let its exit code decide the job.

How it works

Each check is a function. expect_exit runs a command and compares its exit code with the one you expect. expect_json runs a command, parses its JSON output and requires a Python expression about it to be true. Both set a fail flag instead of stopping, so one run reports every failure, and the script exits with that flag at the end.

The reads use --data-source live --no-cache, so an answer from the local store or the response cache cannot satisfy a check. The exit codes are the ones the CLI returned for those cases: 1 invalid input rejected by the CLI, 2 a missing argument or flag, 3 not found, 4 authentication, 5 an API error.

Common issues

The checks assume the example data

ord_1 must be paid, ord_2 must be shipped, and the API must hold 7 orders, 2 customers and 3 products. A different API state changes the expected values and the cancel check's exit code.

Dry runs do not test real writes

The three write checks send nothing. The unchanged order count proves that, and nothing more. Test real writes against a staging API that you can reset.

Smoke tests are not generation checks

Okibi's own test stage reported 40 commands checked. That counted help and doctor responses. This script checks the behavior of the nine API commands your agents use. It does not cover the built-in commands such as sync, search, feedback or mcp.


verification:
  status: verified
  tested_at: "2026-11-02"
  product_version: "okibi 0.2.11; acme-orders 1.0.0"
  command: "sh smoke.sh >/dev/null; echo exit=$?; SMOKE_BREAK=1 sh smoke.sh >/dev/null; echo exit=$?"
  expected_result: "exit=0, then exit=1"

Related Articles