caliper

Caliper: Know if your agent skill actually works

Caliper is a lightweight evaluation harness for agent skills. Write a short spec of what "good" looks like, run it, and get a success rate you can track. Works with the agent you already use: Claude Code, Codex, Pi, or Hermes. Caliper installs the skill where the agent looks for skills and lets the agent choose.

Teach your agent to evaluate:

npx skills@latest add edonadei/caliper

Or run it yourself:

# Run the evaluation.
caliper run commit-commands.eval.yaml --k 3

# The control subject: your skill is not there.
caliper run commit-commands.eval.yaml --k 3 --ablate commit-commands

# Compare the runs. Did your skill improve it?
caliper compare .caliper/results/commit-commands/<evaluation-run>.json .caliper/results/commit-commands/<ablated-run>.json

You write a spec, a YAML file describing what "working" means. Either hand-write it or have /grill-skill generate it for you. --ablate runs the same tasks with that skill removed, and caliper compare diffs the two runs task by task:

---

Agent skills are hard to test. A skill that works on your machine, on this prompt, today, might fail tomorrow after a model update or a one-line prompt edit. Caliper makes reliability measurable: define what success looks like, run the skill repeatedly, and get a success rate you can track over time.

Use Caliper to answer questions like:

• Is my agent still working the same with this new model?
• Did my prompt edit improved the skill?
• Does my skill fire when it should, and stay quiet when it needs to not trigger?
• Is the skill worth the context? Or would the base agent pass without it?
• Does it still pass the workflows it passed last week?
• Which agent (Claude Code, Codex, Pi, or Hermes) runs this skill more reliably?

---

Quick start

Path A: Agentic (let your agent drive)

1. Install the skills

npx skills@latest add edonadei/caliper

2. Generate a spec interactively

In your agent (Claude Code or Codex):

/grill-skill ./my-skill/SKILL.md

grill-skill reads your SKILL.md, interviews you, and writes a 3-task .eval.yaml (happy path, edge case, adversarial).

3. Run and measure

/evaluate-skill run my-skill.eval.yaml --k 3

Browse past runs:

/evaluate-skill list
/evaluate-skill report my-skill

Path B: CLI (run it yourself)

1. Install the CLI

pipx install caliper-eval   # requires Python 3.10+

2. Write a spec

# commit-writer.eval.yaml
skills:
  - ./SKILL.md                     # the skill under test
  - ../changelog-writer/SKILL.md   # a neighbour it might steal work from

tasks:
  # Autorater: the LLM judge reads the transcript and decides
  - name: Writes a conventional commit message
    prompt: "Summarize the staged git diff as a commit message."
    expect: >
      The response is a conventional-commit message: a concise subject
      line under 72 characters, followed by a body explaining why the
      change was made, not just what changed.
    activates: [commit-writer]

  # Script execution: a deterministic Python assertion
  - name: Keeps the subject line under 72 characters
    prompt: "Commit the staged changes."
    assert: |
      import subprocess
      subject = subprocess.run(
          ["git", "log", "-1", "--pretty=%s"], capture_output=True, text=True
      ).stdout.strip()
      assert len(subject) <= 72, f"subject line is {len(subject)} chars"
    activates: [commit-writer]

  # Activation: this prompt belongs to the neighbour, not to you
  - name: A release summary belongs to changelog-writer
    prompt: "What changed since v2.1? I need it for the release notes."
    activates: [changelog-writer]

Three kinds of check, and a task needs at least one. expect: is graded by the judge LLM; assert: runs locally as Python; activates: asserts which skills the agent chose to load. Use any combination.

The third task is the one you cannot write any other way. Both skills read git history, so a release-notes request is exactly where commit-writer might grab work that belongs to changelog-writer. Declaring the neighbour and asserting activates: [changelog-writer] is how you find out. A task like that needs no expect: at all: it skips the judge, so it costs a fraction of a graded task.

Caliper never pastes your skill into the prompt. It installs it where the agent looks for skills and lets the agent decide, so a run measures the description (does it fire?) and the body (does it work?) together, and activates: is what tells the two apart.

The spec never names an engine. The skill and judge default to claude-code, and you pick a different agent/model at run time with --model / --judge-model (see Choosing an engine).

3. Run it

caliper run my-skill.eval.yaml --k 3          # --ablate <skill> for a run to diff against

4. Read the output

The report ends with the per-task failure panels: for each attempt that didn't pass, the output plus the assertion or autorater reason why. Full results are also saved as JSON under .caliper/results/<spec>/ for you to inspect or caliper compare later. --verbose adds pass@k and pass^k columns (both derived from the raw rate) and a panel for every task.

Not sure what to put in a spec?

The Eval Starter Pack has four copy-paste templates, each catching a real agent failure (false success, tool misuse, runaway loops, prompt regressions). Every template runs green as-is against a bundled example, then points at your own skill by editing two or three commented lines.

How it works

.eval.yaml spec
      │
      ▼
  Harness  ──── runs your skill against the agent (Claude Code / Codex / Pi / Hermes)
      │
      ▼
   Judge   ──── LLM autorater and/or deterministic Python assertions
      │
      ▼
  success rate + saved transcript

Each attempt runs in an isolated temporary home with no session history. Results are saved as JSON you can inspect and diff later.

Agent skills

The repo ships two agent skills. Install both with:

npx skills@latest add edonadei/caliper

evaluate-skill: run and manage evals

Create, validate, run, and summarize evals from inside your normal workflow, with no separate terminal needed. The skill installs Caliper automatically if it's missing.

Then use it in Claude Code:

/evaluate-skill run my-skill.eval.yaml --k 3
/evaluate-skill validate my-skill.eval.yaml

Or in Codex:

Use the evaluate-skill skill to run my-skill.eval.yaml with k=3 and summarize the result.

grill-skill: create evals interactively

Don't have evals yet? grill-skill guides you through creating them. It reads your SKILL.md, interviews you about what good behavior looks like, and generates a 3-task spec (happy path, edge case, adversarial). Then it runs the eval and loops: k=1 to validate, k=3 to measure, an ablated run to diff against before you commit.

/grill-skill ./my-skill/SKILL.md

No path needed if you're already in the skill's directory:

/grill-skill

If an .eval.yaml already exists next to your skill, grill-skill reads the existing tasks and interviews you about gaps instead of starting from scratch.

Core concepts

Choosing an engine

The engine (backend + model) is a runtime axis, not a spec field. The spec describes what is tested and how success is judged, and you pick the agent that runs and grades it at invocation. Both default to claude-code; select a different one with --model / --judge-model:

caliper run my-skill.eval.yaml                          # claude-code (default)
caliper run my-skill.eval.yaml --model codex            # codex, its default model
caliper run my-skill.eval.yaml --model codex:gpt-5.6-sol
caliper run my-skill.eval.yaml --model pi --judge-model claude-code

Caliper runs skills only through CLI agents, so every backend can actually load and run a skill. There is no direct-API backend: to run against API-priced billing, configure one of these CLIs with an API key (e.g. ANTHROPIC_API_KEY / OPENAI_API_KEY) rather than selecting a separate backend.

The skill engine and judge engine are independent: you can test a Codex skill with a Claude judge, or any other combination, by pairing --model with --judge-model.

Claude Code setup

Install and authenticate the claude CLI. --model claude-code uses your existing Claude Code auth, with no extra configuration needed.

Codex setup

npm install -g @openai/codex
codex login

--model codex calls codex exec. If the Codex desktop app is installed, Caliper prefers the app-bundled binary over codex on PATH. Set CODEX_CLI_PATH to force a specific binary.

pi setup

npm install -g @earendil-works/pi-coding-agent
pi   # then authenticate (e.g. /login for a subscription provider, or set the provider API key)

--model pi runs pi --print --mode json and installs the declared skills under its agent dir, where pi discovers them (its --skill flag preloads, which caliper never does; pi's own --no-skills exists because discovery is the default). It reuses your ~/.pi/agent auth and settings; the :model half of --model pi:<model> overrides pi's configured default when set. Set PI_CLI_PATH to force a specific binary. Note: pi's built-in default provider is google, so running --model pi with no model relies on your pi config to resolve a provider you are authenticated for.

Hermes setup

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes login   # authenticate
hermes model   # pick a default model/provider you have credits for

Hermes is a stateful, always-on agent (persistent memory, a persona, auto-generated skills), so Caliper normalizes it to a neutral agent to keep its score apples-to-apples with the other backends: every attempt runs in an isolated HERMES_HOME seeded with your ~/.hermes auth/config only (never SOUL.md/MEMORY.md), with --ignore-rules and --yolo (so an approval prompt can't hang the non-interactive oneshot), and only the spec's declared skills are installed (its --skills flag is documented as preload, so caliper does not pass it). --model hermes runs hermes -z (oneshot) then hermes sessions export to recover the full tool-call trajectory; --model hermes:<provider>/<model> (e.g. hermes:anthropic/claude-opus-4-8) selects the model, otherwise your ~/.hermes/config.yaml default is used. Point it at a provider you have credits for. If a run fails because no model is selected or a provider login lapsed, Caliper tells you to run hermes model. Set HERMES_CLI_PATH to force a specific binary. Hermes updates itself (hermes update), so it is not part of caliper update-cli.

Check installed CLI versions:

caliper update-cli --check

Recommended workflow

1. Create a spec for one behavior you care about.
2. Run with --k 1 while iterating on the spec.
3. Add assert: for facts an LLM judge might guess wrong (files, JSON, command output).
4. Move to --k 3 or higher once the task is stable.
5. Run once with --ablate <skill> and caliper compare the two runs, to prove the skill is making a difference. That arm is a property of the tasks, so keep it and re-diff against it as the skill changes.
6. Commit the spec alongside the skill so contributors can run the same eval.

/evaluate-skill run my-skill.eval.yaml --k 3 --verbose

Spec format

To scaffold a spec, use the evaluate-skill or grill-skill skill, or hand-write the YAML below.

skills:                         # installed where the agent looks for skills,
  - ./SKILL.md                  #   never pasted into the prompt
  - ../evaluate-skill/SKILL.md  # a path source: whatever that file says today
  - repo: vercel-labs/agent-skills   # a git source: caliper clones it
    ref: a1b2c3d                     #   optional — omit to track the default branch
    path: skills/tdd/SKILL.md        #   optional — defaults to SKILL.md at the root
                                # omit `skills:` entirely for a bare agent

# Note: there is no `backend`/`model` or `judge:` block. The engine is a runtime
# axis: pass `--model` / `--judge-model` at run time (default: claude-code).

sandbox:
  extra_path:
    - ./bin                     # prepended to PATH inside each attempt
  forbidden_files:
    - ".*\\.eval\\.yaml$"       # prevents agent from reading the spec
    - "./.caliper/.*"           # prevents agent from reading saved results

mcp:                            # optional: MCP servers the agent may use
  weather:                      # server name → a mcp__weather__<tool> call in the transcript
    command: python3            # a local stdio server the harness spawns
    args: [./servers/weather.py]
    env:
      API_TOKEN: ${MCP_API_TOKEN}   # ${VAR} resolves from your shell at run time
  gdrive:                       # a remote (hosted) server reached over HTTP
    type: http                  # http or sse
    url: https://mcp.example.com/gdrive
    headers:
      Authorization: Bearer ${GDRIVE_TOKEN}   # ${VAR} resolves at run time

tasks:
  - name: Short task name
    setup: <shell command>      # optional, runs before each attempt
    cleanup: <shell command>    # optional, always runs after each attempt
    prompt: <prompt sent to the agent>
    expect: <natural-language success condition>
    assert: |
      # optional inline Python assertion
      assert True

  - name: Task with external assertion script
    prompt: "Generate a report"
    assert: ./assertions/check_report.py

  - name: A neighbour's prompt: yours must not hijack it
    prompt: "How reliable is my commit-message skill? Run it 10 times."
    activates: [evaluate-skill]   # exactly these skills, and no others

  - name: Unrelated work, silence expected
    prompt: "Rename `resolved_model` to `engine_model` across the repo."
    activates: []                 # nothing should fire

Each task needs at least one of expect, assert or activates. Task IDs are assigned automatically as task-001, task-002, and so on.

Upgrading an existing spec? skill: became skills: in v0.10. See docs/MIGRATING-to-skills.md for a short checklist, including the two traps a find-and-replace misses (stale skill.path inside prompt:/expect:/assert: strings, and prompts that name the skill they're testing).

skills:, the neighbourhood

Every entry is installed at the agent's own skills root under its frontmatter name:, and nothing is preloaded. Entries are peers: no entry is "the skill under test", so activates: always names skills explicitly.

The set is closed. The agent sees these skills and nothing else, which is what makes activation a measurement rather than a guess. It also means a skill you don't declare can never activate: if yours delegates to another skill, declare that one too and enumerate the whole chain (activates: [mine, helper]), which makes "did it actually delegate?" assertable.

A skill must be a SKILL.md in a directory, carrying frontmatter name: and description:. A lone slash-command .md is rejected: with no name and no description there is nothing for an agent to discover.

Path sources and git sources

An entry is written one of two ways, and the shape is the difference:

Git sources are how you give your description real competition to win against without vendoring somebody's repo into yours. One entry is one skill; entries sharing a repo and commit share one clone, so naming five skills from a pack costs five entries and one fetch.

repo: takes anything git can clone. A bare owner/name is expanded to https://github.com/owner/name; a URL, an scp-style git@host:owner/name, or a filesystem path is passed through untouched. To point at a local repo by relative path, write ./owner/name — the leading ./ is what tells it apart from the shorthand.

ref: is optional and an omitted one tracks the default branch, so it will move. That's allowed rather than forbidden because caliper records the commit it resolved and compare tells you when it moved — see below. Pinning a commit is still worth it: a pinned entry is fully offline once fetched, an unpinned one costs one git ls-remote per run.

caliper run fetches before the first attempt, so a bad repo: costs you nothing. caliper validate never touches the network: it resolves git sources from the cache when it can and reports the rest as not cached (and says so when that means it couldn't check your activates: names).

Checkouts land in ~/.cache/caliper/skills/ (or $XDG_CACHE_HOME/caliper/…), keyed by resolved commit — so they're immutable, shared across every spec that names them, and safe to delete. Set CALIPER_CACHE_DIR to put them elsewhere.

If a git source can't be fetched and isn't cached, the run refuses — a member silently missing would measure your skill against competition that wasn't there. If it's cached but the remote is unreachable, the run uses the cache and says so.

Skill drift

caliper compare reports any member whose text changed between the two runs. A git source that moved gets a warning: the spec said where its bytes came from, and the delta you're reading is confounded. A path source that moved is shown without alarm — that's usually the edit the run exists to measure.

 ⚠ tdd changed between runs — git source, a1b2c3d → e4f5g6h; pin `ref:` to hold it fixed
   my-skill changed between runs — path, 4fc7951 → bcbcbde

This is a change in text at constant membership. A change in membership — different skills installed — is the separate neighbourhood warning.

activates:: did the agent reach for it?

activates: asserts the exact set of skills that loaded on each attempt.

A task with activates: and no expect:/assert: is a trigger probe: it asks only what the agent reached for, skips the judge entirely (so it is much cheaper than an execution task), and reports as trigger only rather than a zero. Use it for neighbour and silence probes, where there is no work worth grading.

Activation is scored on its own scoreboard, never blended into the success rate. A failing description and a failing body are fixed in different places, so one number mixing them would point at neither.

MCP servers (mcp:)

The optional mcp: block declares the MCP servers the agent-under-test may use. It is a capability granted to the agent for the eval, part of the run environment like sandbox:, so it lives in the spec rather than behind a flag. It is a top-level mapping keyed by server name (a sibling of sandbox: and skills:, and it applies whether or not the eval declares any skill). Each server's tools appear in the transcript as a namespaced call an expect: judge can verify (mcp__<server>__<tool> on claude-code and codex, mcp_<server>_<tool> on hermes), so word an expect: around the tool's behavior, not one backend's exact spelling, if the spec is meant to run under more than one engine.

A server is either local (stdio), a command the harness spawns, or remote (type: http or sse), a hosted endpoint at url, the shape most connectors (Google Drive, Notion, and so on) use:

mcp:
  weather:                      # local stdio server (the default transport)
    command: python3            # required: the local stdio command to spawn
    args: [./servers/weather.py]  # optional
    env:                        # optional
      API_TOKEN: ${MCP_API_TOKEN}
  gdrive:                       # remote server
    type: http                  # required for remote: http or sse
    url: https://mcp.example.com/gdrive   # required for remote
    headers:                    # optional: usually auth
      Authorization: Bearer ${GDRIVE_TOKEN}

View on GitHub