A DSH eval gate that runs YAML cases against a real headless agent, asserts on the session trace and compares the report with a baseline to produce a CI exit code — installed from GitHub, with no declared license and no release tag.
DSH integration
Native runtime
Author-claimed
Safety audit
Unaudited
Last verified
2026-11-13
License
Not specified by the author
01What can it help you accomplish?
Put a regression gate on a DSH plugin or skill before you ship it
report.json + report.md per run, and a CI exit code (OVERALL=PASS|WARN|FAIL|N/A) from the baseline comparison
DSH plugin and skill authors who need to catch prompt, tool and protocol regressions
Assert on a real agent's session trace instead of on answer text
a per-case isolated workspace and session directory, with assertions on turn_end, tool call order, tool results, step/token caps and an optional LLM judge
Developers who want proof that a tool was really called and returned the expected content
Run protocol-failure regressions offline, without an API key
deterministic F0-F5 fault-matrix cases plus two mock sentinels that report `healthy: false` when the rig itself breaks
CI owners who need a zero-cost, non-flaky gate for the streaming error paths
02How to install into DeepSeek Harness
Prerequisites
- A working `dsh` CLI — every case forks `dsh --profile headless --patch <overlay> <prompt>`
- Real-model runs need a model credential; the project's own CI passes it as the `DEEPSEEK_API_KEY` secret
Installation steps
- 01
`dsh plugin --profile headless add github:BiBoyang/dsh-eval-harness` — install from the GitHub source; the npm channel is deprecated and stays at 0.3.1
$ dsh plugin --profile headless add github:BiBoyang/dsh-eval-harness
Verify the integration
- `dsh --profile headless --dump-config | grep dsh-eval-harness` lists the plugin
03DSH integration and capability boundaries
DSH plugin for the headless profile: four tools (eval_run / eval_gate / eval_judge_validate / eval_tps_run) plus an `eval` skill, driven by YAML cases
eval_run — headless eval runner
a cases_dir of *.yml cases (prompt + assertions, optional mock / plugins / trials)→report.json + report.md with per-case status, attempt history, token breakdown and reliability
forks one `dsh --profile headless` child process per case and writes an isolated session root and workspacewrites `eval-overlay-*.patch.yml` files into the output directoryeval_gate — baseline gate
before (baseline report.json) and after (this run's report.json)→OVERALL / EXIT_CODE key=value lines, or a single JSON verdict for CI
eval_judge_validate — LLM judge calibration
a hand-labelled JSONL set of `rubric` / `output` / `expect` rows→a confusion matrix with TPR/TNR reported separately and a `calibrated` verdict
calls the judge once per labelled row, so it needs `EVAL_JUDGE_API_KEY` or `DEEPSEEK_API_KEY`eval_tps_run — long-context TPS sweep
a tps.yml with models, nominal context steps, repeats and budget_tokens→a fitted tps(n)=1/(a+b·n), the half-speed point n½ and a cross-model table
sends real streaming requests to OpenAI-compatible endpoints and spends the declared token budgetruns strictly serially and refuses to start when the planned budget is exceeded`eval` skill
teaches the model the case format, how to write assertions and the limits of the built-in YAML subset
04Who is it for? When not to use it?
Good for
- DSH plugin and skill authors who need to catch prompt, tool and protocol regressions
- Developers who want proof that a tool was really called and returned the expected content
- CI owners who need a zero-cost, non-flaky gate for the streaming error paths
Not for
- The npm channel is deprecated: the registry copy stays at 0.3.1 and the README says not to install from npm — distribution goes through GitHub only.
- Declaring `mock.plugins` authorizes the install-time build scripts of the plugins under test to run on your machine (arbitrary code execution) — only declare plugins whose source you trust, and pin the version or commit.
- Mock mode covers structural assertions and protocol-failure paths only; `output_judge` semantic review and any case that depends on real model capability still need a live API, so mock is not a full offline substitute.
05Compatibility, maintenance and safety notes
- The npm channel is deprecated: the registry copy stays at 0.3.1 and the README says not to install from npm — distribution goes through GitHub only.
- Declaring `mock.plugins` authorizes the install-time build scripts of the plugins under test to run on your machine (arbitrary code execution) — only declare plugins whose source you trust, and pin the version or commit.
- Installing `mock.plugins` specs needs network access (the tool does no offline install), and real-model cases need a model endpoint plus `DEEPSEEK_API_KEY`.
- Mock mode covers structural assertions and protocol-failure paths only; `output_judge` semantic review and any case that depends on real model capability still need a live API, so mock is not a full offline substitute.
No license declared · no release tag (npm channel deprecated, GitHub-only distribution) · README evidence only, not security audited
06Frequently asked questions
How do I install it into DeepSeek Harness?
Install from GitHub only: `dsh plugin --profile headless add github:BiBoyang/dsh-eval-harness`, then verify the mount with `dsh --profile headless --dump-config | grep dsh-eval-harness`. The npm channel is deprecated — the registry copy stays at 0.3.1 and the README says not to install from npm. No license is declared and there is no release tag.
What does eval_run actually do inside DSH?
For every case it forks a `dsh --profile headless --patch <overlay> <prompt>` child process in an isolated workspace and session directory, parses the on-disk `session.jsonl` / `session.jsonl.zstd` trace, runs the assertions, and writes report.json + report.md. A per-case `--patch` overlay switches session persistence to that isolated root, so cases can run in parallel without touching your real `~/.dsh`.
Can the gate run in CI without an API key?
Cases with a `mock:` block start an ephemeral mock provider on a random 127.0.0.1 port, generate an isolated `DSH_HOME`, and replay the F0-F5 fault matrix, so the chaos package runs fully offline. `output_judge` semantic review and cases that depend on real model capability still need a live endpoint and key.
Does declaring mock.plugins run third-party code?
Yes. eval_run installs each declared spec through dsh's own installer (`dsh plugin --profile headless add <spec>`), which authorizes that plugin's install-time build scripts to run on your machine. Only declare plugins whose source you trust and pin the version or commit — the README states this risk explicitly.
How does eval_gate decide PASS, WARN or FAIL?
It compares the run report against a baseline: a PASS turning into FAIL/error, a new FAIL/error case or a case missing from the run is FAIL with exit 1; improvements, token-total growth, new flaky cases and new tool-error recoveries are WARN with exit 0 (2 in strict mode); no baseline is N/A with exit 2. A breach of the mock sentinels forces FAIL regardless of the baseline.
07Related DSH workflows
dsh-our-free-model
by ebony-vinyl
Adds model choices, availability checks, usage panels, and a local OpenAI-compatible forwarder.
dsh-routing-suite
by yjh051108
dsh-routing-suite — injector + router-standard kit: install the runtime injector first, then the task-aware reasoning-mode router preset (measured P1-P23).
dsh-ops
by t-auto
Run Bash and PowerShell commands, search and edit local files, and manage long-running background jobs from DSH.
brooks-lint
by hyhmrright
Portable AI code quality reviews for Codex, Cursor, Copilot and other coding agents: six Agent Skills for architecture, technical debt, tests and remediation.
08Data and sources
DSH 插件/skill 作者的回归评测门禁：写 yaml 用例 → headless 驱动真实 agent 跑 → 解析 session trace 断言 → 对比 baseline 出 PASS/WARN/FAIL 报告与 CI 退出码…
| `eval_run` | 跑 cases_dir 下全部用例：headless 驱动真实 agent → 采集 session trace → 断言 → 写 report.json/report.md |
| `eval_gate` | 对比 baseline 与本次报告，输出门禁判定（OVERALL/EXIT_CODE），strict 模式收紧 WARN 退出码 |
This page is generated from the project’s public documentation, repository metadata and a structured parse of DSH Plugins; last verified on 2026-11-13. Found an error? Submit a correction.
Best DeepSeek Harness Plugins
Twelve plugins worth installing first — picked from the whole catalog, across every category.