dsh-agent-eval

English | 简体中文

Repeatable agent and plugin regression evaluation for DeepSeek Harness.

Status: design and implementation planning. This repository does not yet contain an installable plugin.

Goal

Turn an agent task into a versioned test case that can run against a selected DeepSeek Harness agent preset and model, then produce deterministic assertions, transcript evidence, token and timing measurements, and CI-friendly reports.

The project is intended for plugin authors and teams maintaining agent presets. It tests assembled application behavior, not only TypeScript units or mocked tool calls.

Planned user experience

Install one profile bundle:

dsh plugin --profile web add github:ShawnSiao/dsh-agent-eval

Run a suite from the Web command surface:

/eval smoke

Run the same suite in CI:

dsh-agent-eval run .dsh/evals/smoke.yml --report junit

The command names are planned interfaces and are not available yet.

Core principles

• Evaluate real assembled agents through ctx.agents, not a second agent loop.
• Mount the selected agent preset during unpublished agent setup.
• Run every case in a disposable workspace copy by default.
• Keep deterministic assertions separate from optional model-judge assertions.
• Preserve the Session log as transcript evidence while keeping suite summaries in versioned report files.
• Apply explicit request, token, time, and filesystem budgets.
• Register no model-facing evaluation tool in the first release.

See the implementation plan for the case format, architecture, phases, and acceptance criteria.

Compatibility baseline

Planning is based on DeepSeek Harness master commit 47f943859b and package version 0.1.0-rc.5 as observed on 2026-08-13. DeepSeek Harness is in developer preview; implementation work must recheck public APIs before each release.

License

MIT