DeepSeek Self-Harness Plugin

A DeepSeek Harness (dsh) plugin that makes the harness improve itself — by mining your own session logs and evolving the operating rules you run on.

Design based on the paper Self-Harness: Harnesses That Improve Themselves (arXiv 2606.09498), adapted for real-world dsh usage without a benchmark verifier.

---

What problem does this solve?

The quality of an LLM agent is shaped not only by its model, but by its harness: system prompts, operating rules, failure-recovery procedures, and runtime policies. These are usually written once by humans and never revisited. Yet every real conversation leaves durable evidence of what actually goes wrong — tools that fail, commands that get blindly retried, corrections the user has to repeat, timeouts that kill long operations.

This plugin closes that loop:

• Your agent keeps repeating a mistake that already failed in a past session? It gets mined, clustered, and turned into a concrete rule.
• You keep correcting the same behavior? It becomes part of the harness instead of being re-explained every time.
• Every user installs the same plugin, but each one ends up with a personal harness evolved from their own history.

No external "stronger agent", no fine-tuning, no benchmark suite required — the same model that runs your tasks proposes the edits, and nothing is applied without your approval.

How does it work?

The plugin implements the three-stage loop from the paper, with one honest adaptation (see Why a human gate?):

                ┌─────────────────────────────────────────────────┐
                │                   your sessions                 │
                └───────────────┬─────────────────────────────────┘
                                │  session ends (session/disposed)
                                ▼
  ┌───────────────────────────────────────────────────────────────┐
  │ ① Weakness Mining (automatic, background)                    │
  │    scans session logs via session-query's semantic documents │
  │    classifies failures by terminal cause:                    │
  │      not-found · timeout · denied · exit-code · other-error  │
  │    detects blind retries (same call repeated ≥3 times)       │
  │    clusters evidence into (symptom:mechanism) signatures     │
  └───────────────────────────────┬───────────────────────────────┘
                                  ▼
  ┌───────────────────────────────────────────────────────────────┐
  │ ② Harness Proposal (same model, in your next session)        │
  │    a bounded brief: current rules + evidence clusters +       │
  │    output contract; the model proposes K diverse yet minimal  │
  │    edits, each anchored to ONE evidence cluster and ONE       │
  │    surface, with rationale + regression risk                  │
  └───────────────────────────────┬───────────────────────────────┘
                                  ▼
  ┌───────────────────────────────────────────────────────────────┐
  │ ③ Promotion gate (human by default)                           │
  │    each candidate is diff-previewed and applied only after    │
  │    you accept; accepted edits hot-update the runtime rules,   │
  │    write to the lineage log, and are rollback-able            │
  └───────────────────────────────────────────────────────────────┘

What is the "harness" here?

The editable surface is a runtime rules document maintained by the plugin (Self-Harness Runtime Rules), registered as a system-prompt section and hot-updated after every accepted edit. Edits are tiny, auditable operations:

• append — add one rule line at the end
• replace — replace one exact anchor block (must occur exactly once)

Every transition h0 → h1 → … is recorded in a lineage log (version, timestamp, evidence cluster, operation, rationale, who accepted), and the last 20 versions are kept as backups for one-command rollback.

Persistence & background mining

• State (rules doc, lineage, backups, pending candidates, latest evidence) is persisted to .self-harness-state.json in the workspace root and restored across restarts and plugin updates.
• After a session ends (session/disposed), the plugin mines it in the background (5 s delay, watermark-deduplicated, concurrency-guarded). New evidence bumps a counter that is injected into the next session's system prompt, nudging the agent to run the proposal stage.

Why a human gate?

The paper promotes edits automatically via a regression rule (Δin ≥ 0 ∧ Δho ≥ 0 ∧ max > 0) over a held-out benchmark with verifiers. Real conversations have no verifier and are not re-runnable, so that gate cannot be reproduced honestly. This plugin therefore automates mining + proposal (read-only, safe) and keeps promotion human-confirmed by default (diff → accept/reject → instant rollback). A verifier-suite mode (promotionMode: auto with Δin/Δho) is on the roadmap.

Install

The plugin is a bare Cordis plugin — add it to an agent preset you author.

1. Add the package to the preset's dependencies (the preset is a directory under ${DSH_HOME:-$HOME/.dsh}/.agent-presets/<id>/):

   cd ~/.dsh/.agent-presets/<your-preset>
   # From the GitHub repository (recommended until the npm release):
   pnpm add github:kater20070212/deepseek-self-harness-plugin
   # or pin a commit:
   pnpm add github:kater20070212/deepseek-self-harness-plugin#main
   # once published to npm:
   pnpm add deepseek-self-harness-plugin
   

2. Add one row to the preset's agent.cordis.yml:

   # Self-Harness: mines your session logs and evolves your runtime rules.
   - id: self-harness
     name: 'deepseek-self-harness-plugin'
   

3. Restart dsh (or start a new session). The plugin registers 7 tools (self_harness_*) and begins watching sessions in the background.

Prerequisites

• DeepSeek Harness (dsh) running a recent build that includes the session-query semantic documents (filterEvents), systemPrompt, userQuestions, fs, and timer services, and the session/disposed event.
• Node.js ≥ 22.19.

Usage

The everyday loop (no manual steps)

1. Work normally. Every session you finish is mined automatically in the background (session/disposed → 5 s later → evidence clustered by terminal cause).

2. Next session, you (and the agent) see the nudge. If new evidence was found, the system prompt gains a hint like:

   [self-harness] Background mining found 3 new evidence bundle(s) in your session logs. Consider running self_harness_propose …

3. Tell the agent to act — in any session, simply say:

   Run a self-harness round.

   The agent then runs the loop: self_harness_propose → submits candidates → self_harness_validate.

4. You are the gate. Each candidate shows a diff (the exact rule being added or replaced) with its evidence cluster, rationale, and risk. Accept to apply — the rule goes live at the next model step; reject to only log it. When interactive prompts are unavailable, answer in conversation instead (the agent applies your decision with self_harness_apply).

5. Everything is reversible. Every applied edit is a version in the lineage log with a backup; roll back any time.

One manual round, step by step

you:   Run a self-harness round.
agent: self_harness_mine                    → 4 evidence clusters (e.g. tool-error:not-found ×9 …)
agent: self_harness_propose                 → bounded brief (rules + clusters + contract)
agent: self_harness_submit_proposal         → K candidates, each anchored to one cluster
agent: self_harness_validate <candidateId>  → diff preview → you accept/reject
accepted → rules document hot-updates → next model step follows the new rule

You can also drive any stage yourself by name (self_harness_status, self_harness_rollback).

Example: what an accepted edit looks like

Evidence cluster tool-error:not-found (9 occurrences, e.g. "git: command not recognized" after a tool was called without checking it exists) produces the candidate:

- (end of document)
+ - Before running an external command, check it exists (e.g. Get-Command / command -v);
+   when a tool is missing, use an equivalent alternative instead of calling a
+   known-missing tool.

Lineage entry recorded for it:

{ "version": 1, "cluster": "tool-error:not-found", "surface": "harness-doc",
  "op": { "position": "append", "insert": "…" }, "status": "applied", "by": "user" }

Inspect and roll back

• self_harness_status — version, document preview, lineage, pending candidates, evidence summary, background-mining stats, persistence path.
• self_harness_rollback — restore any of the last 20 versions (omit toVersion for the previous one).
• State lives in .self-harness-state.json in the workspace root; it survives restarts and plugin updates.

Model-facing tools

Tool	Stage
self_harness_mine	① Weakness mining over recent sessions (manual run)
self_harness_propose	② Returns the bounded proposal brief (then the model proposes)
self_harness_submit_proposal	② Submits K candidates, each anchored to an evidence cluster
self_harness_validate	③ Diff preview + interactive accept/reject gate
self_harness_apply	③ Explicit accept/reject (fallback / scriptable gate)
self_harness_status	Lineage, pending candidates, evidence summary, background stats
self_harness_rollback	Roll back the rules document to any of the last 20 versions

A typical round, driven by the agent itself:

self_harness_mine → self_harness_propose → self_harness_submit_proposal
→ self_harness_validate (you see the diff) → accepted edits are live next step

Or simply tell the agent: "Run a self-harness round."

Repository layout

src/index.js            the plugin (plain JS, zero runtime imports)
cordis.yml.example      sample preset row
docs/DESIGN.md          paper-to-plugin mapping, contracts, roadmap

Security & scope notes

• Mining only reads your own session corpus via the session-query service; it never sends logs anywhere.
• Edits are text ops on the plugin-owned rules document — they cannot execute code or change other plugins' configs.
• The plugin never self-promotes without your acceptance in the default mode; every applied edit has a backup.
• The proposal stage is the same model you already run; it receives only the clustered evidence bundle, never raw logs.

Roadmap

• Editable surface upgrade: the user's dedicated self-harness agent preset (agentPresets.copy bootstrap + anchored cordis.yml edits)
• Background proposal with the default model after session end (currently proposal runs in-session)
• Optional verifier-suite mode reproducing the paper's Δin/Δho auto-promotion rule
• Client UI: lineage timeline + diff review panel in the Web GUI
• npm publication (dsh-plugin topic)

License

MIT — see LICENSE.

Credits

Loop design adapted from Self-Harness: Harnesses That Improve Themselves — Zhang et al., Shanghai AI Laboratory, 2026 (arXiv:2606.09498).