dsh-trajectory-ablation

A debugger that finds the actual cause of an agent failure, instead of guessing.

A plugin for DeepSeek Harness that reconstructs exactly what your agent saw before a given step, diffs that against another step, and — most importantly — proves which piece of context actually caused a decision by removing it and replaying the same model call for real.

See trajectory-ablation.md for the full design rationale.

Why

When an agent does something wrong, most tools show you a transcript and let you guess, or ask the agent to explain itself — which is just the same model generating a plausible-sounding story about its own past decision, not a verified fact. This plugin replaces both with something checkable:

1. Reconstruct — the exact, complete list of everything the model saw before a step, grouped by source. No model calls, just reading the log.
2. Diff — what changed between two steps: added, removed, compressed, or reordered. No model calls.
3. Ablate — remove one piece of context at a time and replay the model call for real, k times per piece. If removing something changes the decision, it's a cause. If it doesn't, it's bystander context.
4. Interaction screen — for the case single-block ablation can't see: two pieces that are each individually inert but jointly cause the decision. Testing every pair is quadratic, so this instead samples a few random subsets, and when one flips the decision, narrows it down (delta-debugging) to the minimal responsible group. Won't find every such pair, but stays linear in cost instead of quadratic.

Install

Mount it in your DeepSeek Harness profile ($DSH_HOME/cordis.patch.yml):

- insert:
    - id: trajectory-ablation
      name: 'dsh-trajectory-ablation'

Then pnpm install in your profile directory and restart. See cordis.patch.yml.example for a fuller walkthrough.

Usage

Once mounted, four commands are available in any session:

/context-reconstruct <turn> <step>                          — what did the model see before this step?
/context-diff <turnA> <stepA> <turnB> <stepB>               — what changed between two steps?
/ablate <turn> <step> [k] [maxCalls|inf]                     — which block actually caused this decision?
/ablate-interactions <turn> <step> [trials] [maxCalls|inf]  — run this when /ablate finds nothing: is it a pair?

/ablate and /ablate-interactions both refuse to run past an estimated 50 real model calls unless you raise maxCalls (or pass inf to disable the cap) — every call is a real, billable request, so cost should never be a surprise.

Or use it programmatically — reconstructContext and diffContext need only a session's event log (no live model connection required); ablateStep and screenForInteractions need a live ctx.llm:

import { reconstructContext, diffContext, ablateStep, screenForInteractions } from 'dsh-trajectory-ablation'

Try it without a live session

npm install
npm run demo

Runs the full pipeline against a scripted example (a stale README misleading an agent into editing the wrong file) with no API key required.

Development

npm install
npm run build       # compile
npm run typecheck   # type-check only
npm test            # run the test suite

License

MIT — see LICENSE.