Measure Workspace Intelligence Usage

Use this feature to answer a practical question: did focused workspace context help an AI task use fewer tokens, cost less, or reach a verified result?

Workspai records model usage, tool activity, reported cost, and the final verified outcome in one report that the CLI, IDE, CI, and dashboards can read.

It does not store prompt or response text. Optional SHA-256 hashes can connect related calls without copying private content into the report.

Start a measured task

npx workspai workspace eval init repair-readiness workspace-intelligence --json

The optional strategy is one of full-corpus, grep, vector, graph, workspace-intelligence, or custom.

The live source of truth is:

.workspai/reports/workspace-intelligence-evaluation-live.json

Record events

Providers and IDE integrations append privacy-bounded events through stdin:

printf '%s\n' '{
  "kind": "model-call",
  "stage": "context",
  "modelCall": {
    "provider": "vscode-lm",
    "model": "selected-model",
    "source": "provider-reported",
    "inputTokens": 18420,
    "outputTokens": 2150,
    "cachedInputTokens": 6200,
    "reasoningTokens": null,
    "latencyMs": 4820
  }
}' | npx workspai workspace eval record --json

source is mandatory:

Source	Meaning
provider-reported	The provider returned the count
tokenizer-counted	A named tokenizer counted the exact serialized input or output
estimated	A documented estimate; never presented as provider billing
unavailable	The provider exposed no usable count

Tool events record progress and repeated work without storing command output:

{
  "kind": "tool-call",
  "stage": "verify",
  "toolCall": {
    "tool": "workspace verify",
    "result": "passed",
    "changedSource": false,
    "durationMs": 814,
    "artifact": ".workspai/reports/workspace-verify-last-run.json"
  }
}

Finish with a verified outcome event. Tokens per successful outcome remain null until status is passed and verified is true.

{
  "kind": "outcome",
  "stage": "verify",
  "outcome": {
    "status": "passed",
    "verified": true,
    "blockersResolved": 1,
    "summary": "Readiness passed after dependency remediation."
  }
}

Event bodies conform to model-usage-event.v1.json.

Inspect and finalize

npx workspai workspace eval status --json
npx workspai workspace eval report --json
npx workspai workspace eval report --output ./evidence/my-evaluation.json --json

Finalization writes:

.workspai/reports/workspace-intelligence-evaluation-last-run.json

--output keeps that governed last-run artifact and also copies the finalized report to the requested workspace-relative or absolute path. JSON output reports the path that was actually requested, so automation does not need to infer it.

Both live and final reports conform to workspace-intelligence-evaluation.v1.json.

Compare strategies

Save a completed baseline, run the same fixed task using another strategy, then compare:

npx workspai workspace eval compare --from .workspai/reports/baselines/full-corpus.json --json

The comparison reports token, model-call, tool-call, latency, and verified outcome differences. A smaller prompt is not treated as a better result unless the outcomes are task-aligned and independently verified.

Comparison JSON conforms to workspace-intelligence-evaluation-comparison.v1.json, so CI, IDE, and dashboard consumers do not need to infer its shape.

Extension and dashboard contract

The VS Code extension should watch the live artifact and display its values; it must not independently estimate or relabel them. At minimum the UI should show:

• run and session identity;
• live/final status and update time;
• input, output, cached-input, and reasoning token components;
• provider-reported, tokenizer-counted, estimated, and unavailable call counts;
• cost grouped by currency and source;
• tool calls, repeated artifact reads, and no-progress decisions;
• verified outcome and blockers resolved.

This makes the CLI artifact the source of truth for terminal, extension, CI, and future hosted dashboards. MCP clients can read the same artifact through getWorkspaceEvaluation; agent-sync also includes the finalized report in the governed evidence index when it exists.

Relationship to the graph benchmark

workspace graph benchmark measures retrieval payload size using a portable character estimate. workspace eval measures observed agent execution and requires outcome evidence. Keep both: the first is deterministic and provider-independent; the second is the basis for real token, cost, and task efficiency claims.