snipara-evals

Project Intelligence evals for AI coding agents.

snipara-evals is a small open-source scoring harness for questions most agent evals still miss:

• did the agent preserve the project context it was given?
• did it respect decisions and avoid rejected directions?
• did it understand the impact surface of the change?
• did it run or report the expected verification?
• did it preserve enough continuity for another agent to resume the work?

It is deterministic by default and does not require an LLM judge. You describe a case in JSON, pass in the observed agent output, and get a scorecard that can run locally or in CI.

Quickstart

# Run an evaluation case
npx snipara-evals run examples/project-intelligence-case.json

# Output JSON (for CI/CD)
npx snipara-evals run examples/project-intelligence-case.json --json

# Fail CI if any case misses threshold
npx snipara-evals run examples/project-intelligence-case.json --fail-on-threshold

# Multiple cases
npx snipara-evals run cases/*.json --json --fail-on-threshold

Each case is a JSON file with expected Project Intelligence signals (context, decisions, impact, verification, continuity) and observed agent output (text, files, checks).

Why This Exists

Most coding-agent evals ask whether the final answer is correct. That is useful, but not enough for real projects. Agents also need to carry project decisions, blast radius, verification plans, and handoff context across a task.

This repo is part of the open "Mini Snipara" stack:

Repo	Role
snipara-companion	Workflow continuity for AI coding agents
snipara-memory	Local durable project memory
snipara-evals	Project Intelligence scoring

Hosted Snipara remains the managed layer for source authority, reviewed memory, team sync, code graph impact, dashboards, and operations.

Quick Start

npx snipara-evals run examples/project-intelligence-case.json

JSON output:

npx snipara-evals run examples/project-intelligence-case.json --json

Fail CI when thresholds are missed:

npx snipara-evals run examples/project-intelligence-case.json --fail-on-threshold

Case Format

{
  "id": "agent-handoff-example",
  "expected": {
    "context": [
      {
        "id": "license",
        "text": "The public repository uses Apache-2.0.",
        "keywords": ["Apache-2.0", "public repository"]
      }
    ],
    "decisions": [
      {
        "id": "hosted-code-graph",
        "statement": "Code graph remains hosted.",
        "keywords": ["code graph", "hosted"],
        "rejectedKeywords": ["open source code graph"]
      }
    ],
    "impact": [
      {
        "id": "cli-surface",
        "target": "CLI command surface",
        "keywords": ["CLI", "command"],
        "files": ["src/cli.ts"]
      }
    ],
    "verification": [
      {
        "id": "test-suite",
        "check": "Run the package tests.",
        "command": "pnpm test",
        "keywords": ["tests pass"]
      }
    ],
    "continuity": [
      {
        "id": "handoff",
        "handoff": "Leave a concise next-step handoff.",
        "keywords": ["handoff", "next step"]
      }
    ]
  },
  "observed": {
    "answer": "Implemented the CLI in src/cli.ts, kept code graph hosted, and ran pnpm test. Handoff: next step is release packaging.",
    "filesChanged": ["src/cli.ts"],
    "commandsRun": ["pnpm test"]
  }
}

Metrics

snipara-evals reports five metric scores from 0 to 100.

Metric	What it checks
Context Preservation	Required facts, constraints, and source details were carried into the result
Decision Consistency	Canonical decisions were followed and rejected directions were avoided
Impact Awareness	Files, commands, dependencies, risks, or affected surfaces were acknowledged
Verification Coverage	Expected checks were run, mentioned, or marked with status
Continuity	Handoff, resume point, blockers, and next action are visible

The scoring rules are intentionally simple and public. This repo is not a dump of Snipara's internal benchmark fixtures, tuning notes, or private rubrics.

Library Usage

import { evaluateCase } from "snipara-evals";

const result = evaluateCase({
  id: "demo",
  expected: {
    context: [{ id: "fact", text: "The package is Apache-2.0 licensed." }],
  },
  observed: {
    answer: "The package is Apache-2.0 licensed.",
  },
});

console.log(result.overall.score);

Development

pnpm install
pnpm build
pnpm type-check
pnpm lint
pnpm test
pnpm pack:smoke

Open Source Boundary

Good public contributions:

• deterministic scoring helpers
• portable JSON case formats
• CLI/reporting improvements
• sanitized examples
• adapters for public agent transcripts

Keep private:

• customer data
• private runbooks or deployment details
• internal benchmark datasets and raw reports
• proprietary ranking heuristics or hosted code graph internals
• secrets, .env files, screenshots, logs, and generated config dumps