slop-grader

Rule-based slop grader for text files, powered by Jev.

How it works

slop-grader runs as a two-step loop: grade text with the CLI, then paste the output to your AI agent to plan the improvements.

1. Grade the document

Run slop-grader on a document like examples/slop.md:

npx @lukstei/slop-grader@latest -r no-ai-slop -r grammar-english -r tech-docs examples/slop.md

Output:

Use the SKILL `/path/to/slop-grader/SKILL.md` to improve `/path/to/slop-grader/examples/slop.md`.

Rules:
  /path/to/slop-grader/rules/no-ai-slop.md
  /path/to/slop-grader/rules/grammar-english.md
  /path/to/slop-grader/rules/tech-docs.md

A=banned_word, B=empty_adverb, D=binary_contrast, F=faux_insight, G=colon_reveal, ...

A               | L0001: # 🚀 The Ultimate Paradigm Shift in Modern Data Architecture
A               | L0003: In this article, we will delve into the rich tapestry of modern distributed systems and explore how they seamlessly empower developers to unlock their true potential.
D,F,G,Q         | L0007: What most people get wrong about databases is simple: it's not about speed, it's about trust.
G               | L0009: The secret: it's all about asynchronous event-driven pipelines.
N               | L0011: Studies show that 90% of architectures fail because of poor alignment.
...

## Document Scores

structure_navigability  0.6/3   (confidence mid )  "Wall of text" ↔ "Has headings but they are vague or inconsistent"
task_orientation        0.6/3   (confidence mid )  "Architecture dump" ↔ "Mixed"
completeness            0.0/3   (confidence high)  "Fragment — critical steps, configuration, or context are missing"
code_example_quality    0.1/3   (confidence high)  "No examples, or examples are pseudocode fragments that cannot run"
prerequisite_clarity    0.6/3   (confidence mid )  "No prerequisites stated" ↔ "Partially stated"

2. Fix with an AI agent

Pass the output to your AI agent:

• The agent distinguishes real violations from false positives and generates concrete replacements (example plan with Gemini 3.8 Flash: full plan):

  ### Line 1 — `banned_word`
  - **Original:** `# 🚀 The Ultimate Paradigm Shift in Modern Data Architecture`
  - **Fix:** `# Modern Data Architecture`
  - **Reason:** Removes the banned phrase "paradigm shift" and decorative emoji.
  
  ### Line 7 — `binary_contrast` + `faux_insight` + `colon_reveal` + `fake_profound_kicker`
  - **Original:** `What most people get wrong about databases is simple: it's not about speed, it's about trust.`
  - **Fix:** `Database design balances speed and trust.`
  - **Reason:** Removes rhetorical framing and fake insight.
  

• After your review the plan is applied to produce an improved document.

Quick Start

Requirements

• Node.js 18+
• An API key for your chosen provider:
  • TYPESAFE_API_KEY — typesafe.ai (uses jev provider)
  • OPENROUTER_API_KEY — openrouter.ai (uses openrouter provider)

Run

export TYPESAFE_API_KEY=...
npx @lukstei/slop-grader@latest -r no-ai-slop -r article-scores my-draft.md

Rulesets

Built-in rulesets

Pass built-in rulesets by name:

Custom rulesets

Define custom rules in Markdown (-r ./my-rules.md), organized under # Line Rules and # Document Rules sections.

You can be creative and ask any plain-text question about a single line or the whole document. Rules work on any text format—prose, git diffs, server logs, legal contracts, or structured text like CSV files:

# Line Rules

## empty_adverb
Does the line use an adverb that adds nothing to the meaning?

### Criteria
- **true**: The adverb could be deleted without changing the sentence.
- **false**: The adverb carries real emphasis or spoken rhythm.

# Document Rules

## narrative_arc
Rate the narrative arc of the document.

### Criteria
- No clear arc — sections feel disconnected
- Loosely organized — a theme but no build
- Clear progression — each section sets up the next
- Tight arc — the ending pays off the opening

Inspirations for rulesets

# Line Rules

## suspicious_refund
Does this CSV transaction row show a refund exceeding \$500 without a manager approval ID in column 6?

### Criteria
- **true**: The row records a refund over \$500 and column 6 lacks an approval ID.
- **false**: The amount is \$500 or less, column 6 contains an approval ID, or the row is not a refund.

# Document Rules

## root_cause_depth
Evaluate whether this postmortem addresses systemic engineering safeguards instead of individual human error.

### Criteria
- Blames operator error without addressing missing guardrails
- Identifies immediate triggers but ignores underlying architecture
- Identifies failure modes and plans concrete monitoring or test coverage
- Proposes systemic automated defenses, blast-radius containment, and architectural fixes

# Line Rules

## uncapped_indemnity
Does this clause expose the company to uncapped indemnification for third-party claims?

### Criteria
- **true**: The clause creates an indemnification obligation without liability caps.
- **false**: The obligation falls under the standard aggregate liability limit.

# Line Rules

## unauthorized_promise
Does this agent response promise an unreleased feature date or custom contract concession?

### Criteria
- **true**: Agent commits to an unannounced date or non-standard term.
- **false**: Agent refers customer to public docs or defers to account managers.

# Line Rules

## unjustified_type_cast
Does this line use a type assertion (`as`), non-null assertion (`!`), or loose cast to silence a compiler error without proper narrowing or input validation?

### Criteria
- **true**: Casts away type safety without an upstream type guard, schema validation, or explanatory comment.
- **false**: Type is narrowed safely, or the assertion bridges an external API boundary with runtime checks.

# Line Rules

## hardcoded_secret
Does this line contain a hardcoded API key, bearer token, private key, or password rather than referencing an environment variable or secret manager?

### Criteria
- **true**: Line contains a literal credential, private token, or hardcoded secret string.
- **false**: Line references an environment variable, config placeholder, mock test fixture, or public key.

# Line Rules

## swallowed_error
Does this catch block or fallback expression silence an unexpected error without diagnostic logging or recovery?

### Criteria
- **true**: Catches an exception and returns null or an empty default without logging context.
- **false**: Logs the error with context, rethrows, or implements a documented recovery strategy.

# Document Rules

## commit_intent
Does this commit message or PR description explain the motivation and problem context rather than merely describing code changes?

### Criteria
- Mechanical change list only with no rationale
- Mentions the fix with minimal explanation of the problem
- Explains the failure trigger, bug condition, and rationale clearly
- Details root cause, design tradeoffs considered, and verification evidence

See docs/SYNTAX.md for the complete Markdown rule syntax specification and validation reference.

Validate ruleset syntax offline without an API key:

npx @lukstei/slop-grader@latest --check -r ./my-rules.md

Use the create-slop-grader-rules skill to create and validate custom rulesets with an AI assistant.

Custom JSON rulesets (-r ./my-rules.json) are also supported.

CLI Reference

npx @lukstei/slop-grader@latest [-c|--check] -r <ruleset> [-r <ruleset> ...] [--provider <jev|openrouter>] [--model <model>] [--json] [--stats] [--debug] [file]

Flags

Providers and environment variables

Provider resolution order:

1. --provider (-p) flag
2. TYPESAFE_PROVIDER environment variable
3. Auto-detected from keys (TYPESAFE_API_KEY selects jev; OPENROUTER_API_KEY selects openrouter)

Grading runs on jev-latest (TypeSafe) or ~typesafe/jev-latest (OpenRouter) by default, overridable via --model (-m).

Output Formats

Human-readable report

By default, slop-grader prints a human-readable report. Clean lines are omitted; only lines crossing the 0.8 confidence threshold appear. Pass --stats (or -s) to append execution metrics (rules applied, lines evaluated, API calls).

If line rules run but find no violations, No line rule violations found. is displayed.

JSON report (--json)

Pass --json (or -j) for structured machine-readable output:

npx @lukstei/slop-grader@latest -r no-ai-slop -r article-scores --json --stats my-draft.txt | jq .

{
  "file": "/abs/path/to/my-draft.txt",
  "rules": ["/abs/path/to/no-ai-slop.md"],
  "violations": {
    "lines": [
      { "lineNum": 1, "text": "Our platform empowers teams...", "rules": ["banned_word"] }
    ],
    "document": {
      "narrative_arc": { "score": 1.4, "max": 3, "confidence": 0.72, "label": "Loosely organized" }
    }
  },
  "stats": {
    "rules": 6,
    "lineRules": 5,
    "docRules": 1,
    "lines": 12,
    "questions": 61,
    "apiCalls": 6
  }
}

violations.lines and violations.document are empty when the file is clean. Useful for CI pipelines and editor integrations.

How Evaluation Works

Evaluation separates line-level checks (spotting specific patterns or phrases) from document-level checks (evaluating tone or overall structure).

Batching by rule instead of line

Documents have hundreds of lines, but the amount of rules is fixed.

Sending one API request per line would mean hundreds of calls to the AI. A 300-line document with 5 rules would take 300 requests.

Instead, slop-grader groups lines into batches of 255 and evaluates each rule across the entire batch in a single call. That same 300-line document runs in just 10 parallel requests.

Document rules run in a single request across the entire text.

Development

npm test         # Run tests
npm run verify   # Run typecheck, biome lint, and tests
npm run build    # Build

Contributing

See CONTRIBUTING.md for contribution guidelines, development setup, and coding best practices.

License

MIT