😐👍...lgtm?

Prove that your tests actually test something. Bring your own TypeSafe, Vercel AI Gateway, or OpenRouter API key.

Your agent wrote 40 tests. They're all green. What do they prove? lgtm reads every test block with its implementation and tells you which ones are useless.

It runs on Jev by TypeSafe, directly or through Vercel AI Gateway or OpenRouter.

With this, you can prove that your agent actually wrote code that actually works, so you can say it lgtm 😐👍.

Evaluated on real code: 16,000+ test blocks scored across real codebases, 347 read against their implementation and labelled, thresholds fitted so a finding on real code is worth your time. The held-out numbers are in Evals.

What you get

• the lgtm CLI: run it on a file, a directory or --diff, in your terminal or in CI
• a /lgtm skill for your coding agents, so the agent that wrote the tests runs the audit and fixes what it finds
• an /actually-test skill: write the tests the change needs, prove them red, then iterate with lgtm until they pass

Install

bun add -g @stardeckai/lgtm
pnpm add -g @stardeckai/lgtm
npm i -g @stardeckai/lgtm

# or per project
pnpm add -D @stardeckai/lgtm   # then: pnpm lgtm ...   (npm: npx lgtm ...)

If lgtm is not found after a global install, the package manager's global bin directory is not on your PATH. Run npm prefix -g (or pnpm bin -g, bun pm bin -g) and add its bin to PATH, then open a new shell or run rehash in zsh.

Setup

lgtm init                    # choose TypeSafe, Vercel, or OpenRouter, then install the skills

Bring your own TypeSafe API key (https://typesafe.ai), Vercel AI Gateway key (https://vercel.com/ai-gateway), or OpenRouter key (https://openrouter.ai/settings/keys). lgtm init asks which one to use, then whether to install the /lgtm and /actually-test skills with npx skills.

All three provider keys can live in ~/.config/lgtm/config.json (mode 0600), with one selected as the default. The matching TYPESAFE_API_KEY, AI_GATEWAY_API_KEY, or OPENROUTER_API_KEY environment variable wins over its saved key. Existing TypeSafe configs still work.

Skip the prompts with --skill <where>:

--skill
global	every project, via the skills CLI (what --yes picks)
project	this project only, via the skills CLI
claude	write ~/.claude/skills/{lgtm,actually-test}/SKILL.md directly, no npx
none	skip them

If the skills CLI can't run, init falls back to writing the Claude Code skills itself.

Running lgtm before setup exits with 😐✋ No API key. Run: lgtm init.

Use

lgtm .                     # every *.test.* / *.spec.* file under a directory (a path is required)
lgtm src/user.test.ts      # one file, or a directory
lgtm --diff origin/main    # only tests changed vs a base, with the diff as evidence
lgtm --diff                # just what you're working on: changed and new tests, plus tests of changed code, vs the default branch
lgtm --dry-run src         # only the plan: files, API mode, estimated cost and runtime; no key needed

Every run starts with that plan and asks Run? [Y/n]. Outside a terminal (CI, an agent) it stops after the plan unless you pass --yes.

lgtm . --yes --format json # non-interactive

For a one-off run without installing: npx @stardeckai/lgtm --dry-run src.

test/payment.test.ts:42  "rejects expired cards"
  😐👏 mocks-seam-under-test 0.93 — The collaborator that decides this behaviour is a mock, so the test proves the mock's script, not the code; use the real one here.

test/refund.test.ts:17  "refunds a captured charge"
  😐🤏 swallowed-error-as-success 0.88 — The test passes whether the error is thrown, caught or never raised; assert the specific failure by class, code or message.

😐🫵  2 tests prove nothing, out of 31 test cases.
4 contract-integration · 19 mocked-seam · 8 pure-logic

98120 input tokens used ≈ \$0.0041 (\$0.0001 per test)
9.8s (0.3s per test)

Four faces, one per family: 😐🤏 the assertion proves this much, 😐👏 you tested the mock, 😐🤌 what exactly are we doing here, 😐🫸 do not merge this. A finding within 0.15 of its check's threshold carries no face, is yellow and reads "Worth a look."; it does not count against the test; one clear of that margin is red and does. --verbose also shows suspects under the threshold, dim. The verdict is one line, 😐👍 N tests. fine...lgtm?, 😐🤞 N tests worth a look, out of M test cases. or 😐🫵 N tests prove nothing, out of M test cases. --format github emits a warning for a proven finding and a notice for one worth a look; --format json stays plain.

/actually-test

Gets your agent to actually test the code it just wrote, and iterates on /lgtm (don't worry, it's cached) until it proves the code is actually tested.

Flags

flag
--diff [base]	only the tests your change touches, with the diff in the state. Without a base it uses the repo's default branch (origin/HEAD, else origin/main, else main, else master), and always compares against git merge-base <base> HEAD, so a branch that is behind does not report the base's own commits. Counts uncommitted and untracked files, keeps only the test blocks that overlap a changed line, and adds any test whose imports include a changed source file (all of its blocks). The two diff checks only run on blocks the diff touched
--diff-all-blocks	with --diff, audit every block of a changed file instead of only the changed ones
--threshold <0..1>	override every check's threshold
--only <ids,…> / --skip <ids,…>	pick checks
--format text|github|json	github emits ::warning annotations
--concurrency <n>	parallel requests, default 4 for TypeSafe and OpenRouter, 1 for Vercel
--rate <n>	maximum live request starts per minute; Vercel defaults to 10, TypeSafe and OpenRouter have no pacing by default
--no-impl	don't send implementation source
--ignore <pattern>	skip paths; repeatable. Also reads .lgtmignore in the cwd, one gitignore-style pattern per line (evals/, **/fixtures/**, *.stories.test.ts)
--lean	send ~3x fewer tokens (8k of implementation, no test file or guidelines). Thresholds are calibrated on full context, so expect several times more false positives; only for rate limits or enormous test files
--no-cache	ignore the answer cache
--fail	exit 1 when there are findings
--fail-on-error	exit 1 when a block was skipped by an API error
--classes	list every test with its class before the findings
--verbose	also show 0.5-to-threshold findings
--list-checks	print the checks

Why

Coding agents are prolific test writers and terrible test critics. They mock whatever is inconvenient, assert that the mock was called, compute the expected value with the code under test, and hand you a suite where every line is covered and nothing is verified. A test that checks a trace's name. A test for the thing you decided not to build. Nobody reads those files. The PR says "added tests" and gets merged.

The bug that pages you lives in a seam. One side writes, the other reads, and every unit test mocked at least one of them to agree. No linter catches that. It's a judgment call, and judgment used to cost a senior engineer's afternoon per PR.

Reads like a review, runs like a linter. Every finding is one test, one smell, one sentence you can act on. 😐👏 mocks-seam-under-test means you tested the mock. 😐👏 reimplements-logic means the test and the implementation share the same bug. The summary tells you how much of your suite actually crosses a seam.

Opinionated by design. Few wide tests with real collaborators beat a hundred mocked units. Delete with confidence: a good audit shrinks the suite. And when the suite is clean, it says so.

Now it costs a cent. Jev bills $0.042 per million input tokens and answers in under a second. lgtm shows you the bill and the runtime before it spends, and caches every answer. Thresholds are fitted so that no labelled real test fires wrongly; the held-out numbers are in Evals.

What lgtm likes

A test that crosses a seam with both sides real and asserts that they agree. That test fails when the wiring breaks, which is how most things actually break.

It dislikes tests of one-line helpers (any real test of the feature covers them for free), tests that mock everything except the function name, and one-off assertions that would survive the feature being deleted. So the summary prints what your suite is made of: N contract-integration · N mocked-seam · N pure-logic. --classes lists every test with its class, which is the number to watch during an audit. Retiring three unit tests for one wider test that really fails is a win, not a coverage loss.

Checks

--format json adds a checks map with a longer explanation and the fix, once per check.

	check
😐⁠🤏	would-pass-if-broken	Remove the behaviour in the name and every assertion stays green: the fixture never reaches that branch. Move it to the failing side.
😐⁠🤏	vacuous-assertion	The assertion (toBeDefined, truthy, length ≥ 0) accepts wrong output too; pin the exact value a bug would change.
😐⁠🤏	assertion-weaker-than-name	The name promises a behaviour the assertions never check; assert it, or rename the test to what it proves.
😐⁠👏	reimplements-logic	The expected value is computed with the same logic as production, so both can be wrong together; write the expected value by hand.
😐⁠👏	mocks-seam-under-test	The collaborator that decides this behaviour is a mock, so the test proves the mock's script, not the code; use the real one here.
😐⁠👏	mock-mirrors-implementation	The mock re-encodes the production logic, so an implementation that agrees with the copy passes even when both are wrong; use the real collaborator or fixed data.
😐⁠🤏	tests-calls-not-outcomes	It checks that a function was called, not what the call changed; assert the resulting state or output.
😐⁠🤌	tests-internals	It asserts private state, class names or call order rather than observable behaviour, so a refactor breaks it and a bug does not; assert the output.
😐⁠🤌	setup-dominates	Most of the setup never reaches the assertion; cut it to what the assertion depends on, or assert more of it.
😐⁠🤏	broad-snapshot	The assertion is a snapshot of the whole output, so any change re-records it and nobody reads what changed; pin the fields that matter.
😐⁠🤏	swallowed-error-as-success	The test passes whether the error is thrown, caught or never raised; assert the specific failure by class, code or message.
😐⁠🤌	impossible-fixture	The fixture is a state production validation could never produce, so the branch it exercises cannot happen; build it through the real constructor or validator. (off by default; name it in --only)
😐⁠🤌	happy-path-only-of-risky-boundary	The refusal path this code exists for (the reject, the limit, the wrong tenant) has no test here or among its siblings; add one.
😐⁠🤌	trivial-primitive	A one-line helper tested on its own; any real test of the feature that uses it would catch the same break. Delete it, or test the feature.
😐⁠👏	over-mocked	Every asserted value came out of a fake; the only real code left is glue between stubs. Fake fewer collaborators, or test the integration.
😐⁠🫸	regression-does-not-distinguish	This regression test also passes on the pre-fix code, so it does not lock the fix; assert the value the bug got wrong. (needs --diff)
😐⁠🫸	changed-in-lockstep	The expected values changed in the same diff as the code that produces them, so the test may only mirror the new behaviour; derive them from the requirement. (needs --diff) (off by default; name it in --only)

Evals

Numbers on the held-out test set, which nothing was fitted or tuned on. lgtm prints two kinds of finding: a high-confidence one sits at or above its check's high-confidence line (0.15 above the threshold unless the check pins its own) and is what the verdict counts and --fail blocks on; a worth a look one sits between the threshold and that margin. "All flagged" below means both together, everything lgtm prints.

held-out	high-confidence findings	all flagged findings
precision	0.92 (49 findings, 4 wrong)	0.87 (99 findings, 13 wrong)
recall	0.33	0.63

Per-check numbers on the same held-out set, every miss and false positive, the corpus composition and the class confusion matrix are in evals/RESULTS.md; every scored case is a dot in evals/atlas.html, per check, with both lines drawn. The public cases reproduce with pnpm eval.

CI

- uses: actions/cache@v4
  with:
    path: node_modules/.cache/lgtm
    key: lgtm-${{ hashFiles('**/pnpm-lock.yaml', '**/package-lock.json') }}
    restore-keys: lgtm-
- run: npx @stardeckai/lgtm --diff origin/${{ github.base_ref }} --yes --format github
  env:
    TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}

The cache step is optional: --diff already limits a PR run to the tests it touched. Answers are keyed by the exact ordered request (test and evidence, selected questions, criteria and choices), model tag, and cache revision. An unrelated lgtm version bump does not invalidate answers. Question and choice order are preserved; changing the selected checks still needs a fresh request. Do not commit the cache directory; it never shrinks automatically.

lgtm is advisory by default: it prints findings and exits 0. Fetch enough history for the base ref (fetch-depth: 0) and add --fail once the findings are clean enough that you want them blocking.

Cost

TypeSafe and OpenRouter Jev 1.13 list $0.042 per million input tokens and nothing for output. Vercel billing follows its own model pricing; lgtm shows token estimates without a dollar estimate in Vercel mode. One request per test block. Each state is trimmed to at most 100,000 chars (~25,000 tokens, under TypeSafe's 32k state limit): the test code, the whole test file with the block fenced, the file's imports and sibling test names, up to 60,000 chars of the directly imported implementation and the test sections of any CLAUDE.md/AGENTS.md above it. --lean cuts that back to an 8,000-char implementation and no test file or guidelines.

run	blocks	input tokens	cost
a PR touching 20 tests (--diff)	20	~100k	~$0.004
a mid-sized suite	500	~2.5M	~$0.11
a large monorepo suite	5,000	~25M	~$1.05
the eval corpus (pnpm eval, cold)	868	see the Evals section

Answers are cached under node_modules/.cache/lgtm. An unchanged request costs no API calls, and changing only thresholds or output format reuses the stored probabilities. Each block's evidence includes the whole test file, so editing one test can invalidate every block in that file; edits to shared implementation can invalidate more. Malformed or incomplete cache entries are treated as misses by both the plan and the executor; cache I/O errors stop the run before sending the affected request. A cache-revision change requires a fresh run; --no-cache bypasses reuse, and clear-cache removes stored answers. The model tag is jev-latest for TypeSafe, typesafe-ai/jev for Vercel, and typesafe/jev-1.13 for OpenRouter. Cached answers do not automatically refresh when a provider changes a model behind a tag.

The implementation source is the main cost lever: --no-impl cuts the bulk of each request at the price of weaker would-pass-if-broken and reimplements-logic answers. Every run prints its input tokens and the estimated cost.

Config

All saved keys and the default provider live in ~/.config/lgtm/config.json. The selected provider’s matching environment variable (TYPESAFE_API_KEY, AI_GATEWAY_API_KEY, or OPENROUTER_API_KEY) wins over its saved key. With no saved key, an environment key selects its provider. lgtm --dry-run shows the selected mode and pace. Vercel requests start at most 10 per minute by default; use --rate <n> to adjust. The existing SDK retries 503 responses, while 429 responses also honor Retry-After. Vercel requests use its TypeSafe-compatible API. OpenRouter uses its Decisions API with model typesafe/jev-1.13 and the normal four-worker limit; use --rate if needed.

lgtm key                     # choose provider and paste a new key
lgtm key <new-key>           # replace the key for the default provider
lgtm key <new-key> --provider vercel  # save a Vercel key and make it the default
lgtm key <new-key> --provider openrouter  # save an OpenRouter key
lgtm key default             # show the default provider
lgtm key default set typesafe  # switch the default, keeping all keys
lgtm --provider vercel --dry-run src  # use Vercel for one run
lgtm usage                   # cost so far: all time, last day, last week, this worktree
lgtm clear-cache             # drop this project's cached answers (node_modules/.cache/lgtm)
lgtm skill                   # (re)install the /lgtm and /actually-test skills, e.g. to add another agent
lgtm init                    # both steps again

Thresholds

Each check has two lines. The threshold is fitted on the train cases to the lowest probability that fires on no real negative and keeps precision at or above 0.95, then one step up; a finding at or above it is worth a look. The high-confidence line sits 0.15 above that unless the check pins its own, and only findings above it count in the verdict and trip --fail. A few checks are pinned by hand where the fit would switch them off; each pin carries a comment saying why. That trades recall for precision on purpose: a finding should be worth your time. To see what sits just under the line, run --verbose, or --threshold 0.5 --format json on a suite you know well and pick your own number. A check that is consistently wrong for your codebase belongs in --skip.

Development

This repo uses pnpm.

Each check is one file under src/checks/<category>/; add a new one there and register it in src/checks/index.ts.

pnpm install
pnpm typecheck
pnpm test
pnpm build
node dist/cli.js --dry-run src   # no API key needed
npm link                         # expose this checkout as the global `lgtm` (symlink to dist/cli.js; rebuild to update)
pnpm gen:skill                   # regenerate skills/*/SKILL.md after changing a check or a skill (a test guards drift)
pnpm eval                        # run the labelled corpus against Jev; --offline re-scores, --fit-thresholds refits
node --import tsx scripts/cache-replay.ts # reproduce cache hit/miss scenarios without paid API calls

License

MIT