Detection Accuracy Report

June 10, 2026 · View on GitHub

This document describes what each static analysis pattern in pytest-mrt detects, what it misses, and the false-positive risk for each check. The false-positive suite (tests/test_false_positives.py) enforces the "will NOT trigger" column automatically on every CI run.

Last updated: 2026-06-10 · pytest-mrt v1.4.0 · 34 Alembic patterns + 10 Django patterns

Overall accuracy summary

The numbers below are measured against the false-positive test suite (tests/test_false_positives.py, 310 cases) and the detection test suite (tests/test_patterns.py, 44 pattern × multiple variants each).

Metric	Alembic	Django
Patterns covered	34	10
False-positive suite cases	248	62
Measured false-positive rate	0 % (0 / 248)	0 % (0 / 62)
Detection rate on synthetic cases	100 %	100 %
Known blind spots (architectural)	8	4

False-positive rate is enforced at 0 % by CI — the false-positive suite fails the build if any safe migration triggers a warning. The "Medium" or "Low" labels in the per-pattern table below refer to theoretical risk from code patterns that are syntactically similar to unsafe ones, not observed misfires.

Detection rate on real-world projects is not yet formally measured. If you find a pattern that pytest-mrt misses, please open an issue tagged accuracy.

How to read this table

Column	Meaning
Severity	`error` — blocks rollback; `warning` — may block or degrade
Will catch	Cases the pattern reliably detects
Will NOT catch	Known blind spots (not false negatives from bugs — architectural limits)
False-positive risk	How often the pattern fires on safe code

Alembic patterns (34)

Per-file checks

1. Missing downgrade


Severity	error
Will catch	Migration file with no `downgrade()` function at all
Will NOT catch	`downgrade()` that exists but does nothing useful (covered by #2)
False-positive risk	None — a missing function is unambiguous

2. No-op downgrade


Severity	error
Will catch	`downgrade()` whose body is `pass` or a bare docstring/comment
Will NOT catch	Downgrade that calls a helper which is itself a no-op
False-positive risk	None — AST check confirms the body is effectively empty

3. DROP COLUMN in upgrade


Severity	error
Will catch	Any `op.drop_column()` in `upgrade()`
Will NOT catch	Column dropped via raw `ALTER TABLE ... DROP COLUMN` SQL
False-positive risk	Low — intentional destructive drops should be in dedicated migrations with documented rationale

4. DROP TABLE in upgrade


Severity	error
Will catch	Any `op.drop_table()` in `upgrade()`
Will NOT catch	`DROP TABLE` via `op.execute()` raw SQL
False-positive risk	Low — same rationale as #3

5. TRUNCATE


Severity	error
Will catch	`op.execute("TRUNCATE ...")` or `op.execute(sa.text("TRUNCATE ..."))` in `upgrade()`
Will NOT catch	`DELETE FROM` without a WHERE clause (functionally equivalent but different SQL)
False-positive risk	None — TRUNCATE is always destructive

6. NOT NULL without default


Severity	warning
Will catch	`op.add_column` / `op.alter_column` with `nullable=False` and no `server_default` or `default`
Will NOT catch	NOT NULL added via raw SQL `ALTER TABLE ... SET NOT NULL` (covered by #25)
False-positive risk	Low — fires on new tables too; intentional cases can be suppressed with `# noqa: MRT201`

7. Column type change


Severity	warning
Will catch	`op.alter_column(..., type_=...)` in `upgrade()`
Will NOT catch	Type change via raw `ALTER TABLE ... ALTER COLUMN ... TYPE` SQL
False-positive risk	Medium — safe casts (e.g. `VARCHAR(50)` → `VARCHAR(100)`) also trigger; validate manually

8. Raw SQL (op.execute)


Severity	warning
Will catch	`op.execute()` in `upgrade()` with no corresponding `execute` in `downgrade()`
Will NOT catch	Safe read-only `SELECT` in execute (SELECT never needs reversal — but this check fires regardless)
False-positive risk	Medium — DDL-only migrations that also SELECT will trigger; a matching `execute` in downgrade suppresses it

9. Data transform without reverse


Severity	warning
Will catch	Bulk `UPDATE` in `upgrade()` with no `UPDATE` in `downgrade()`
Will NOT catch	`INSERT` or `DELETE` data transforms without reversal (partial coverage)
False-positive risk	Low — one-way data transforms are rarely intentional without documentation

10. CASCADE DELETE


Severity	warning
Will catch	`ondelete="CASCADE"` on any FK definition in `upgrade()`
Will NOT catch	Cascade behavior defined in database triggers outside migration files
False-positive risk	Medium — some schemas intentionally use cascade; audit each finding

11. INDEX without CONCURRENTLY


Severity	warning
Will catch	`op.create_index()` without `postgresql_concurrently=True`
Will NOT catch	Index creation via raw SQL without `CONCURRENTLY` keyword
False-positive risk	Medium — non-PostgreSQL databases don't support `CONCURRENTLY`; safe to ignore for SQLite/MySQL

12. ADD COLUMN with volatile DEFAULT


Severity	warning
Will catch	`op.add_column(..., Column(..., default=...))` — Python-side default rewrites the table on all PG versions
Will NOT catch	Default expressed as a callable or ORM-level event
False-positive risk	Low — Python-side defaults in migrations almost always cause table rewrites

13. ADD COLUMN with server_default


Severity	warning
Will catch	`op.add_column(..., Column(..., server_default=...))` — rewrites table on PostgreSQL < 11
Will NOT catch	Server defaults set via `op.execute("ALTER TABLE ... SET DEFAULT ...")`
False-positive risk	Medium — safe on PostgreSQL 11+; verify your database version

14. UNIQUE constraint on existing data


Severity	warning
Will catch	Any `op.create_unique_constraint()` in `upgrade()`
Will NOT catch	UNIQUE added via raw SQL or as part of a new table creation
False-positive risk	Medium — safe on empty tables or freshly populated ones; check for duplicates before deploying

15. DROP INDEX without reverse


Severity	warning
Will catch	`op.drop_index()` in `upgrade()` with no `op.create_index()` in `downgrade()`
Will NOT catch	Index drop via raw SQL
False-positive risk	Low — intentional permanent index removal is uncommon without documentation

16. DROP CONSTRAINT without reverse


Severity	warning
Will catch	`op.drop_constraint()` in `upgrade()` without a matching `create_foreign_key` / `create_unique_constraint` / `create_check_constraint` / `create_primary_key` in `downgrade()`
Will NOT catch	Constraint drop via raw SQL
False-positive risk	Low

17. rename_table without reverse


Severity	error
Will catch	`op.rename_table(old, new)` without `op.rename_table(new, old)` in `downgrade()`
Will NOT catch	Rename via raw SQL `ALTER TABLE ... RENAME TO`
False-positive risk	None — verifies both argument positions exactly

18. rename_column without reverse


Severity	error
Will catch	`op.alter_column(..., new_column_name=...)` in `upgrade()` without corresponding rename in `downgrade()`
Will NOT catch	Column rename via raw SQL
False-positive risk	None — checks for the presence of `new_column_name` kwarg

19. DROP VIEW without reverse


Severity	error
Will catch	`op.execute("DROP VIEW ...")` in `upgrade()` without `CREATE VIEW` in `downgrade()`
Will NOT catch	View dropped via dialect-specific API
False-positive risk	Low

20. SEQUENCE modification


Severity	warning
Will catch	`CREATE SEQUENCE`, `ALTER SEQUENCE`, or `setval(...)` in `upgrade()`
Will NOT catch	Sequence manipulation via ORM-level events
False-positive risk	Low — sequences are not transactional; the warning is almost always relevant

21. ENUM value added


Severity	error
Will catch	`ALTER TYPE ... ADD VALUE` in `upgrade()` (PostgreSQL-specific)
Will NOT catch	ENUM changes on MySQL (different syntax), ENUM type replacement
False-positive risk	None — `ADD VALUE` is irreversible once any row uses the new value

22. Multi-step destructive migration


Severity	error
Will catch	Migration that adds a column, bulk-updates data into it, and drops the original column — all in one step
Will NOT catch	Same pattern spread across multiple migrations
False-positive risk	Low — the combination of add + UPDATE + drop in one migration is inherently unsafe

23. NOT NULL without reverting nullable


Severity	warning
Will catch	`op.alter_column(..., nullable=False)` in `upgrade()` without `nullable=True` in `downgrade()`
Will NOT catch	NOT NULL set via raw SQL
False-positive risk	Low

24. NOT NULL via raw SQL without reverse


Severity	warning
Will catch	`ALTER TABLE ... SET NOT NULL` in `upgrade()` without `DROP NOT NULL` in `downgrade()`
Will NOT catch	NOT NULL enforced by a CHECK constraint
False-positive risk	Low

25. bulk_insert without reverse


Severity	warning
Will catch	`op.bulk_insert()` in `upgrade()` without `op.delete()` or `DELETE` SQL in `downgrade()`
Will NOT catch	Data inserted via `op.execute("INSERT ...")` — covered by #8
False-positive risk	Low

26. context.execute without reverse


Severity	warning
Will catch	`context.execute()` / `ctx.execute()` / `conn.execute()` / `connection.execute()` in `upgrade()` without a matching execute in `downgrade()`
Will NOT catch	Execute called through an arbitrary variable name
False-positive risk	Medium — same caveats as #8; a matching execute in downgrade suppresses it

27. DROP COLUMN in batch_alter_table


Severity	error
Will catch	`op.drop_column()` inside `op.batch_alter_table()` context manager in `upgrade()`
Will NOT catch	Drop occurring outside a batch context (covered by #3)
False-positive risk	None

28. DROP CONSTRAINT in batch_alter_table


Severity	warning
Will catch	`op.drop_constraint()` inside `op.batch_alter_table()` without recreating it in `downgrade()`
Will NOT catch	Constraint dropped outside batch context (covered by #16)
False-positive risk	Low

29. DROP FOREIGN KEY without restore


Severity	error
Will catch	`op.drop_constraint(type_='foreignkey')` in `upgrade()` without `op.create_foreign_key()` in `downgrade()`
Will NOT catch	FK dropped via raw `ALTER TABLE ... DROP FOREIGN KEY` SQL
False-positive risk	Low — referential integrity should always be restored on rollback

30. CREATE TRIGGER without DROP TRIGGER


Severity	error
Will catch	`CREATE TRIGGER` SQL in `upgrade()` without `DROP TRIGGER` SQL in `downgrade()`
Will NOT catch	Trigger created via a stored procedure or database-specific API
False-positive risk	Low

31. CREATE TYPE without DROP TYPE


Severity	error
Will catch	`CREATE TYPE` SQL in `upgrade()` without `DROP TYPE` SQL in `downgrade()`
Will NOT catch	Custom type created via SQLAlchemy TypeDecorator at ORM level
False-positive risk	Low — custom types left behind block re-running the migration

Cross-migration graph checks

G1. Multiple migration heads


Severity	error
Will catch	Two migration files that both reference the same `down_revision` (branched history)
Will NOT catch	Merge migrations already using a tuple `down_revision`
False-positive risk	None — detects unresolved branches that will cause `alembic upgrade head` to fail

G2. Data hole chain


Severity	warning
Will catch	Migration A drops column X; migration B adds column X back — schema is restored on rollback but original data is permanently gone
Will NOT catch	The same pattern across non-adjacent migrations in a long chain
False-positive risk	Low — the structural pattern (drop then re-add same column name) is rarely coincidental

G3. Orphaned migration


Severity	warning
Will catch	Migration files unreachable from any current head in the dependency graph
Will NOT catch	Orphans in branches that were intentionally kept separate
False-positive risk	Medium — feature branches and squashed migrations may appear as orphans; use `# noqa: MRTXXX` on the line to suppress

Django patterns (10)

D1. RemoveField


Severity	error
Will catch	`migrations.RemoveField(...)` — permanent column data loss on rollback
Will NOT catch	Field removed via `RunSQL`
False-positive risk	None

D2. DeleteModel


Severity	error
Will catch	`migrations.DeleteModel(...)` — all table data lost on rollback
Will NOT catch	Table dropped via `RunSQL`
False-positive risk	None

D3. AddField NOT NULL (no default)


Severity	error
Will catch	`AddField` with `null=False` and no `default=` — will fail on non-empty tables
Will NOT catch	NOT NULL field added via `RunSQL`
False-positive risk	Low

D4. AlterField NOT NULL (no default)


Severity	warning
Will catch	`AlterField` changing an existing field to `null=False` without a `default=`
Will NOT catch	NULL constraint change via `RunSQL`
False-positive risk	Low

D5. RunSQL without reverse_sql


Severity	warning
Will catch	`RunSQL(forward_sql)` without a `reverse_sql` argument
Will NOT catch	Reverse SQL that is present but logically incorrect
False-positive risk	Low — read-only SQL in `RunSQL` is uncommon; add `reverse_sql=""` to document intentional one-way operations

D6. RunSQL with TRUNCATE/DROP


Severity	error
Will catch	`RunSQL` containing `TRUNCATE` or `DROP TABLE`
Will NOT catch	Destructive SQL in a helper called from `RunSQL`
False-positive risk	None

D7. RunPython without reverse_code


Severity	warning
Will catch	`RunPython(forwards_func)` without `reverse_code=` argument
Will NOT catch	Reverse function that exists but does nothing (not checked for no-op)
False-positive risk	Low

D8. RenameModel without reverse


Severity	error
Will catch	`migrations.RenameModel(old_name, new_name)` — checks that downgrade performs the inverse rename
Will NOT catch	Rename via `RunSQL`
False-positive risk	None

D9. AddIndex without atomic=False


Severity	warning
Will catch	`AddIndex` on a migration without `atomic = False` — `CREATE INDEX CONCURRENTLY` requires a non-atomic migration on PostgreSQL
Will NOT catch	Index added via `RunSQL` with `CONCURRENTLY`
False-positive risk	Medium — only relevant for PostgreSQL with CONCURRENTLY; MySQL does not have this requirement

D10. Dangerous DDL without atomic=False


Severity	warning
Will catch	`RemoveField` or `DeleteModel` inside an atomic migration — on some databases these cannot be rolled back even within a transaction
Will NOT catch	Multi-database setups where only one database requires non-atomic
False-positive risk	Low

Summary

Scope	Total patterns	Error	Warning
Alembic per-file	31	13	18
Alembic graph	3	1	2
Django	10	5	5
Total	44	19	25

False-positive risk distribution across all 44 patterns:

Risk level	Count
None	17
Low	17
Medium	8

The false-positive test suite (tests/test_false_positives.py) enforces 30+ cases that must produce zero warnings, covering the most common medium-risk patterns.

Suppressing false positives

When a pattern fires on intentional code, suppress it with # noqa: MRTxxx on the offending line (same convention as ruff/flake8):

def upgrade():
    op.drop_column("users", "legacy_col")  # noqa: MRT103
    op.alter_column("orders", "amount", type_=sa.Numeric)  # noqa: MRT202

Use a bare # noqa to suppress all MRT warnings on a line. The legacy # mrt: ignore syntax is also supported.

Reporting accuracy issues

If you find a false positive or false negative, open an issue tagged accuracy at github.com/croc100/pytest-mrt/issues. Include the migration file and the unexpected finding.