subtitle-skill

Words make video reach further — deterministically.

Typed subtitle documents · Local execution · No AI reasoning · No cloud
Part of the same execution-skill ecosystem as ffmpeg-skill

pip install -e .

subtitle-skill is not "AI that writes captions." It is the execution layer that turns a subtitle decision — already made by an upstream agent — into a real artifact: validated, deterministic, verified, and traceable. It never decides what a caption should say; it decides whether the timeline is valid, generates SRT/WebVTT, and — by delegating to ffmpeg-skill's caption tool — burns it into a video, then checks that the result is actually correct.

If a typed SubtitleDocument and (for burn-in) an ffmpeg-skill install are on hand, it works: no network access, no API keys, nothing decided that wasn't handed to it.

---

Contents Why · Quick start · How it works · Design principles · Operations · Format support · Validation is not cosmetic · Built for agents · ffmpeg-skill integration · Verified · Install · Development · Docs · Support

---

Why

An agent that "handles captions" tends to guess: it burns a subtitle past the end of the video and calls it done, it forgets that a WebVTT file can't be burned in by the tool underneath it, or it reports success because a process exited 0 without checking what actually got written. subtitle-skill exists to take the guessing out of the execution half of captioning:

• Typed input, not a dict. A subtitle is a SubtitleDocument of SubtitleCues — parsed and validated once, never touched as raw JSON again.
• Fatal vs. observation, never confused. A broken timeline is rejected before anything is written. A borderline one (overlapping cues, a line that's too long) is reported, never silently "fixed" — no cue is ever deleted, reordered, merged, or shifted.
• Verification after execution. A render is only "done" when the output file exists, has a video stream, and its duration matches the actual, measured input — not when a subprocess happens to exit 0.
• A contract the agent can read. contract --json states exactly which operations, formats, parameters and error codes this version supports. doctor --json states what can actually run right now.
• No decisions made here. What the text says, how it's split, which speaker to show — that's video-production-agent's job. subtitle-skill only ever receives the result, as a typed request.

Quick start

pip install -e .
subtitle-skill doctor --json     # what can actually run on this machine
subtitle-skill contract --json   # operations, formats, parameters, errors

Generate an SRT file from a typed subtitle document — no video needed:

cat > request.json <<'EOF'
{
  "operation": "generate",
  "workspace": "/tmp/subtitles-demo",
  "format": "srt",
  "output_path": "captions.srt",
  "subtitle": {
    "id": "doc-1",
    "language": "en",
    "cues": [
      {"id": "c1", "start": 0.0, "end": 2.5, "text": "Hello and welcome"},
      {"id": "c2", "start": 2.5, "end": 5.0, "text": "to the show.", "speaker": "Host"}
    ]
  }
}
EOF
subtitle-skill run request.json --json

{"status": "ok", "operation": "generate", "output": ".../captions.srt",
 "sha256": "4ffe0640...", "size": 103, "reused": false,
 "observation": [], "timeline": {"cue_count": 2}, ...}

Burning that same subtitle into a video — "operation": "render" — needs format: "srt" and a reachable ffmpeg-skill install (see ffmpeg-skill integration); the request shape is identical otherwise, just add video_input. Add "mode": "mux" instead of the default "burn" for a soft, toggleable subtitle track rather than pixels burned into the picture, and "audio_stream": 1 (0-based) to pick a non-default audio track on a multi-track input — see Operations.

How it works

flowchart LR
    T[transcription-skill] -->|TranscriptSegment| A[video-production-agent]
    A -->|decides text, splitting,<br/>speaker labels, timing| A
    A -->|typed SubtitleDocument| S[subtitle-skill]
    S -->|render only| E[ffmpeg-skill / caption]
    E --> V[video with burned-in captions]
    S -->|generate| F[SRT / WebVTT file]

subtitle-skill sits at the bottom of that chain, as a leaf execution skill: it receives an already-decided, typed request and executes it mechanically. Internally:

flowchart TD
    R[Typed request] --> P[Parse + security screen<br/>recursive forbidden-key check]
    P --> V[Validate<br/>fatal errors reject, observations report]
    V --> G[Generate SRT / WebVTT]
    G -->|render only| B[ffmpeg-skill probe → caption]
    B --> O[Verify output<br/>exists, has video, duration matches]
    G --> ID[Compute deterministic identity]
    O --> ID
    ID --> C{Cache hit?}
    C -->|yes, sha256 verified| Reused[Return cached result]
    C -->|no| W[Write output + provenance sidecar]

Design principles

These are the rules the code enforces — not aspirations.

1. Typed, not stringly. Every subtitle is a SubtitleDocument / SubtitleCue / SubtitleStyle — parsed through from_dict once; nothing downstream touches a raw dict.
2. Fatal and observation are never the same code path. start < 0, end <= start, NaN/Infinity, duplicate ids, empty/invalid text, and a cue past the video's real duration reject the request outright. Overlaps, long lines, short/long cues, and reading speed are reported in observation and never auto-corrected.
3. The request's video_duration is a hint, not a fact. For render, the document is re-validated against ffmpeg-skill's own measured duration of the actual video — an omitted or wrong hint can't let a cue past the real end of the video through silently.
4. No shell, no arbitrary executable, no arbitrary filter. Every subprocess call is a fixed argv list. command, argv, shell, executable, filter, filter_complex, vf, af, env, api_key are rejected anywhere in the request, at any nesting depth.
5. Workspace-confined paths. Absolute paths, drive-qualified paths, .., and Windows reserved device names are rejected; a resolved (symlink-following) path must still land inside the workspace root.
6. Exit code 0 is never sufficient. A render is accepted only when ffmpeg-skill reports "status": "completed", the output file exists and is non-empty, it has a video stream, and its duration matches the input within 0.25s.
7. Deterministic identity, content-addressed. Same document, format, constraints (and for render, the same video and the same ffmpeg-skill script content) → same identity → a verified cache hit, not a re-render. A corrupted or hash-mismatched cache is never returned as reused.
8. Machine-readable contract. contract --json is generated from what's actually implemented — an operation or format that isn't executable doesn't appear as supported.

Operations

Two operations exist. Both appear in contract --json; nothing else does — convert, offset, merge, and ASS/SSA are not implemented and cannot be called.

Operation	Video I/O	Formats	What it does
generate	none	SRT, WebVTT	Validate the document, write a subtitle file
render	required	SRT only	Validate against the real video duration, generate the SRT, delegate burn-in (or mux) to ffmpeg-skill's caption tool, verify the output

render's mode (default "burn", or "mux") and audio_stream (0-based, optional) map directly onto ffmpeg-skill caption.py's own --mode/--audio-stream flags — see ffmpeg-skill integration for exactly what each does and subtitle.language's forwarding as the mux language tag.

contract --json also publishes provides: generate as Capability id subtitle.generate, render as subtitle.render — the cross-repository identifiers a registry (see kajisho5/AI-video-production-OS) can resolve to this Skill without hardcoding it. Additive; not a behavior change to either operation.

{
  "operation": "generate | render",
  "workspace": "/absolute/path/to/workspace",
  "format": "srt | vtt",
  "output_path": "relative/output.srt",
  "video_input": "relative/input.mp4",
  "video_duration": 123.4,
  "mode": "burn | mux",
  "audio_stream": 0,
  "subtitle": {
    "id": "doc-1", "language": "ja",
    "cues": [{"id": "c1", "start": 0.0, "end": 2.0, "text": "...", "speaker": "A", "style": {"align": "center"}}]
  },
  "constraints": {"max_chars_per_line": 42, "max_lines": 2, "min_duration": 0.5, "max_duration": 10, "reading_speed_cps": 20}
}

video_input / video_duration / mode / audio_stream apply to render only (mode defaults to "burn"; audio_stream defaults to caption.py's own default, track 0); constraints is optional for both (see Validation).

Format support

	generate	render (burn-in)
SRT	✅	✅
WebVTT	✅	❌ UNSUPPORTED_FORMAT

Generating a WebVTT file does not mean it can be burned in. This is a real constraint of the tool render delegates to — ffmpeg-skill's caption burns SRT or ASS, never WebVTT — not an arbitrary restriction. Requesting render with format: "vtt" fails immediately, before anything is generated.

Neither generator silently drops style it can't represent — it raises UNSUPPORTED_FORMAT instead:

	SRT	WebVTT
align / position / line / size	rejected (no native positioning)	native cue settings
bold / italic	<b> / <i> tags	<b> / <i> tags
color	rejected	rejected (no safe, non-CSS-injecting form here)
speaker	"Speaker: " text prefix	"Speaker: " text prefix

Cue/document metadata is auxiliary/provenance data — never written into the rendered subtitle body of either format.

This table describes the SRT/WebVTT text files themselves — what a plain subtitle file can and can't encode on its own. render's actual burn additionally forwards a subset of style to ffmpeg-skill's caption tool as its own flags (a completely separate mechanism from anything inside the SRT file) — see What SubtitleStyle forwards to caption.py below for exactly what.

Validation is not cosmetic

Fatal and observation are separate systems, not a severity slider:

Fatal — nothing is written: start < 0, end <= start, NaN/Infinity timestamps, duplicate cue ids, empty/whitespace-only or invalid-Unicode/control-character text, a cue past the video's actual duration.

Observation — reported, never auto-corrected: CUE_OVERLAP, TOO_MANY_LINES, LINE_TOO_LONG, CUE_TOO_SHORT, CUE_TOO_LONG, READING_SPEED_TOO_HIGH.

Thresholds (max_chars_per_line, max_lines, min_duration, max_duration, reading_speed_cps) are caller-supplied constraints, not hard-coded domain assumptions — defaults exist only so a caller may omit constraints entirely.

Built for agents

subtitle-skill is a process boundary meant to be called by another agent, not a general-purpose CLI tool.

subtitle-skill contract --json   # every operation, format, parameter, error this version supports
subtitle-skill doctor --json     # what can actually run right now
subtitle-skill run req.json --json   # execute one operation — always one JSON document on stdout

contract --json is generated to match what's implemented, not maintained beside it by hand: deterministic: true, per-operation formats, the full error-code list under errors, and out_of_scope naming what this skill deliberately does not do. doctor --json reports render as available only when an ffmpeg-skill install is found and its own doctor confirms the caption tool's required capabilities (ffmpeg, ffprobe, encoder:libx264, encoder:aac, filter:subtitles) — subtitle-skill asks ffmpeg-skill's own detection rather than re-implementing FFmpeg capability probing.

SKILL.md is the agent-facing usage guide: when to call this skill, how to build a SubtitleDocument, the fatal/observation distinction, and failure handling — written so an agent doesn't have to read the source to use this correctly.

No MCP server, plugin loader, or agent-framework integration exists in this repository — only the CLI process boundary above.

Deterministic identity and reuse

A content-addressed identity is computed from the skill version, contract version, operation, the full canonical subtitle document, format, and constraints — never from timestamps, PIDs, or temp paths. For render, it additionally includes the input video's sha256 and a sha256 of the ffmpeg-skill scripts that will actually execute (caption.py + _common.py) — a content hash, not ffmpeg-skill's self-reported package.json version string, because a version string is only as trustworthy as whoever last edited it. If an output and its <output>.subtitle-skill.json sidecar already match the current identity, the sidecar's recorded sha256 is re-verified against the file on disk before being reported as "reused": true; a corrupted or hash-mismatched output is regenerated, never returned as reused.

Provenance

Every artifact's sidecar and response record how it was produced: skill / skill_version / contract_version, operation, sha256, size, reused, observation, timeline, and for render: engine ("ffmpeg-skill"), engine_version (display only — its package.json version, or the literal string "unknown" if it has none — always a non-empty string, never null, since a known consumer treats a falsy engine_version on a render response as invalid rather than retryable), engine_script_sha256 (the real identity anchor, above), and engine_response (ffmpeg-skill's own reported commands and probe — always its actual values, never fabricated here).

Security

• No shell (shell=True is never used); every subprocess call is a fixed argv list.
• No caller-supplied executable path, FFmpeg filter string, or raw command, at any layer.
• Every request, at any nesting depth, is screened before parsing: command / argv / shell / executable / filter / filter_complex / vf / af / env / api_key anywhere reject it.
• Workspace-confined, symlink-aware PathPolicy: absolute/drive paths, .., and Windows reserved names rejected; a resolved path must still land inside the workspace root.
• Typed, stable error codes everywhere — see errors in contract --json.

This is what subtitle-skill refuses to do, not a claim that it's unbreakable.

ffmpeg-skill integration

render delegates burn-in to ffmpeg-skill's caption tool. There is no single dispatch endpoint in ffmpeg-skill — every tool is its own script:

python3 <ffmpeg-skill-install-dir>/scripts/<tool>.py [args] --json

subtitle_skill.engine:

1. locates the ffmpeg-skill install directory via SUBTITLE_SKILL_FFMPEG_SKILL_DIR, or the directories ffmpeg-skill's own installer writes to (~/.claude/skills/ffmpeg-skill, ~/.cursor/skills/ffmpeg-skill, ~/.codex/skills/ffmpeg-skill, ./.claude/skills/ffmpeg-skill);
2. runs scripts/probe.py <video> --json first, to confirm a video stream exists and measure the actual duration for validation;
3. runs scripts/caption.py <video> --srt <srt> -o <output> [--mode mux] [--audio-stream N] [--language <lang>] [--color RRGGBB] [--bold] [--size N] --json — a fixed argv list, never a shell. --mode mux is added only when request.mode == "mux" (the default, "burn", sends no --mode flag at all, matching caption.py's own default); --audio-stream only when request.audio_stream is given; --language only alongside --mode mux (caption.py's --language also feeds --transcribe, which subtitle-skill never uses, so it is otherwise omitted) and only when subtitle.language is non-empty; --color/--bold/--size only for mode: "burn" and only for whichever of style.color/style.bold/style.size the document actually sets — see What SubtitleStyle forwards to caption.py below;
4. accepts the result only when exit code 0, "status": "completed", a non-empty output file, a probe.video in the response, and an output duration within 0.25s of the input's — all hold, for both modes (--mode mux copies video/audio untouched, so the duration check still applies and still passes).

mode: "mux" is a real, distinct behavior from the default "burn", not a variant of it: caption.py --mode mux copies the input's video and audio streams byte-for-byte and adds the SRT as a separate, player-toggleable subtitle stream, rather than rendering the text into the picture. audio_stream (0-based) selects which audio track of a multi-track input caption.py keeps, for either mode — caption.py itself probes the input and rejects an out-of-range value (mapped to INVALID_INPUT here, like any other kind: "input" failure). subtitle.language — required and BCP47-validated by models.py, but otherwise never read — is forwarded as --language only for mode: "mux", tagging the newly added subtitle stream's language metadata (-metadata:s:s:N language=..., built by caption.py itself). That value is forwarded exactly as given, not normalized: confirmed directly against real ffmpeg, .mp4/.m4v/.mov output (mov_text, the codec caption.py picks for those extensions) silently drops a language tag that is not a 3-letter ISO 639-2 code — a plain 2-letter BCP47 tag like "ja"/"en" can vanish from that container's stream tags with no error, warning, or non-zero exit anywhere in the chain, while .mkv (srt) and .webm (webvtt) output write the same value verbatim. Pick .mkv/.webm output when the exact tag value must survive.

What SubtitleStyle forwards to caption.py

SubtitleStyle (align, position, line, size, bold, italic, color) is a per-cue field on this skill's own document model. ffmpeg-skill's caption.py styling, in contrast, is a single, whole-burn force_style setting applied once per invocation — there is no per-cue equivalent on ffmpeg-skill's side at all. Reconciling those two shapes losslessly is only possible when every cue that sets a given field agrees on its value (kajisho5/subtitle-skill#5):

SubtitleStyle field	caption.py flag	Conversion	Notes
color	--color	none (forwarded verbatim)	caption.py's own color_hex() normalizes "#RRGGBB"/"RRGGBB"/"0xRRGGBB" and fails cleanly (INVALID_INPUT) on anything else; models.py does not itself validate the string.
bold	--bold	True → flag present; False/None → omitted	--bold is action="store_true" — there is no --no-bold to send for False, and omitting it already matches caption.py's own default.
size	--size	points = round(size / 100 × 288)	size is documented as "0..100, percent" with no stated percent-of-what; caption.py's own --size help text says its unit is "ASS points relative to a 288p script height, scales automatically". Percent-of-that-same-288-line baseline is the one interpretation that lines up with a number caption.py's own author already chose — confirmed empirically, not just from the help text, by burning the same --size into two real videos of different heights and measuring the rendered glyph's pixel height in each: it scaled proportionally with the real video height.
align	(none)	—	caption.py has no --align distinct from --position; --position's vocabulary (see next row) is not a lossless target for pure left/center/right text justification — e.g. its "center" means the screen's dead center (ASS Alignment 5), not "bottom, center-justified" (Alignment 2, caption.py's own default), so mapping align="center" onto it would silently relocate the caption vertically too. Left unmapped, not guessed at.
position / line	(none)	—	0..100 percent, from-left/from-top coordinates. caption.py has no numeric/percent placement for an SRT burn at all — only 7 named anchors (bottom, top, center, bottom-left, bottom-right, top-left, top-right) plus one uniform --margin. There is no percent-to-bucket formula that doesn't throw away the caller's actual number. Left unmapped, not guessed at.
italic	(none)	—	caption.py's plain-SRT force_style string never includes an Italic= key (only Bold= is exposed there). Smaller gap than it looks in practice: generate_srt already wraps a cue's text in <i>...</i> when style.italic is set, and libass honors that inline SRT tag on its own, independent of force_style — so per-cue italic already renders correctly, just not via anything render forwards as an argv flag.

Wired fields only take effect for mode: "burn" — caption.py's own docs say styling has no meaning for a soft-muxed subtitle stream, so mode: "mux" silently ignores none of it: setting color/bold/size with mode: "mux" is rejected up front (UNSUPPORTED_FORMAT, from the plain-SRT-generation step, same as align/position/line always are) rather than accepted and silently having no effect.

Because color/bold/size can only be sent once per burn, render requires every cue that sets one of them to agree on the value: two cues with the same style.color render fine (one, correctly-shared, --color call); two cues with different style.color values raise INVALID_INPUT rather than silently keeping one cue's request and dropping the other's. align/position/line are rejected the same way generate already rejects them for a plain SRT file (UNSUPPORTED_FORMAT) — they have no burn-time representation either, so there is nothing render-specific to exempt them into.

Failure responses follow ffmpeg-skill's own shape — {"status": "failed", "error": {"kind": "input"|"ffmpeg"|"missing_tool", "message": "..."}} — mapped to INVALID_INPUT / TOOL_ERROR / DEPENDENCY_ERROR respectively; anything ffmpeg-skill didn't produce at all (crash, timeout, malformed stdout) is DEPENDENCY_ERROR.

ffmpeg-skill's own contract marks caption as needing visual verification (ffmpeg-skill/look, a PNG contact sheet, for a human or agent to inspect). subtitle-skill deliberately does not run or interpret look — judging whether burnt-in captions look right is exactly the visual/AI judgement this skill's mandate excludes. That verification, if wanted, belongs to whoever is driving the render.

Verified

Result	Measurement
148 / 148	full test suite — models, validation, formats, security, PathPolicy, CLI/contract, doctor, the Agent Skill installer, engine boundaries, SubtitleStyle → caption.py wiring, and render delegation (burn and mux)
against real ffmpeg-skill	render tests run a vendored, byte-identical copy of ffmpeg-skill's actual caption.py / probe.py / _common.py / _contract.py (kajisho5/ffmpeg-skill, skill version 0.12.2) — not a hand-rolled stub — including a real burn-in and a real mux, each verified by ffprobe, and by asserting ffmpeg-skill's own reported command line used the right flags (subtitles= for burn, --mode mux / --audio-stream N / --language for mux, --color/--bold/--size for styled burns)
vendor drift checked weekly	scripts/check_vendor_drift.py (.github/workflows/vendor-drift.yml) clones current ffmpeg-skill main and diffs it against the vendored copy, separately from normal-PR CI
6 CI jobs green	Ubuntu, macOS, Windows × Python 3.9, 3.11
cache correctness proven both directions	a bare ffmpeg-skill version bump with unchanged scripts does not invalidate the cache; a script content change with an unbumped version does

pytest -q

Only the Linux CI job is guaranteed to have ffmpeg on PATH; the real-burn-in tests skip gracefully elsewhere, so "CI green" means "logic verified everywhere," not "burn-in verified identically on every OS."

Install

pip install -e .          # from a checkout; not yet published to PyPI
subtitle-skill install    # place SKILL.md for Claude Code (--cursor, --codex, --all, --project, --dir, --uninstall)

install only places SKILL.md for agent discovery — it does not copy the runtime. The subtitle-skill command itself must already be on PATH from the pip install step above; unlike ffmpeg-skill's standalone scripts, this is a regular installed Python package.

Requirements

• Python 3.9+ (declared in pyproject.toml; the floor version is actually exercised in CI, not just declared)
• For render only: an ffmpeg-skill install reachable via SUBTITLE_SKILL_FFMPEG_SKILL_DIR or its standard install locations, itself requiring FFmpeg with libx264, aac, and the subtitles (libass) filter — doctor --json reports exactly what's missing.

Development

pip install -e .
pip install pytest
pytest -q

CI (.github/workflows/ci.yml) runs on Ubuntu, macOS, and Windows, on Python 3.9 and 3.11 (6 jobs). PathPolicy behavior is consistent across all three OSes.

Docs

SKILL.md	what an agent reads: when to call this skill, building a SubtitleDocument, fatal vs. observation, failure handling
contract --json	the authoritative, machine-readable operation/format/error list — if this README and the live contract ever disagree, the contract is correct
tests/fixtures/ffmpeg_skill_vendor/README.md	provenance of the vendored ffmpeg-skill scripts used in the render tests
CLAUDE.md	repository status for maintainers: ecosystem integration state, known gaps, next tasks

There is no separate docs/ directory at this time.

Limitations / Out of scope

Not implemented, and not this skill's mandate — these belong to video-production-agent (decision-making) or a dedicated skill: automatic transcription, speaker diarization, translation, summarization, AI-generated or AI-edited subtitle text, automatic cue splitting, automatic subtitle "design" decisions, scene/semantic understanding, cloud upload, MCP or plugin-loader integration, arbitrary FFmpeg command or filter execution.

Not yet implemented, and not advertised in contract --json: convert, offset, merge, ASS/SSA generation or rendering.

Support

If this skill saves you time, you can help keep it maintained through GitHub Sponsors. Issues and pull requests are just as welcome.

License

MIT