mm-sensor

Models are mighty these days — but not every model can read an image, let alone watch a video or listen to audio. Yes, GLM, little whale, I'm talking to you.

What is this?

mm-sensor is an agent skill. It feeds images, videos, and audio to an external multimodal model and gets back a structured text description. That's the whole job. It has one job, and it takes it with the seriousness of a border guard.

It solves a very real problem: large models without vision see nothing — yes, GLM and DeepSeek, that means you two.

What it does: look, watch, listen

Images

Screenshots, photos, diagrams, design mockups, icons, SVG — all fair game. It identifies layout, element hierarchy, text, and colors; at high detail it even reports border radii, font sizes, and margins. Supports PNG, JPG, JPEG, GIF, WebP, BMP, SVG.

Video

MP4, MOV, AVI, WMV, M4V, WebM, MKV, TS, FLV. Described as a timeline: scenes, people, actions, subtitles, visual details. File over the size limit? ffmpeg compresses it automatically. The only thing you need to do is pretend not to notice.

Audio

MP3, WAV, FLAC, M4A, OGG, AAC, OPUS... Transcription, summarization, speaker segmentation. The transcription quality is verified by actual testing — movie dialogue comes out nearly word-perfect.

Three detail levels

Level	Behavior	Good for
low	A quick glance: main elements, overall layout, key text	"just tell me what's on it"
medium (default)	An actual look: element hierarchy, positions, style overview, timeline segments	understanding the structure
high	A magnifying glass: full detail, exact styles, all text	pixel-faithful mockup restoration

Quick usage

# Look at an image (the most common move)
python3 ~/.config/opencode/skills/mm-sensor/vision.py /path/to/screenshot.png

# Watch a video (frame sampling, auto-compression)
python3 ~/.config/opencode/skills/mm-sensor/vision.py /path/to/demo.mp4

# Listen to audio (transcription + summary)
python3 ~/.config/opencode/skills/mm-sensor/vision.py /path/to/meeting.mp3

# High detail on a design mockup
python3 ~/.config/opencode/skills/mm-sensor/vision.py --detail high /path/to/mockup.png

# JSON instead of HTML
python3 ~/.config/opencode/skills/mm-sensor/vision.py --format json /path/to/image.png

# A mixed batch in one go
python3 ~/.config/opencode/skills/mm-sensor/vision.py /img1.png /video.mp4 /audio.mp3

# Probe: check what the configured model actually supports (free)
python3 ~/.config/opencode/skills/mm-sensor/vision.py --probe

Replace ~/.config/opencode/skills/mm-sensor/ with the absolute path where the skill actually lives.

The anti-laziness defense system (the good part)

Refusal detection & auto-retry

Models get stubborn, so there's a two-layer detection. A phrase layer: regex claims on the classics — media-perception verbs like "cannot view" / "cannot hear" are strong signals, and self-declarations like "I'm a text-only model" are caught too; with care taken so legitimate descriptions (like "the page shows an image failed to load") don't get caught in the crossfire. A structural layer: any response that tells you to "please provide..." while producing zero content evidence counts as dodging. Caught one? Retry once with a much firmer prompt. Still refusing? The output carries a visible quality warning: this content is not trustworthy. Your call what to do with it.

If regex feels too flimsy for your taste, you can swap in agentic monitoring. It burns noticeably more tokens, but it handles the model's "I don't wanna" much better. It also takes longer to decide.

Long-screenshot tiling

Screenshots over 4000px tall stump most tools, and their descriptions come out blurry. This one cuts them into 2048px strips (with a 128px overlap so no element gets severed mid-way) and describes them as a whole in a single call. Max 9 strips; beyond that it politely errors and tells you to split it yourself.

Auto-compression for oversized files

Video or audio over 50MB? ffmpeg quietly re-encodes — lower bitrate for video, mp3 for audio. By the time you notice, the result is already out.

Capability fallback & lazy fallback

Model can't do video? It degrades to 9 evenly sampled frames and describes them like a comic strip — with an honest label: this is a degraded result; the soundtrack, dialogue, and motion are lost. No audio capability? It skips with a clear notice instead of dying or pretending. And then there's "lazy fallback": even when configured as capable, if the API errors with a modal-related complaint, it retries once via frame-sampling — but genuine errors like rate limits won't trigger it, so real problems don't get buried.

Traceability

Every output carries a model field: you know who said this, and you can weigh it accordingly. VLM output is generative by nature, so two runs on the same input can differ slightly — high detail is best-effort, not pixel-perfect. Cross-check against the source for anything that matters.

For the web frontend folks

Building web apps? Generally speaking, image descriptions are almost always enough. Verifying a page renders right, checking a mockup, reading an error off a screenshot — one image call settles it. No need to burn tokens on video and audio.

Besides, the image path is the one polished the most: in my own day-to-day use I mostly lean on image capability anyway, and it's worked out well — it carried me through 3-4 real projects.

Configuration

Edit config.toml. The default is Xiaomi MiMo Token Plan (mimo-v2.5 — a native all-modal model that understands images, video, and audio together).

[vision]
provider = "openai"
base_url = "https://token-plan-cn.xiaomimimo.com/v1"
api_key = "tp-xxx"          # supports ${ENV_VAR} references
model_name = "mimo-v2.5"
max_tokens = 16384
temperature = 0.1
timeout = 300
max_media_mb = 50           # Base64 cap (MB); oversized files auto-compress
fps = 2.0                   # video frame rate [0.1, 10]

Image-only models (llava, qwen-vl, etc.) work too:

[vision]
provider = "openai"
base_url = "http://localhost:11434/v1"
api_key = "ollama"
model_name = "llava"
media_capabilities = ["image"]   # video auto-degrades to frames; audio skips with a notice

Anthropic (images only): set provider = "anthropic" and model_name = "claude-3-5-sonnet-20241022".

Testing & credibility

• Every test is an empirical verification against the real gateway — no mocks, no theater
• The baseline agent is opencode: all acceptance, regression, and tuning work happens in an opencode environment; any divergence on other agents defers to opencode behavior
• Coverage: audio format compatibility (aac, opus auto-transcode to mp3), URL passthrough, detail-field isolation, capability probe, refusal retries, tiling...

Deploying to each agent

At its core, this skill is a pile of files (SKILL.md + vision.py + config.toml + vision-spec.md). Any agent that honors the SKILL.md convention can host it. The directory conventions for the three mainstream agents:

Agent	Global directory	Project directory	Invocation
opencode	~/.config/opencode/skills/mm-sensor/	.opencode/skills/mm-sensor/	auto-loaded by skill description
Claude Code	~/.claude/skills/mm-sensor/	.claude/skills/mm-sensor/	auto-triggered, or /mm-sensor
Codex	~/.agents/skills/mm-sensor/	.agents/skills/mm-sensor/	auto-triggered, /skills picker, or $mm-sensor

opencode (the baseline agent)

opencode is the baseline agent for development, testing, and tuning. Every acceptance case, every refusal-retry tuning pass, was run in an opencode environment. If behavior differs on another agent, opencode's behavior and this documentation win the argument.

./install.sh    # Linux / macOS, pick 1 = global, 2 = project
install.bat     # Windows

Then put your own API key in config.toml and restart opencode.

Claude Code

# Global (available in all projects)
mkdir -p ~/.claude/skills
cp -r mm-sensor ~/.claude/skills/

# Or project-only
cp -r mm-sensor /path/to/project/.claude/skills/

SKILL.md carries name and description frontmatter per the Agent Skills standard, and Claude Code discovers it at startup. Let it trigger on the description, or invoke /mm-sensor yourself. Start a new claude session to pick it up.

Codex

Codex's skill support only exists in versions from December 2025 onward, and it may not be enabled by default. Check first:

codex features list    # is skills set to true?
codex --enable skills  # or add [features] skills = true to ~/.codex/config.toml

Then copy it over:

# Global (current standard dir; legacy ~/.codex/skills/ still works)
cp -r mm-sensor ~/.agents/skills/

# Or project-level (standard .agents/skills/; legacy .codex/skills/ still works)
cp -r mm-sensor /path/to/project/.agents/skills/

Three ways to use it: let Codex auto-load by description, pick it in /skills, or force it with $mm-sensor.

Notes for everyone

• The shared frontmatter is the open Agent Skills standard, so the directory must be named mm-sensor. Don't get creative.
• vision.py needs Python 3.9+ and requests; ffmpeg is optional but strongly recommended (auto-compression depends on it). Same for every agent.
• Agent doesn't support the skill mechanism? No problem — vision.py is a standalone CLI. python3 vision.py /path/to/img.png works anywhere; you just lose the discipline of "media must go through this door."
• This open-source copy is functionally identical to the dev version, minus the test suite, test media, and any real API key — config.toml ships with placeholder values. First thing after installing: put in your own key.

If manual copying feels like too much work, just hand this repository's URL to your agent and let it install itself — that's how you should handle things in the age of agents, isn't it?

Files

File	Purpose
SKILL.md	The skill itself: rules and calling conventions
vision.py	The one that actually does the work
vision-spec.md	Output format spec (HTML/JSON field definitions)
config.toml	Model configuration
README_zh.md / README.md	This documentation, in Chinese / English

Disclaimer

mm-sensor guarantees it went and looked. It cannot fully guarantee the model understood what it saw. For important scenes, eyeball it yourself — and for the record, making it replace YOLO is neither cost-effective nor reliable.