VideoARM-Skill 🎬

March 18, 2026 Β· View on GitHub

This repository contains the official OpenClaw implementation of VideoARM accepted at CVPR 2026. VideoARM is a coarse-to-fine video reasoning paradigm over hierarchical multimodal memory (HM3) for long-form video understanding. By leveraging the proposed complementary toolsets and HM3, VideoARM can progressively localize, interpret, and abstract evidence in an adaptive observe–think–act–memorize loop. Extensive experiments on prevalent long-video understanding benchmarks demonstrate that VideoARM maintains strong performance while significantly reducing token consumption.

How It Works

Main Session (clean context)
  └── spawn VideoQA Controller
        β”œβ”€β”€ videoarm-download    β†’ download YouTube / URL
        β”œβ”€β”€ videoarm-info        β†’ get metadata (fps, duration, frames)
        β”œβ”€β”€ videoarm-audio       β†’ transcribe audio segments
        β”œβ”€β”€ videoarm-extract-frames β†’ extract frame grids
        β”œβ”€β”€ spawn Image Analyzer β†’ analyze frames in clean context
        └── return final answer

The controller runs in an isolated sub-agent to keep the main session context clean. Frame analysis is delegated to further sub-agents β€” no vision API keys needed, just OpenClaw's built-in image understanding.

Installation

Prerequisites:

  • Python 3.10+
  • ffmpeg (brew install ffmpeg / apt install ffmpeg)
  • OpenClaw running
git clone https://github.com/qiankemeng/VideoARM-skill.git
cd VideoARM-skill

# Basic install
pip install -e .

# With audio transcription
pip install -e ".[audio]"

# With YouTube download
pip install -e ".[download]"

# Everything
pip install -e ".[all]"

After installing, verify your environment:

videoarm-doctor

This checks Python version, ffmpeg, yt-dlp, Python packages, and Whisper model status.

Quick Start

# Install
git clone https://github.com/qiankemeng/VideoARM-skill.git
cd VideoARM-skill && pip install -e ".[all]"

# Verify installation
videoarm-doctor

# Try it out
videoarm-info /path/to/video.mp4
videoarm-audio /path/to/video.mp4 --start 0 --end 60
videoarm-extract-frames --video /path/to/video.mp4 \
  --ranges '[{"start_frame":0,"end_frame":1500}]' --num-frames 20

Configuration

Platform Support

VideoARM works with any video source that yt-dlp supports, including:

  • YouTube
  • Bilibili (ε“”ε“©ε“”ε“©)
  • Twitter/X
  • TikTok / ζŠ–ιŸ³
  • Local files (.mp4, .mkv, .avi, etc.)

For region-restricted content, set HTTPS_PROXY in your .env file.

Audio transcription works out of the box β€” faster-whisper is included as a default dependency. No API keys needed.

On first run, the Whisper base model (~150MB) is automatically downloaded.

Multilingual Support

Whisper auto-detects the language β€” no configuration needed. It supports 99+ languages. For better accuracy with non-English videos, use a larger model:

# In .env
WHISPER_MODEL=small   # better for non-English, accents, dialects

Optional: Use a cloud API instead

If you prefer faster transcription or better accuracy, set a cloud API in .env:

cp .env.example .env
OptionSetupSpeed
Local (default)Nothing to configure~1x realtime on CPU
GroqSet WHISPER_API_KEYVery fast, free tier
OpenAISet WHISPER_API_KEYFast, ~$0.006/min

Local model selection

Set WHISPER_MODEL in .env to choose model size:

ModelSizeRAMAccuracy
tiny39MB~1GBBasic
base74MB~1GBGood (default)
small244MB~2GBBetter
medium769MB~5GBGreat
large-v31.5GB~10GBBest

Usage with OpenClaw

As an OpenClaw Skill

Place this directory in your OpenClaw workspace. When you ask a video question, the agent reads SKILL.md and spawns a controller sub-agent automatically.

User: Analyze this video and tell me who opened the watermelon
      https://www.youtube.com/watch?v=...

Agent: [reads SKILL.md β†’ spawns controller β†’ returns answer]

CLI Tools (Standalone)

Each tool works independently from the command line:

# Get video metadata
videoarm-info video.mp4

# Extract 30 frames from a time range (returns grid image path)
videoarm-extract-frames --video video.mp4 \
  --ranges '[{"start_frame":0,"end_frame":1500}]' \
  --num-frames 30

# Transcribe audio (start/end in seconds)
videoarm-audio video.mp4 --start 0 --end 300

# Download from YouTube
videoarm-download "https://www.youtube.com/watch?v=..."

CLI Reference

CommandDescriptionOutput
videoarm-info <video>Video metadataJSON: fps, total_frames, duration, has_audio
videoarm-extract-framesExtract frame grid imageJSON: image_path, frame_ranges
videoarm-audio <video>Transcribe audio segmentJSON: transcript, segments[]
videoarm-download <url>Download videoJSON: path, cached
videoarm-doctorCheck all dependenciesHuman-readable or --json
videoarm-cleanClean temporary filesHuman-readable or --json; supports --dry-run, --downloads

Project Structure

VideoARM-skill/
β”œβ”€β”€ SKILL.md              # OpenClaw skill instructions (the brain)
β”œβ”€β”€ videoarm_cli/          # CLI tools
β”‚   β”œβ”€β”€ videoarm_info.py
β”‚   β”œβ”€β”€ videoarm_extract_frames.py
β”‚   β”œβ”€β”€ videoarm_audio.py
β”‚   β”œβ”€β”€ videoarm_download.py
β”‚   β”œβ”€β”€ videoarm_doctor.py  # Dependency checker
β”‚   └── videoarm_clean.py   # Cache cleaner
β”œβ”€β”€ videoarm_lib/          # Shared library
β”‚   β”œβ”€β”€ config.py          # Paths and database
β”‚   β”œβ”€β”€ frames.py          # Frame extraction logic
β”‚   β”œβ”€β”€ resolve.py         # Video path resolution
β”‚   β”œβ”€β”€ video_meta.py      # Metadata extraction
β”‚   └── logger.py          # Tool tracer
β”œβ”€β”€ videoarm_local_whisper/ # Local Whisper server
β”‚   β”œβ”€β”€ server.py
β”‚   └── setup.py
β”œβ”€β”€ examples/              # Usage examples
β”œβ”€β”€ .env.example           # Configuration template
β”œβ”€β”€ pyproject.toml         # Package config
└── LICENSE                # MIT

How the Agent Reasons

  1. Download & Inspect β€” Get video metadata (duration, fps, audio availability)
  2. Strategy β€” Choose audio-first (dialogue questions) or frames-first (visual questions)
  3. Extract & Analyze β€” Use tools iteratively, writing findings to memory
  4. Cross-verify β€” Confirm with a second modality if confidence is low
  5. Answer β€” Return answer with evidence chain and confidence score

Data Storage

All cached data stored under ~/.videoarm/:

  • video_database/temp/downloads/ β€” Downloaded videos
  • video_database/temp/processing/ β€” Temporary processing files
  • VideoARM β€” The research paper this skill is based on
  • OpenClaw β€” The agent platform this skill runs on

Troubleshooting

ProblemSolution
yt-dlp not foundpip install yt-dlp
ffmpeg not foundbrew install ffmpeg (macOS) or apt install ffmpeg (Linux)
opencv-python not foundpip install opencv-python
Download timeoutSet HTTPS_PROXY=http://... in .env
Poor transcription accuracyUse larger model: WHISPER_MODEL=small in .env
videoarm-doctor shows issuesFollow the suggested fix for each item

Run videoarm-doctor to diagnose most issues automatically.

License

MIT β€” see LICENSE