Multimodal AI Taxonomy

October 22, 2025 · View on GitHub

A comprehensive, open-source taxonomy mapping the complex landscape of multimodal AI model capabilities. Navigate and filter AI models by understanding which input modalities can produce which output modalities.

The Problem

Current AI platforms (Replicate, FAL AI, etc.) lack adequate filtering for multimodal characteristics:

Example: FAL AI lists 20+ models in their image-to-video category
These models differ significantly in modality support beyond just resolution and duration
Key distinctions often hidden: Some generate video without audio, others with audio, some with lip sync
Audio generation methods vary: Text-to-audio prompting, reference audio input, lip sync capabilities

Real-world scenario: You want to generate a video of a crowded Jerusalem marketplace with ambient background audio (vendors calling prices, conversation noise). Current platforms don't make it easy to filter for this specific capability.

The Solution

A folder-based JSON taxonomy that:

Organizes modalities by output type (video, audio, image, text, 3D)
Separates creation vs editing operations
Provides a common schema for consistency
Enables filtering for specific modality combinations
Includes extensible metadata (maturity level, platforms, example models, use cases)

Project Structure

taxonomy/
├── schema.json                          # Common schema for all modality files
├── README.md                            # Complete guide to the structure
├── video-generation/
│   ├── creation/modalities.json        # Creating video from scratch
│   └── editing/modalities.json         # Transforming existing video
├── audio-generation/
│   ├── creation/modalities.json        # Creating audio from scratch
│   └── editing/modalities.json         # Transforming existing audio
├── image-generation/
│   ├── creation/modalities.json        # Creating images from scratch
│   └── editing/modalities.json         # Transforming existing images
├── text-generation/
│   ├── creation/modalities.json        # Creating text from scratch
│   └── editing/modalities.json         # Transforming existing text
└── 3d-generation/
    ├── creation/modalities.json        # Creating 3D models
    └── editing/modalities.json         # Transforming 3D models

Quick Start

Query the Taxonomy

# Run example queries with filtering
python3 query_taxonomy.py

This demonstrates:

Filtering by output modality (video, audio, image, etc.)
Filtering by operation type (creation vs editing)
Filtering by characteristics (lip sync, audio type, maturity level)
Combining multiple filters
Statistics and analytics

Validate Use Cases

# Validate that the taxonomy supports the core use cases
node validate_use_cases.js

Key Features

Folder-Based Organization

Organized by OUTPUT modality (what is being generated):

Image-to-video → video-generation/
Text-to-audio → audio-generation/
Image-to-image → image-generation/editing/

Creation vs Editing:

Creation: Generating new content from scratch
Editing: Transforming existing content

Common Schema

All modality files follow taxonomy/schema.json:

{
  "fileType": "multimodal-ai-taxonomy",
  "outputModality": "video|audio|image|text|3d-model",
  "operationType": "creation|editing",
  "description": "Human-readable description",
  "modalities": [
    {
      "id": "unique-kebab-case-id",
      "name": "Human Readable Name",
      "input": {
        "primary": "main-input-type",
        "secondary": ["additional-inputs"]
      },
      "output": {
        "primary": "output-type",
        "audio": true|false,
        "audioType": "speech|music|ambient|etc"
      },
      "characteristics": {
        // Modality-specific features
      },
      "metadata": {
        "maturityLevel": "mature|emerging|experimental",
        "commonUseCases": [...],
        "platforms": [...],
        "exampleModels": [...]
      }
    }
  ]
}

Fine-Grained Distinctions

The taxonomy captures subtle but important differences:

Audio vs. no audio: Critical for many use cases
Audio type: Speech, music, ambient, synchronized
Generation method: Synthesized, reference-based, text-to-speech
Special capabilities: Lip sync, audio-reactive motion, inpainting

Usage Examples

Python

from pathlib import Path
import json

# Load all modalities
all_modalities = []
for modality_folder in Path('taxonomy').iterdir():
    if not modality_folder.is_dir():
        continue
    for operation_folder in modality_folder.iterdir():
        if not operation_folder.is_dir():
            continue
        modalities_file = operation_folder / 'modalities.json'
        if modalities_file.exists():
            with open(modalities_file, 'r') as f:
                data = json.load(f)
                all_modalities.extend(data.get('modalities', []))

# Find video generation with audio
video_with_audio = [
    m for m in all_modalities
    if m['output']['primary'] == 'video'
    and m['output'].get('audio') == True
]

# Find lip sync capabilities
lip_sync = [
    m for m in all_modalities
    if m['characteristics'].get('lipSync') == True
]

Current Statistics

22 modality definitions
5 output modality categories (video, audio, image, text, 3D)
2 operation types (creation, editing)
3 maturity levels (experimental, emerging, mature)

By Output Modality

video-generation: 13 modalities
audio-generation: 5 modalities
image-generation: 2 modalities
3d-generation: 2 modalities

By Operation Type

creation: 17 modalities
editing: 5 modalities

By Maturity Level

mature: 8 modalities
emerging: 9 modalities
experimental: 5 modalities

Documentation

taxonomy/README.md: Complete guide to the folder-based structure
taxonomy/schema.json: Schema definition for all modality files
MIGRATION.md: Migration guide from single-file to folder-based structure
CLAUDE.md: Instructions for AI agents working on this project
query_taxonomy.py: Example Python script demonstrating queries

Adding New Modalities

Determine output modality: video, audio, image, text, or 3d-model
Determine operation type: creation (generating new) or editing (transforming existing)
Navigate to folder: taxonomy/{output-modality}/{operation-type}/modalities.json
Follow the schema: Add your modality object following taxonomy/schema.json

Example:

{
  "id": "text-to-vid-avatar-lipsync",
  "name": "Text to Video (Avatar with Lip Sync)",
  "input": {
    "primary": "text",
    "secondary": []
  },
  "output": {
    "primary": "video",
    "audio": true,
    "audioType": "speech"
  },
  "characteristics": {
    "processType": "synthesis",
    "audioGeneration": "text-to-speech",
    "lipSync": true,
    "lipSyncMethod": "generated-from-text",
    "motionType": "facial"
  },
  "metadata": {
    "maturityLevel": "emerging",
    "commonUseCases": [
      "AI presenter videos",
      "Automated content creation",
      "Virtual spokespersons"
    ],
    "platforms": ["Synthesia", "HeyGen", "D-ID"],
    "exampleModels": []
  }
}

Contributing

We welcome contributions to:

Add new modality definitions
Update metadata (platforms, models, maturity levels)
Improve documentation
Add validation tests
Expand examples

See taxonomy/README.md for detailed guidelines.

Project Vision

This taxonomy aims to:

Serve as a personal reference for navigating the multimodal AI landscape
Function as an open-source resource for the community
Enable better platform filtering for model discovery
Track the evolution of multimodal AI capabilities
Support informed decision-making when selecting models for specific use cases

More granular audio distinctions (genres, emotions, accents)
Video characteristic expansion (camera motion, scene complexity, duration ranges)
Multi-step workflow recommendations
Quality and performance benchmarking
Community-driven model discovery integration
Web-based taxonomy browser