1. Prerequisites

• Python 3.10 (check "Add Python to PATH" during installation)
• uv:

  powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
  

2. Install

# Run in the project directory
uv sync

3. Verify

uv run python -c "import cv2; import torch; print('OK')"

1. Install uv

curl -LsSf https://astral.sh/uv/install.sh | sh

2. Prepare pyproject.toml

macOS uses a dedicated dependency file (the default pyproject.toml contains dependencies that are not compatible with macOS).

cp pyproject.toml pyproject.win.toml
cp pyproject.macos.toml pyproject.toml

3. Base packages

uv venv .venv && uv sync
uv pip install pip setuptools wheel torch==2.0.1 torchvision==0.15.2

4. Build xtcocotools from source

mkdir -p deps && cd deps
git clone https://github.com/jin-s13/xtcocoapi.git
cd xtcocoapi && ../../.venv/bin/python -m pip install -e . && cd ../..

5. Build mmcv-full from source (~5 minutes)

cd deps
curl -L https://github.com/open-mmlab/mmcv/archive/refs/tags/v1.7.0.tar.gz -o mmcv-1.7.0.tar.gz
tar xzf mmcv-1.7.0.tar.gz && cd mmcv-1.7.0
MMCV_WITH_OPS=1 FORCE_CUDA=0 ../../.venv/bin/python setup.py develop
MMCV_WITH_OPS=1 FORCE_CUDA=0 ../../.venv/bin/python setup.py build_ext --inplace
cd ../..

6. Remaining packages

uv pip install --no-build-isolation anime-face-detector
uv pip install mmdet==2.28.0 mmpose==0.29.0

7. Launch

.venv/bin/python mouth_track_gui.py

Notes

• Do not delete the deps/ directory
• Use +/- keys for calibration zoom (scroll wheel may not work)

1. Prerequisites

• Ubuntu 22.04 LTS
• Python 3.10
• NVIDIA GPU / CUDA 11.7 compatible environment recommended

2. Install uv

curl -LsSf https://astral.sh/uv/install.sh | sh

3. System packages

sudo apt-get update
sudo apt-get install -y libportaudio2 pulseaudio-utils

4. Install

uv sync

5. Verify

uv run python -c "import cv2; import torch; print('OK')"

6. Launch

uv run python mouth_track_gui.py

Notes

• Linux audio input enumeration can differ from Windows/macOS.
• If your USB mic does not appear as a normal sd: device, try a pa:... (via pulse) item in the audio list.
• If pactl is unavailable, Linux audio fallback support is limited.

Video (.mp4)

• A short looping video (a few seconds)
• Face should not be obscured

Mouth Sprites (.png x 5)

• Format: PNG (with transparency)
• Recommended size: ~128px width
• Having just open.png can sometimes compensate for missing sprites, but having all 5 is more stable

Mouse

Keyboard

Presets

After completing (2) Generate mouthless video, the following files are exported to the same folder:

• mouth_track.json
• *_mouthless_h264.mp4 (if ffmpeg is found)

Use these when passing mouth track data to a browser implementation.

• mouth_track.json is the main track export
• *_mouthless_h264.mp4 is recommended for browser / player use when ffmpeg is available
• If ffmpeg is missing, the GUI still exports mouth_track.json and logs that H.264 export was skipped

For MotionPNGTuber_Player and other browser implementations that use AudioWorklet, opening files via file:// may block mic / worklet loading. In that case, serve the folder locally:

python -m http.server 8000
# then open http://localhost:8000

# Face tracking
uv run python auto_mouth_track_v2.py --video loop.mp4 --out mouth_track.npz

# Calibration
uv run python calibrate_mouth_track.py --video loop.mp4 --track mouth_track.npz --sprite open.png

# Generate mouthless video
uv run python auto_erase_mouth.py --video loop.mp4 --track mouth_track_calibrated.npz --out loop_mouthless.mp4

# Real-time execution
uv run python loop_lipsync_runtime_patched_emotion_auto.py \
  --loop-video loop_mouthless.mp4 \
  --mouth-dir mouth_dir/Char \
  --track mouth_track_calibrated.npz

MotionPNGTuber/
├── mouth_track_gui.py                              # Main GUI (entry point)
├── mouth_track_gui/                                # Main GUI package
│   ├── __init__.py
│   ├── __main__.py
│   ├── _paths.py                                   #   Path definitions
│   ├── app.py                                      #   Application core
│   ├── ui.py                                       #   UI construction
│   ├── state.py                                    #   Session management
│   ├── services.py                                 #   File/device helpers
│   ├── actions.py                                  #   Command assembly
│   ├── runner.py                                   #   Subprocess execution
│   ├── preview.py                                  #   Lightweight preview
│   └── live_ipc.py                                 #   Live runtime IPC
├── mouth_sprite_extractor_gui.py                   # Mouth PNG creation GUI
├── mouth_sprite_extractor.py                       # CLI wrapper for mouth sprite extraction
├── auto_mouth_track_v2.py                          # Mouth position tracking
├── calibrate_mouth_track.py                        # Calibration
├── auto_erase_mouth.py                             # Mouthless video generation
├── erase_mouth_offline.py                          # Mouth erase core processing
├── loop_lipsync_runtime_patched_emotion_auto.py    # Live runtime
├── face_track_anime_detector.py                    # Anime face detection
├── motionpngtuber/                                 # Shared library package
│   ├── __init__.py
│   ├── mouth_sprite_extractor.py                   #   Core mouth sprite extraction logic
│   ├── python_exec.py                              #   Python executable resolver
│   ├── audio_linux.py                              #   Linux audio helper (PulseAudio/PipeWire)
│   ├── mouth_color_adjust.py                       #   Mouth PNG color correction logic
│   ├── lipsync_core.py                             #   Shared module
│   ├── image_io.py                                 #   Image I/O (Unicode path support)
│   ├── platform_open.py                            #   Platform-specific open handling
│   ├── auto_crop_estimator.py                      #   Auto crop estimation
│   ├── mouth_auto_classifier.py                    #   Mouth shape auto classification
│   ├── mouth_feature_analyzer.py                   #   Mouth feature analysis
│   ├── realtime_emotion_audio.py                   #   Real-time emotion audio analysis
│   └── workflow_validation.py                      #   Workflow validation
├── convert_npz_to_json.py                          # npz → JSON conversion
├── tests/                                          # Test suite
├── assets/                                         # Sample assets
├── mouth_dir/                                      # Mouth sprites (sample)
├── pyproject.toml                                  # Dependencies (Windows / Linux)
└── pyproject.macos.toml                            # Dependencies (macOS)

Mouth position is misaligned

• First try Recalibrate only
• If still too small / too large, use Visual check (lightweight) to compare margin factor values
• Then adjust the margin factor in advanced settings slightly if needed

Black smudge in mouth erase

• Try turning Shadow blending OFF

uv sync fails

uv cache clean
uv sync

CUDA not recognized

uv run python -c "import torch; print(torch.cuda.is_available())"

Analysis stops on RTX 50-series / newer GPUs

• Root cause: the current Torch / CUDA build may not fully match your GPU architecture
• Prevention: the GUI now starts analysis with --device auto, which tries CUDA first and then CPU fallback
• If analysis still fails, rerun from CLI with --device cpu and check the log for CUDA compatibility messages

Analysis looks frozen

• Initial analysis can take time, especially on long videos
• If GPU fallback switches to CPU, processing becomes much slower but is still expected
• Check the progress area and log before assuming the app has hung

Analysis error appears immediately after selecting a video in the mouth PNG GUI

• Root cause (fixed on 2026-04-13): the GUI could fail before tracking started if the bundled detector script path was resolved incorrectly
• Prevention: the detector launcher now searches both the package directory and the repository root, and this lookup is covered by a regression test
• If you still see the error after updating, check the log pane for the exact exception and verify that face_track_anime_detector.py exists in the project

Mouth Sprite Extraction CLI

uv run python mouth_sprite_extractor.py --video loop.mp4 --out mouth/

Extract mouth sprites (5 PNGs) directly from the command line without the GUI.