Speech-to-Text (STT) Module - Comprehensive Testing Documentation

November 2, 2025 ยท View on GitHub

Overview

The STT module has been enhanced with comprehensive unit, integration, and performance tests covering all transcription implementations including MLX, ONNX, buffered transcription, WebSocket streaming, and external provider support.

Test Coverage Summary

1. Unit Tests

test_parakeet_mlx.py

  • Coverage: Parakeet MLX implementation
  • Key Tests:
    • Model loading and caching
    • Simple transcription
    • Chunked transcription
    • Audio preprocessing (resampling, stereo to mono)
    • Error handling
    • Progress callbacks
    • File path and numpy array inputs

test_parakeet_onnx.py

  • Coverage: Parakeet ONNX implementation
  • Key Tests:
    • ONNX session creation
    • Tokenizer functionality
    • Mel-spectrogram preprocessing
    • Chunked transcription with merge algorithms
    • Device selection (CPU/CUDA)
    • Error handling

test_buffered_transcription.py

  • Coverage: Advanced buffered/chunked transcription
  • Key Tests:
    • Chunk creation logic
    • Middle merge algorithm
    • LCS (Longest Common Subsequence) merge
    • Audio resampling
    • Progress callbacks
    • Memory efficiency

test_streaming_transcription.py

  • Coverage: WebSocket-based streaming
  • Key Tests:
    • AudioBuffer functionality
    • Voice activity detection
    • Buffer accumulation
    • WebSocket connection handling
    • Error handling during streaming
    • Concurrent stream support

test_external_provider.py

  • Coverage: External OpenAI-compatible API support
  • Key Tests:
    • Configuration validation
    • Provider management (add/list/remove)
    • Retry logic on rate limiting
    • Timeout handling
    • Different response formats (json, text, srt, vtt)
    • Authentication handling

2. Integration Tests

test_parakeet_mlx.py - Integration Section

  • Integration with Nemo module
  • Integration with main transcription library
  • Cross-module functionality

test_buffered_transcription.py - Integration Section

  • Integration with MLX backend
  • Integration with ONNX backend
  • Different merge algorithm comparisons

test_streaming_transcription.py - Integration Section

  • Full streaming session simulation
  • Concurrent streams handling
  • WebSocket protocol compliance

3. Performance Benchmarks

test_transcription_benchmarks.py

  • Coverage: All implementations performance testing
  • Key Benchmarks:
    • MLX scaling with audio duration
    • Chunking vs non-chunking performance
    • ONNX inference speed
    • Merge algorithm efficiency
    • Memory usage profiling
    • Real-world scenarios (podcast, meeting, batch processing)

Running the Tests

Run All Tests

cd tldw_server
python -m pytest tldw_Server_API/tests/Media_Ingestion_Modification/ -v

Run Specific Test Categories

Unit Tests Only

python -m pytest -m "unit" tldw_Server_API/tests/Media_Ingestion_Modification/ -v

Integration Tests Only

python -m pytest -m "integration" tldw_Server_API/tests/Media_Ingestion_Modification/ -v

Performance Tests Only

python -m pytest -m "performance" tldw_Server_API/tests/Media_Ingestion_Modification/ -v

Run Tests for Specific Implementation

MLX Tests

python -m pytest tldw_Server_API/tests/Media_Ingestion_Modification/test_parakeet_mlx.py -v

ONNX Tests

python -m pytest tldw_Server_API/tests/Media_Ingestion_Modification/test_parakeet_onnx.py -v

Buffered Transcription Tests

python -m pytest tldw_Server_API/tests/Media_Ingestion_Modification/test_buffered_transcription.py -v

Streaming Tests

python -m pytest tldw_Server_API/tests/Media_Ingestion_Modification/test_streaming_transcription.py -v

External Provider Tests

python -m pytest tldw_Server_API/tests/Media_Ingestion_Modification/test_external_provider.py -v

Run with Coverage Report

python -m pytest --cov=tldw_Server_API.app.core.Ingestion_Media_Processing.Audio \
    --cov-report=html \
    --cov-report=term \
    tldw_Server_API/tests/Media_Ingestion_Modification/

Test Markers

Tests are organized with markers for easy filtering:

  • @pytest.mark.unit - Unit tests (fast, isolated)
  • @pytest.mark.integration - Integration tests (cross-module)
  • @pytest.mark.performance - Performance benchmarks
  • @pytest.mark.external_api - Tests requiring external APIs
  • @pytest.mark.slow - Slow tests (>5 seconds)
  • @pytest.mark.asyncio - Async tests

Mock Strategy

All tests use comprehensive mocking to ensure:

  • Fast execution
  • No external dependencies
  • Predictable results
  • Resource isolation

Key mocked components:

  • ML models (MLX, ONNX)
  • HTTP clients for external APIs
  • WebSocket connections
  • File I/O operations
  • Audio processing libraries

Performance Baseline

Expected performance metrics from benchmarks:

ImplementationSpeed (vs real-time)Memory UsageStatus
MLX (chunked)100x+<2GBProduction Ready
MLX (no chunks)60x+~2GBProduction Ready
ONNX20-30x~1.5GBRequires Setup
Buffered (LCS)70x~2GBProduction Ready
External APIVariesMinimalProvider Dependent

External Provider Configuration

Setting Up External Providers

External providers allow forwarding transcription requests to any OpenAI-compatible Audio API.

Via Configuration File

Edit config.txt:

[external_providers.myapi]
base_url = https://api.example.com/v1/audio/transcriptions
api_key = your-api-key
model = whisper-1
timeout = 300
max_retries = 3

Via Environment Variables

export EXTERNAL_TRANSCRIPTION_MYAPI_BASE_URL=https://api.example.com
export EXTERNAL_TRANSCRIPTION_MYAPI_API_KEY=your-api-key
export EXTERNAL_TRANSCRIPTION_MYAPI_MODEL=whisper-1

Programmatically

from Audio_Transcription_External_Provider import (
    ExternalProviderConfig,
    add_external_provider
)

config = ExternalProviderConfig(
    base_url="https://api.example.com/v1/audio/transcriptions",
    api_key="your-api-key",
    model="whisper-1"
)

add_external_provider("myapi", config)

Using External Providers

# Use default external provider
result = transcribe_audio(
    audio_data,
    transcription_provider="external"
)

# Use specific external provider
result = transcribe_audio(
    audio_data,
    transcription_provider="external:myapi"
)

Test Data Requirements

Audio Generation

Tests generate synthetic audio data:

  • Sample rate: 16kHz (standard for speech)
  • Duration: 1-300 seconds based on test
  • Format: Float32 numpy arrays
  • Content: Sine waves with speech-like modulation

Real Audio Testing

For production validation, use actual audio files:

python test_all_transcriptions.py

This uses the sample.mp4 file extracted to WAV format.

Continuous Integration

GitHub Actions Configuration

name: STT Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: '3.10'
      - name: Install dependencies
        run: |
          pip install -e .[dev]
      - name: Run unit tests
        run: |
          pytest -m "unit" tests/Media_Ingestion_Modification/
      - name: Run integration tests
        run: |
          pytest -m "integration" tests/Media_Ingestion_Modification/

Troubleshooting

Common Test Failures

  1. Import Errors

    • Ensure all dependencies are installed
    • Check PYTHONPATH includes project root

  2. Async Test Failures

    • Requires pytest-asyncio
    • May need event loop configuration

  3. Performance Test Variations

    • Results vary by hardware
    • Use relative comparisons, not absolute times

  4. Mock Failures

    • Verify mock paths match actual module structure
    • Check mock return values match expected types

Future Testing Improvements

  1. Property-based Testing

    • Use hypothesis for fuzzing
    • Test edge cases automatically

  2. Load Testing

    • Concurrent transcription stress tests
    • Memory leak detection

  3. End-to-End Tests

    • Full API endpoint testing
    • Real audio file processing

  4. Mutation Testing

    • Verify test effectiveness
    • Identify untested code paths

Conclusion

The STT module now has comprehensive test coverage ensuring:

  • Reliability across all implementations
  • Performance validation
  • Easy regression detection
  • Safe refactoring capability
  • External provider extensibility

All critical functionality is tested, mocked appropriately, and benchmarked for performance.