Media Metrologist

February 16, 2025 · View on GitHub

Measure video quality using a suite of metrics on a per-scene and per-frame basis. Compare multiple encoded videos against a reference in a single operation. Track comparison data live as it's computed and generate quality reports per video, scene, and metric.

Features

  • JSON Configuration and rich typings via Typescript
  • Standalone VapourSynth script: Metrologist.vpy
  • Support for either video file or VapourSynth script paths as inputs
  • Support for several VapourSynth video import methods
  • Compare multiple distorted videos against a reference video at once
  • Process multiple frame ranges defined as scenes with the potential to trade accuracy for more speed
  • Compute as many metrics as desired per scene
  • Split frames into a grid of regions for more spatial data
  • Realtime feedback on processing speed and computed scores
  • Progress is saved and can be resumed at any time
  • Simple statistics for each distorted video input and for each scene

Getting Started

Media Metrologist is designed to be used with the upcoming graphical user interface (GUI) so installation would not be strictly necessary. It is also in the process of being improved as it is currently unrefined. While some functionality such as statistics and report generation is done via NodeJS, most functionality is provided by the Metrologist.vpy VapourSynth script which can be executed by itself without NodeJS. For early users who want to test or implement Media Metrologist into their own NodeJS applications here are some loose instructions:

VapourSynth (Python)

  1. Clone or download metrologist.py
  2. Ensure Prerequisites are met
  3. Create a JSON file adhering to Configuration
  4. Execute metrologist.py with the JSON file
    • > python ./metrologist.py ./MyConfiguration.json
  5. Read the updated JSON file for results as configured

NodeJS

  1. Clone or download this repository
    • > git clone https://wwww.github.com/Av1ation-Association/Media-Metrologist.git --depth 1
  2. Install dependencies
    • > npm install
  3. Build
    • > npm run build
  4. Ensure Prerequisites are met
  5. Create a new typescript file ./src/main.local.ts
  6. Import Metrologist and instantiate with the desired configuration
  7. Execute the evalution with Metrologist.measure()
  8. Read results and generate statistics

Currently, there are no specific exports yet but the main.ts exports the class Metrologist which provides all the features. Until there is more documentation please peruse the typings provided by the IDE. Visual Studio Code (VS Code) is recommended and pre-configured for debugging. While using VSCode, you can test your own script by following the above instructions and debugging with the Main Local configuration. Below is an example main.local.ts which compares 2 videos imported with DGDecodeNV and/or BestSource using VapourSynth-HIP to evaluate the SSIMULACRA 2 metric:

Example 
import { Metrologist } from './main.js';
import { type Configuration } from './types/Configuration/Configuration.js';
import {
    ImportMethodType,
    type ImportMethod,
} from './types/Configuration/Import.js';
import { SSIMULACRA2Implementation } from './types/Configuration/Metric.js';
import { type ScoringStatus } from './types/Status.js';

const testConfig: Configuration = {
    $schema: ``,
    reference: {
        path: 'C:/My Reference Video.mkv',
        importMethods: new Set([
            {
                type: ImportMethodType.DGDecNV,
            },
            {
                type: ImportMethodType.BestSource,
            },
        ] as ImportMethod[]),
    },
    distorted: {
        '1': {
            path: 'C:/My Distorted Video.mkv',
            importMethods: new Set([
                {
                    type: ImportMethodType.DGDecNV,
                },
                {
                    type: ImportMethodType.BestSource,
                },
            ] as ImportMethod[]),
        },
    },
    metrics: {
        SSIMULACRA2: {
            implementation: SSIMULACRA2Implementation.CUDA,
        },
    },
    scenes: [
        ...Array.from({ length: 1 }, (_, index) => ({
            reference: {
                start: index * 100,
                end: (index + 1) * 100,
            },
            distorted: {
                '1': {
                    start: index * 100,
                    end: (index + 1) * 100,
                    scores: {
                        SSIMULACRA2: [],
                    },
                },
            },
        })),
    ],
    output: {
        console: true,
        verbose: true,
    },
    threads: 4,
};

const metrologist = new Metrologist(testConfig);
const finalConfig = await metrologist.measure();

metrologist.on('scoring', (event: ScoringStatus) => {
    if (event.frameIndex === 0) {
        console.log(`Scene ${event.sceneIndex} Distorted ${event.distortedId}: ${event.metric} = ${event.score}`);
    }
});

console.log(`TOTAL FRAMES: ${metrologist.totalFrames}`);
console.log(`${(await import('util')).inspect(metrologist.completedFramesScored)}`);
console.log(`${metrologist.framerate} FPS`);
console.log(`${(await import('util')).inspect(metrologist.statistics, undefined, 10, true)}`);

console.log('Final config:', (await import('util')).inspect(finalConfig, undefined, 10, true));

Prerequisites

Besides being a NodeJS application, Media Metrologist uses VapourSynth and several VapourSynth plugins in order to decode input videos and evaluate the desired metrics. At a minimum, metrologist.vpy and by extension Media Metrologist both require the following to be installed:

For more information on the plugins see Import Methods and Metrics.

Available Scripts

  • clean - remove coverage data, Jest cache and transpiled files,
  • prebuild - lint source files and tests before building,
  • build - transpile TypeScript to ES6,
  • build:watch - interactive watch mode to automatically transpile source files,
  • lint - lint source files and tests,
  • prettier - reformat files,
  • test - run tests,
  • test:watch - interactive watch mode to automatically re-run tests