Plot Generation

Overall Results

The plot module handles the plot generation from the data generated by the evaluation module.

You can use the multirag-cli plot command to generate plots to visualize the evaluation data. The command comes with the following command line interface:

usage: multirag-cli plot [-h] [-d [DATA_PATH]] [-f [{pdf,png,svg}]] [-o [OUTPUT]]

Plotting

optional arguments:
  -h, --help            show this help message and exit
  -d [DATA_PATH], --data-path [DATA_PATH]
                        Path to the evaluation data file. (default: test-results.json)
  -f [{pdf,png,svg}], --format [{pdf,png,svg}]
                        Format for the plot files. (default: pdf)
  -o [OUTPUT], --output [OUTPUT]
                        Path to the output directory. (default: plots)

Input Format

The data format for the JSON files with results is as follows:

{
    "standard-rag": {
        "1": {
            "success": [[...25 queries (int success)...], [...25 queries...], ...1-32 doc fetches],
            "success_ratio": [[...25 queries (float ratio)...], [...25 queries...], ...1-32 doc fetches],
            "category_success": [[...25 queries (int success)...], [...25 queries...], ...1-32 doc fetches]
            "category_success_ratio": [[...25 queries (float ratio)...], [...25 queries...], ...1-32 doc fetches]
        },
        "2": {...}
    },
    "multirag": {...},
    "split-rag": {...},
    "fusion-rag": {...},
    "fusion-multirag": {...},
    "multirag-strategy-decay": {...},
    "multirag-strategy-distance": {...},
    "split-rag-strategy-weighted": {...},
}

Per-Head Analysis

In addition to plotting various aspects of the overall results, we also provide visualizations for the performance of individual heads.

usage: plot_per_head.py [-h] --per_head PER_HEAD --regular REGULAR [--categories CATEGORIES] [--category_dist CATEGORY_DIST] [--strategy STRATEGY] [--output_dir OUTPUT_DIR]
                        [--metric {success_ratio,category_success_ratio,success,category_success}]

Plot per-head coverage metrics for MultiRAG

options:
  -h, --help            show this help message and exit
  --per_head PER_HEAD   Path to per-head results JSON file
  --regular REGULAR     Path to regular results JSON file
  --categories CATEGORIES
                        Path to per-head category coverage JSON file (optional)
  --category_dist CATEGORY_DIST
                        Path to per-head category distribution JSON file (optional, for MI heatmap)
  --strategy STRATEGY   Strategy name to plot (default: multirag)
  --output_dir OUTPUT_DIR
                        Output directory for plots (default: per_head_plots)
  --metric {success_ratio,category_success_ratio,success,category_success}
                        Metric to plot (default: success_ratio)