Protocol audit

This document separates three things that were previously easy to conflate: the executed notebooks, the NILMBench2026 paper, and the protocol now used for new leaderboard runs.

Evidence recovered

The executed notebooks were recovered from the result_notebooks directory of an installed nilmtk_contrib environment on the Ramanujan server. They are not committed here because they contain bulky cell outputs and machine-specific paths. The key inputs can be independently identified by these SHA-256 hashes:

Notebook	SHA-256
redd-t1-1min-seed10(hyperparam).ipynb	248167b6c19f5668ebc90dc1fbe9caed2eb06ae8313e956c94cc6727a3b8bf31
ukdale-t1-1min-seed10(hyperparam).ipynb	0616835fbc9a5532b4fb2dce14b38145a3d6657e5362efc2c12a7af1af3f2b61
refit-t1-1min.ipynb	06199acc2a93031e997534b01c2421c48b3553c22ee100f9dfa438b69ed83824
redd-t2.ipynb	483c6a5cf8c4e07b1e77854e00a44ac4c4815e4e006695221d94c68c96e0955e
redd-t2-f1score.ipynb	4725094a469903cb8575720137c866c6b4660237374d8f796c3772b5aefca807
ukdale-t2.ipynb	68d75b071a27e68eeced367582ecdef44e821cd05d4ceed8132cc123571ae5b5
refit-t2-seed10.ipynb	f2f13e57ea2fc5d21037518ab219a872d7d792f1d8c1beea4d69a859cd5d9efc
redd-refit-t3.ipynb	68c1a29d7473954951ff29670deeda549d1a19564057dd279fbfe6eb016b8813
redd-refit-t3 to refit-redd.ipynb	92124fcbb3818f86514cd274de850945f817b5d69d038518d53450611de3e8f1

Recovered differences

• The REDD T1 notebook requested 1–30 April 2011, but the canonical converted file begins on 18 April. A legacy run therefore uses only part of its requested training window.
• The REFIT T1 notebook requests a microwave in building 1. That building has no microwave in the canonical conversion.
• The REDD T2 MAE notebook tests building 4, while the REDD T2 F1 notebook and Section 3.4 of the paper test building 6. Building 4 also lacks the requested fridge in the canonical conversion.
• Executed notebook F1 uses nilmtk.losses.f1score, which applies a fixed 10 W threshold. Section 3.5 of the paper specifies appliance thresholds instead.

The historical-* tasks preserve the recovered notebook inputs and legacy 10 W metric behavior for forensic reproduction. They may warn or fail where the hashed canonical files cannot satisfy those requests.

The corrected-* tasks are the basis for new claims. They use the paper's building splits, appliance-specific thresholds, strict time-envelope checks, and per-appliance alignment. In particular, corrected REDD T2 is B1/B2/B3 → B6 and uses fridge, washing machine, and dish washer.

NILM Metadata resolution

Task configurations use canonical NILM Metadata names. NILMTK resolves those names through its appliance taxonomy and synonyms; NILMbench does not maintain a competing alias table. Every observed window records:

• the exact resolved appliance types and instances;
• the exact meter or MeterGroup identifiers;
• unrelated appliances sharing a selected physical meter; and
• the active/apparent power type selected from the configured preference list.

This audit surfaced real data-quality constraints. Examples include:

• REDD washing machine resolving to washer dryer, including multi-channel MeterGroups in buildings 1 and 3;
• UK-DALE building 4 using apparent mains power while buildings 1 and 2 use active mains power;
• UK-DALE building 4 microwave and washing machine resolving to the same shared meter, which also contains a breadmaker; and
• REFIT building 1 fridge resolving to one fridge and two freezers.

Paper-compatible corrected tasks warn and record shared-meter contamination. A future clean-meter profile can set shared_meter_policy = "strict" and use only targets with isolated ground truth.

Real-data preflight

On 17 July 2026, all eight corrected tasks passed a read-only preflight against the three SHA-256-pinned HDF5 files on Ramanujan. The check loaded each train and test building, resolved every canonical appliance through NILM Metadata, selected an available AC power type, and found aligned samples. This is a data and protocol preflight; it is not a replacement for the CUDA model benchmark.

Release sequence

Model implementations and their model-level tests merge in nilmtk-contrib first. NILMbench then advances its single nilmtk-contrib dependency and container build context to the reviewed batch commit, refreshes uv.lock, and runs CPU and CUDA checks against that same revision. ModernTCN and DLinear were adopted together at 8d745493ed9f84dd00fb502ffe85943eaeedc4c8, avoiding a separate benchmark image for every algorithm while preserving an immutable campaign environment. TimesNet and SGN were similarly batched at c130293e24e16817b9859d1b78ae18bd988b1219 after their independent model and shared-runtime PRs passed. TSMixer then advanced the same single image family to 5767286078f5853a2ef1d6f431eb95a1c47ba4e8; the image still contains the complete model suite and is not a model-specific image. NILMMoE then advanced that same image family to 9d28870d0df378c34092d95ea8a11479e0fe7db3, and ResidualMoE advanced it to 0769d00893a178277d06e83ccd2e92c03a5ba9ef. The explicit-duration HSMM baseline advanced it to 1148e1c65f43878dfa1b8e08dc6411f5991d7dbd.