Implementation Guide

July 22, 2026 · View on GitHub

Implement the code for a new dataset that reformats source data into Zarr.

This guide is the Implement stage of the end-to-end dataset development guide; explore, backfill, validate, and publish are separate stages there. To add a single variable to an existing dataset instead, follow add_new_variable.md.

Overview

Integrating a dataset in dynamical.org reformatters is done by subclassing a trio of base classes, customizing their behavior based on the unique characteristics of your dataset.

There are three core base classes to subclass.

  1. TemplateConfig defines the dataset structure.
  2. RegionJob defines the process by which a region of that dataset is reformatted: downloading, reading, rewriting.
  3. DynamicalDataset brings together a TemplateConfig and RegionJob and defines the compute resources to operationally update and validate a dataset.

Words

  • Provider - the agency or organization that publishes the source data. e.g. ECMWF
  • Model - the model or system that produced the data. e.g. GFS
  • Variant - the specific subset and structure of data from the model. e.g. forecast, analysis, climatology. Variant may include any other information needed to distinguish datasets from the same model.
  • Dataset - a specific provider-model-variant. e.g. noaa-gfs-forecast

Two kinds of datasets

reformatters produces two kinds of datasets; you pick one at step 1 and it selects which RegionJob base class you subclass.

  • Materialized — download the source files, rechunk, and rewrite the bytes into a new Zarr/Icechunk store. Subclass MaterializedRegionJob.
  • Virtual — an Icechunk store whose chunks are references into the provider's source files, decoded at read time, with chunks following the source files' native layout. Subclass VirtualRegionJob; see docs/virtual_datasets.md.

Most source weather files are laid out for spatial access, so most virtual datasets are spatial-optimized; materialized (rechunked) datasets provide the complementary time-optimized chunking over the same source data.

Integration steps

Before getting started, follow the brief setup steps in README.md > Local development > Setup.

0. Explore the source dataset

Explore the source dataset to understand the nuances of what's available and how to access it. See docs/source_data_exploration_guide.md.

1. Initialize a new integration

uv run main initialize-new-integration <provider> <model> <variant> --kind <materialized|virtual>

Provider, model and variant can contain letters, numbers and dashes (e.g. ICON-EU or analysis-hourly). Capitalization will be normalized for you. --kind selects the teaching template that is scaffolded.

This adds a number of files within src/reformatters/<provider>/<model>/<variant> and tests/<provider>/<model>/<variant>.

These files contain placeholder implementations of the subclasses referenced above, commented with guidance specific to the kind you chose. Follow the rest of this doc to complete the implementations and integrate your new dataset.

The scaffolded docstrings and comments are deliberately verbose to guide implementation. As you implement, strip them down to the repo's conventions (see the Code Style notes in AGENTS.md) — minimal, one-line comments only where something is non-obvious. The scaffold also carries # noqa: F401 on imports it only uses in commented-out example code; remove those noqas (and any now-unused imports) once you've filled the code in.

If a dataset for the same provider/model already exists, reuse its shared config models and utilities (e.g. a <provider>/<model>_config_models.py defining the shared DataVar/attrs types) instead of the scaffolded copies.

2. Register your dataset

Add an instance of your DynamicalDataset subclass to the DYNAMICAL_DATASETS constant in src/reformatters/__main__.py:

from reformatters.provider.model.variant import ProviderModelVariantDataset

DYNAMICAL_DATASETS = [
    ...,
    ProviderModelVariantDataset(
        primary_storage_config=ProviderModelIcechunkAwsOpenDataDatasetStorageConfig(),
    ),
]

If you plan to write this dataset to a location not maintained by dynamical.org, you can instantiate and pass your own StorageConfig, contact feedback@dynamical.org for support.

3. Implement TemplateConfig subclass

Work through src/reformatters/$DATASET_PATH/template_config.py, setting the attributes and method definitions to describe the structure of your dataset. The report generated by following the source_data_exploration_guide.md will be helpful here.

Set dataset_id and name in dataset_attributes following the id/name convention: a materialized dataset uses dataset_id="<provider>-<model>-<variant>" and name="Provider Model variant"; a virtual dataset carries a -virtual id suffix and a , virtual name suffix — dataset_id="<provider>-<model>-<variant>-virtual" and name="Provider Model variant, virtual".

Read the chunk/shard layout tool docs and use the tool to find chunk and shard sizes for your data variables.

Follow the keep_mantissa_bits guidance in AGENTS.md when setting each data variable's encoding.

Using the information in the TemplateConfig, reformatters writes the Zarr metadata for your dataset to src/reformatters/$DATASET_PATH/templates/latest.zarr. Run this command in your terminal to create or update the template based on the your TemplateConfig subclass:

uv run main $DATASET_ID update-template
git add src/reformatters/$DATASET_PATH/templates/latest.zarr

Tracking the template in git lets us review diffs of any changes to the structure of our dataset.

Run the tests, making any changes necessary.

uv run pytest tests/$DATASET_PATH/template_config_test.py

If your dataset mixes single-level variables with variables on a dense vertical dimension, it becomes a Zarr group hierarchy; see "Vertical levels" in AGENTS.md.

4. Implement RegionJob subclass

Work through src/reformatters/$DATASET_PATH/region_job.py, implementing the methods the scaffolded code describes. Both kinds implement generate_source_file_coords (the source files a region needs) and operational_update_jobs (the update factory — skippable until you implement updates; a backfill runs without it). The rest differs by kind:

  • Materialized: download_file (fetch a source file to disk) and read_data (load one variable into a numpy array). The base runs the download → read → transform → write pipeline; optional transformation hooks are described in the example.
  • Virtual: discover_available (which pending files are fetchable now) and file_refs (the byte-range references one file contributes). The base owns the write loop and atomic commits.

Write tests for any custom logic you create.

uv run pytest tests/$DATASET_PATH/region_job_test.py

You've reached the point where you can run the reformatter locally!

uv run main $DATASET_ID backfill-local <append_dim_end> --filter-variable-names <data var name>

Reformatting locally can be slow. Choosing an <append_dim_end> not long after your template's append_dim_start and selecting a single variable to process with --filter-variable-names can limit the amount of work.

5. Implement DynamicalDataset subclass

To operationalize your dataset and have the update and validate Kubernetes cron jobs be deployed automatically by GitHub CI, implement the two methods in src/reformatters/$DATASET_PATH/dynamical_dataset.py.

The scaffold sets suspend=True on both cron jobs so updates and validation stay off until the store is backfilled. Leave it set through this PR; a follow-up PR removes it once the backfill is complete (the Backfill stage of the dataset development guide).

Kubernetes resource values, materialized (fan-out across indexed jobs):

  • shared memory: Round the value calculated in the chunk/shard size tool output up to the nearest half GB.
  • memory: 1.5x shared memory.
  • cpu: the number of spatial dimension shards minus 1 to account for kubernetes headroom. e.g. if 2 latitude shards * 4 longitude shards = 8, choose 7 cpu to schedule on an 8 cpu node.
  • ephemeral_storage: 20GB is a good starting point.
  • Parallelism: set workers_total and parallelism on the ReformatCronJob using self.num_variable_groups(). Multiply by 2 if operational_update_jobs reprocesses the most recent time slice (see GEFS datasets for examples).

For virtual updates, use workers_total = 1 and parallelism = 1 (they are single-writer); cpu="1.7" and memory="7G" are good starting points. See Operational updates, and Manifest splitting for sizing the manifest split the storage config needs.

The update cron schedule should run shortly after the source data is expected to be available and the validate cron should run at update cron start + update pod_active_deadline.

Integration test with snapshot values

In dynamical_dataset_test.py create a test that runs backfill_local followed by update for a couple data variables and a minimal number of time steps, lead times and ensemble members. Include snapshot value assertions for every data variable that the test processes — check specific known values at specific coordinates (e.g. assert_allclose(point["temperature_2m"].values, [28.75, 29.23])). Snapshot values catch silent regressions in data reading, unit conversion, or coordinate alignment that other tests miss.

Wrap the trimmed template in tests.chunk_utils.shrink_chunks_and_shards in your test's get_template monkeypatch (see existing dataset tests for examples), for materialized datasets only — a virtual dataset's one-message-per-chunk geometry must not be shrunk. At test scale, production chunk geometry means writing thousands of nearly empty chunks, which dominates test runtime without adding coverage. The helper reads each var's existing chunks/shards and the trimmed sizes and shrinks them automatically, keeping each dimension's production chunk and shard counts (capped at two) so multi-chunk and multi-shard dims stay multi and the corresponding write paths stay exercised. Call it after trimming (after any .sel(...)) so it sees the sizes the test actually writes; pass dims=[...] to shrink only some dimensions (e.g. leave the append dim at production size for a shard-boundary test).

uv run pytest tests/$DATASET_PATH/dynamical_dataset_test.py

Next

Once the code is merged to main, backfill the store (backfill.md), then validate and publish — the remaining stages of the dataset development guide.