KnowVal Knowledge-Aware Value Model

May 21, 2026 ยท View on GitHub

This module contains a knowledge-aware Value Model for scoring driving scenes against rule embeddings. It is implemented as an MMDetection3D plugin so that existing OpenMMLab training and evaluation tools can load it through custom_imports.

It corresponds to the Value Model component in KnowVal: A Knowledge-Augmented and Value-Guided Autonomous Driving System.

Components

  • knowval_value_model/projects/mmdet3d_plugin/models/knowledge_aware_value_model.py
    • Registers KnowledgeAwareValueModel.
  • knowval_value_model/projects/mmdet3d_plugin/datasets/knowledge_aware_value_dataset.py
    • Registers KnowledgeAwareValueDataset.
  • knowval_value_model/configs/knowledge_aware_value_model.py
    • Training/evaluation config for the released model.
  • knowval_value_model/configs/_base_/default_runtime.py
    • Minimal OpenMMLab runtime defaults used by the config.
  • scripts/test_knowledge_aware_value_model.py
    • Lightweight tensor-shape tests.
  • knowval_value_model/weights/MANIFEST.md
    • Weight download instructions and checksum.
  • docs/RETRIEVAL.md
    • Knowledge retrieval module that produces retrieved_knowledge_entries as K_1..K_NK and a rule_embed handoff for the Value Model interface.

Model Design

The implementation maps to the value-assessment path in Section 3.3.1 of the paper:

  • Retrieved knowledge entry feature fKj: rule_embed.
  • Candidate trajectory and ego context: ego_trajs and the ego token from instance_query.
  • Future-state tokens Si: scene_feats together with instance_query.
  • Transformer Encoder and MLP Decoder: KnowledgeAwareValueModel.
  • Per-rule scalar values s_i,j: pred_scores returned by forward_test.
  • Weighted-decay aggregation in Equation (1): KnowledgeAwareValueModel.aggregate_rule_scores.

The paper uses NK = 16 retrieved knowledge entries, L = 3 Transformer iterations, and decay factor gamma = 0.7; these are the defaults in knowval_value_model/configs/knowledge_aware_value_model.py.

This release corresponds to the Value Model setting that uses scene features and per-knowledge-entry subscores in the paper's architecture comparison. The paper reports training the Value Model on 160K trajectory-knowledge pairs; the training data are not included in this Git repository.

The module scores one candidate trajectory per sample. The paper-level setting with multiple candidate trajectories (NT) is handled by the caller/planner, which should run this model over candidates and select or rank trajectories using the aggregated score.

The checkpoint preserves the training implementation: the decoder is trained against bounded value labels, but the final regression head is not hard-clamped with tanh or clip in code. Downstream consumers that require a strict numeric range should clamp the returned values after inference.

This document covers the learned Value Model and its dataset wrapper. The repository also includes a knowledge retrieval module in docs/RETRIEVAL.md; it does not include candidate trajectory generation, world prediction, the full traffic-law knowledge graph, or final planner selection.

Framework Scope

This release targets the MMDetection/MMCV stack used by the training pipeline:

  • mmdet.models.builder.DETECTORS
  • mmdet.datasets.CustomDataset
  • mmcv.cnn.bricks.transformer.build_transformer_layer_sequence

It is not currently a pure PyTorch package and should not be described as MMEngine/MMDetection3D 1.x native without a compatibility pass.

Expected Data Schema

Data are not released in this repository. The default config expects local features under data/knowval/:

data/knowval/
  annotations/
    train.json
    val.json
    test.json
  rule_embeddings/
    data_xxx.pkl
  ego_trajectories/
    data_xxx.pkl
  <scene feature pkl files referenced by pkl_path>

Each annotation item should contain pkl_path.

The scene feature pkl should contain:

  • scene_feats
  • instance_query
  • scores
  • optionally ego_trajs or traj_res

Rule embedding pkl files are loaded from rule_embed_dir with the same filename as pkl_path and should contain:

  • rule_embed

If the scene feature pkl does not contain ego_trajs or traj_res, the dataset falls back to ego_traj_root/<filename>.

Configuration

The config imports the plugin modules explicitly:

custom_imports = dict(
    imports=[
        'knowval_value_model.projects.mmdet3d_plugin.models.knowledge_aware_value_model',
        'knowval_value_model.projects.mmdet3d_plugin.datasets.knowledge_aware_value_dataset',
    ],
    allow_failed_imports=False)

The registered component names are:

  • KnowledgeAwareValueModel
  • KnowledgeAwareValueDataset

Weights

Weights are distributed outside Git through Google Drive:

https://drive.google.com/file/d/1lnuQLUONzRU5d8AGz5w9WxcL4Qrd9Q_F/view?usp=share_link

After downloading, place the state-dict-only checkpoint at:

knowval_value_model/weights/knowledge_aware_value_model.pth

The released checkpoint is exported from a verified training checkpoint for this model architecture with training metadata and optimizer state removed. Verify the file size and SHA256 against knowval_value_model/weights/MANIFEST.md.

Tests

From the repository root, in an environment with MMDetection/MMCV/MMDetection3D dependencies installed:

python -m pytest -q \
  scripts/test_knowledge_aware_value_model.py

This only validates module construction and tensor-shape behavior. Full evaluation requires the feature files described above.

License

This repository is released under the Apache License 2.0. See LICENSE for details. In addition, the project is free only for academic research purposes; it requires authorization for commercial use. For collaboration or commerce permission, please contact wyt@pku.edu.cn.