KnowVal Knowledge Retrieval

This module contains the public knowledge retrieval code for KnowVal. It maps scene perception facts into traffic-law retrieval queries, ranks retrieved law evidence, and formats the result as the fixed-size knowledge list consumed by the Value Model.

It corresponds to the knowledge retrieval stage that precedes the value assessment module in KnowVal: A Knowledge-Augmented and Value-Guided Autonomous Driving System.

Retrieval Method Figure

The paper constructs a knowledge graph by collecting driving-related resources, including laws, regulations, defensive driving principles, moral guidelines, and experiential knowledge. These texts are organized into an initial knowledge forest; LLM-based entity extraction then defines graph vertices and edges. During inference, KnowVal generates queries enriched with 3D perception information, retrieves and ranks relevant graph entries by relevance, filters back to native clauses, and hands the retrieved entries to the Value Model as K_1..K_NK. This open release keeps the same retrieval interface, while the bundled sample knowledge base contains only a compact traffic-law subset.

Retrieval Pipeline

The knowledge retrieval module follows the retrieval data flow described in the paper:

scene perception
  -> perception verbalizer
  -> two-layer keyword extraction
  -> KG entity retrieval
  -> top-K graph expansion
  -> native clause filtering
  -> retrieved_knowledge_entries K_1..K_NK
  -> rule_embed for KnowledgeAwareValueModel

The released Value Model uses NK = 16 retrieved knowledge entries. The helper format_knowledge_entries pads or truncates retrieval output to this fixed interface. The text entries are the public handoff format; downstream feature extraction can embed them into the rule_embed tensor expected by KnowledgeAwareValueModel.

The public class KnowValRetrievalPipeline exposes the same stage boundaries as the paper. For paper-aligned retrieval, attach the vLLM-backed Qwen2.5-3B keyword extractor. Its default offline_demo mode is deterministic and does not call any LLM, remote API, network service, or LightRAG experiment cache; this local mode is included for tests and interface examples, not as a replacement for the paper's LLM-based retrieval stage. The full experiment's Qwen checkpoint, embedding model, and LightRAG cache are not included in this repository.

Environment

The default retrieval demo has minimal dependencies:

• Python 3.8 or newer is recommended.
• No GPU, LLM API, network service, or LightRAG server is required for the offline_demo local mode.
• pytest is only needed for running the tests.
• Paper-aligned retrieval requires --keyword-backend vllm, vllm, a CUDA-capable PyTorch environment, and a local Qwen2.5-3B model path.

Use offline_demo only when checking the public interface without the Qwen runtime assets.

Components

• knowval_retrieval/core.py
  • Loads a compact traffic-law knowledge base.
  • Supports direct text retrieval.
  • Maps perception JSON into perception-guided retrieval from the knowledge graph.
  • Formats ranked law clauses as K_j entries.
• knowval_retrieval/cli.py
  • Command-line JSON interface for direct query or perception-file retrieval.
• knowval_retrieval/vllm_backend.py
  • vLLM-backed Qwen keyword extractor for the paper's two-layer keyword extraction stage.
• knowval_retrieval/examples/traffic_law_sample_kg.json
  • Small offline sample knowledge base for tests and interface examples.
• knowval_retrieval/examples/perception_case_crosswalk.json
  • Example scene perception input.
• scripts/test_knowval_retrieval.py
  • Offline tests for retrieval ranking, method metadata, and CLI output.

Knowledge Base Scope

The original experiments used a LightRAG-style traffic-law graph cache. This repository does not vendor the full upstream LightRAG project, WebUI, services, or large cache files. It also does not publish the full experimental traffic-law knowledge base.

The figure above follows the paper's supplementary caption, "Local view of the knowledge graph." The complete graph used in the paper contains 1,324 nodes and 2,785 edges. The open-source sample below is intentionally smaller and is included to demonstrate the retrieval interface without requiring the complete research knowledge base.

For reproducibility, Git contains only a partial English sample knowledge base:

• knowval_retrieval/examples/traffic_law_sample_kg.json

The sample currently contains 38 selected traffic-rule entries. They are English paraphrases of public road-passage rules, mainly from Articles 21-68 of the Road Traffic Safety Law of the People's Republic of China. They are included to exercise the retrieval interface and should not be treated as a complete or authoritative legal corpus.

The JSON file also records its release scope, language, included focus areas, excluded experiment assets, and quality notes so downstream users can see that it is a curated sample rather than the full experimental KG.

As a graph example, each law entry is treated as a traffic-rule clause node. The public retrieval module forms lightweight graph-expansion edges when two nodes share scene_tags or risk_tags. The current sample has 38 law nodes and 92 tag-overlap edges under this schema, which is enough to exercise seed retrieval and tag-based top-K graph expansion without publishing the full experimental KG. All 38 public rule-clause nodes are explicitly marked as native clauses, so the native-clause filtering stage can be checked directly in the public sample.

Reference pages for the source law:

• https://english.court.gov.cn/2015-08/17/c_761518_5.htm
• https://english.court.gov.cn/2015-08/17/c_761518_6.htm

The retrieval module also supports a local LightRAG-style enhanced cache directory with enhanced_custom_kg.json and enhanced_metadata.json for users who have their own separately maintained or externally distributed cache. This keeps the KnowVal release focused on the retrieval interface needed by the paper while avoiding unrelated LightRAG runtime, deployment code, and unreleased KG data.

Usage

From the repository root, use an environment with Python and the dependencies needed by the retrieval module. The default offline retrieval path uses only the Python standard library; pytest is needed only for tests.

git clone https://github.com/VDIGPKU/KnowVal.git
cd KnowVal
conda create -n knowval-public python=3.8 pytest
conda activate knowval-public

Direct traffic-law retrieval:

python -m knowval_retrieval.cli \
  --knowledge-base knowval_retrieval/examples/traffic_law_sample_kg.json \
  --query "What should a vehicle do when pedestrians are crossing a crosswalk?" \
  --top-k 3

Perception-guided knowledge retrieval:

python -m knowval_retrieval.cli \
  --knowledge-base knowval_retrieval/examples/traffic_law_sample_kg.json \
  --perception-file knowval_retrieval/examples/perception_case_crosswalk.json \
  --top-k 4

Paper-aligned perception-guided retrieval with vLLM/Qwen keyword extraction:

CC=/usr/bin/gcc CXX=/usr/bin/g++ CUDAHOSTCXX=/usr/bin/g++ \
python -m knowval_retrieval.cli \
  --knowledge-base knowval_retrieval/examples/traffic_law_sample_kg.json \
  --perception-file knowval_retrieval/examples/perception_case_crosswalk.json \
  --top-k 4 \
  --keyword-backend vllm \
  --vllm-model /path/to/Qwen2.5-3B

The compiler environment variables are useful on some conda installations where Triton otherwise resolves a stale compiler path. --vllm-enforce-eager can be used to avoid vLLM CUDA graph capture and torch compile during quick validation runs, at the cost of lower throughput.

Qwen/vLLM retrieval validation:

python scripts/evaluate_public_release.py \
  --qwen-model /path/to/Qwen2.5-3B \
  --qwen-dtype float16 \
  --qwen-gpu-memory-utilization 0.5 \
  --qwen-enforce-eager

Retrieval tests:

python -m pytest -q \
  scripts/test_knowval_retrieval.py

Python API:

from knowval_retrieval import (
  KnowValRetrievalPipeline,
  VLLMKeywordExtractor,
  format_knowledge_entries,
  load_knowledge_base,
  retrieve_scene_knowledge,
)

kb = load_knowledge_base('knowval_retrieval/examples/traffic_law_sample_kg.json')
result = retrieve_scene_knowledge(
    'knowval_retrieval/examples/perception_case_crosswalk.json',
    kb,
)
pipeline = KnowValRetrievalPipeline.from_knowledge_base(kb)
stage_outputs = pipeline.retrieve_for_scene(
  'knowval_retrieval/examples/perception_case_crosswalk.json',
  top_k=4,
)
vllm_extractor = VLLMKeywordExtractor('/path/to/Qwen2.5-3B')
vllm_result = retrieve_scene_knowledge(
  'knowval_retrieval/examples/perception_case_crosswalk.json',
  kb,
  keyword_extractor=vllm_extractor,
  mode='vllm_qwen',
)
entries = format_knowledge_entries(
  result['retrieved_knowledge_entries'],
  num_entries=16,
)
handoff = result['value_model_handoff']

The main paper-aligned output fields are:

• legal_query_frame: the scene-conditioned retrieval query frame.
• keyword_layers: high-level context keywords and low-level entity/event keywords. In paper-aligned vLLM/Qwen mode, used_llm is true. In public offline_demo mode, used_llm is false. With VLLMKeywordExtractor, this field records backend="vllm_qwen", the model path, Qwen's raw response, and merged deterministic local keywords for retrieval robustness.
• graph_candidates: seed entities plus tag-based graph-expansion candidates.
• native_clauses: unmodified traffic-rule clauses selected after filtering.
• retrieved_knowledge_entries: ranked K_j entries.
• knowledge_entries: fixed-size K_1..K_NK list with padding.
• value_model_handoff: metadata for embedding the textual entries into rule_embed.
• supplementary_perception_items: retrieval-derived perception checks that mirror the paper's feedback from knowledge retrieval to perception.
• method_metadata: metadata for method-stage names, K_j symbols, and the Value Model field names.

method_metadata["stages"] records the eight public stage names:

perception_verbalizer
two_layer_keyword_extraction
kg_entity_retrieval
graph_topk_expansion
native_clause_filter
knowledge_item_ranking
knowledge_embedding_handoff
supplementary_perception_feedback

For compatibility, the generic fields query_frame and ranked_laws are retained as aliases.

Input Format

The paper-level runtime input to retrieval is scene perception, which is converted by the perception verbalizer into a query and then into two-layer keywords. The public JSON knowledge-base file is a compact release format for the offline sample KG; it is not the raw paper input.

Knowledge Base Schema

The compact knowledge base is a JSON object with a laws list. Required fields are article_id, title, clause, and is_native_clause. The keywords, aliases, scene_tags, and risk_tags fields are lightweight sample index metadata used by the deterministic offline retrieval path to mimic the paper's entity/key retrieval and graph expansion without shipping the full LightRAG cache or calling an LLM.

{
  "laws": [
    {
      "article_id": "road_safety_law_47",
      "title": "Crosswalk Pedestrian Yielding",
      "clause": "When passing a pedestrian crosswalk, a motor vehicle should reduce speed. If pedestrians are crossing the crosswalk, the vehicle should stop and yield.",
      "is_native_clause": true,
      "keywords": ["crosswalk", "pedestrian", "reduce speed", "stop and yield"],
      "scene_tags": ["crosswalk"],
      "risk_tags": ["pedestrian"]
    }
  ]
}

In the full paper system, analogous retrieval keys are produced during knowledge-graph construction and inference by LLM-based entity/key extraction. In this public sample, they are stored explicitly so the demo can run deterministically and offline.

Perception Input Schema

The perception JSON follows the lightweight schema used by the retrieval module. This is the scene-side input corresponding to the paper's perception-verbalizer stage:

{
  "scene": {
    "type": "crosswalk",
    "pedestrian_crossing": true
  },
  "ego": {
    "action": "go_straight",
    "speed_kmh": 24
  },
  "objects": [
    {
      "type": "pedestrian",
      "relative_position": "front",
      "distance_m": 12,
      "state": "crossing"
    }
  ]
}

License

This repository is released under the Apache License 2.0. See LICENSE for details. In addition, the project is free only for academic research purposes; it requires authorization for commercial use. For collaboration or commerce permission, please contact wyt@pku.edu.cn.