---

Overview

A wearable assistant should do two things: answer questions about its visual history, and recognize when that history is useful to the present situation. Existing video-memory systems mostly support question-conditioned recall, while proactive assistants typically bolt on separate memory and control mechanisms.

GROVE supports both behaviors with a single memory grown causally from a continuous stream. It retains fine-grained perceptual evidence and incrementally consolidates it into time-stamped moments, coherent episodes, and recurring cross-day patterns. Each stratum is paired with a scale-native retrieval skill — for locating an observation, replaying an activity, or traversing long-range regularities.

Reactive QA and proactive assistance share this memory and the same access interface. They differ only in whether retrieval is initiated by a user query or by the current situation.

Key properties

• Training-free. No fine-tuning; the memory is built and read by prompted LLM/VLM calls.
• Causal by construction. Every retrieval is clamped to the moment it is issued, so no answer can depend on video the system had not yet seen.
• Incremental. Construction is streaming and checkpointed — memory is queryable at any point, not only after the video ends.
• One memory, two behaviors. The same strata serve query-driven recall and situation-driven proactive assistance.

---

Results

GROVE is evaluated on five benchmarks spanning minutes-long clips to month-long recordings. All numbers are accuracy or macro-F1 (higher is better); bold is best, underline second best.

Long-horizon reactive QA — MM-Lifelong

Model	Month	Week	Day
Human	80.4	95.6	99.2
GPT-5	14.87	15.00	15.25
Qwen3-VL-235B-A22B	14.33	15.63	12.44
Video-XL-2-8B	9.07	12.00	9.00
VideoMind-7B	8.35	11.75	7.50
DeepVideoDiscovery	10.57	9.02	10.25
ReMA	18.62	18.82	16.75
GROVE (ours)	19.98	22.75	23.50

Online video understanding — OVO-Bench & StreamingBench

Model	OVO Real-Time	OVO Backward	OVO Forward	OVO Overall	SB Real-Time	SB Contextual	SB Avg
Human Agents	93.20	92.33	92.90	92.81	91.46	93.55	92.51
GPT-4o	64.5	60.8	53.4	59.5	73.3	38.7	56.0
Dispider	54.6	36.1	34.7	41.8	67.6	33.6	50.6
StreamForest	61.2	52.0	53.5	55.6	77.3	–	–
StreamBridge	71.3	68.1	48.4	62.6	77.0	32.6	54.8
ViSpeak	66.3	57.5	54.3	59.4	70.4	43.9	57.2
TimeChat-Online	58.6	42.0	36.4	45.7	75.4	35.3	55.4
GROVE (ours)	76.2	69.9	56.5	67.5	77.0	53.1	65.1

Proactive assistance — EgoServe

Ten service sub-types grouped into four response horizons. Macro-F1 over all ten.

Model	Instant SA / TU	Short-term NSG / ER / RR	Episodic MR / TR	Long-term HC / ML / RO	Overall
Qwen3-VL-Plus	5.2 / 1.5	8.6 / 10.4 / 3.4	0.0 / 1.8	4.4 / 0.0 / 0.0	3.5
GPT-5-mini	12.5 / 3.6	9.5 / 1.0 / 5.2	0.0 / 9.4	5.7 / 0.0 / 0.0	4.7
EgoMemo	11.4 / 7.5	24.7 / 1.7 / 4.7	3.8 / 5.7	3.7 / 4.9 / 11.8	8.0
GROVE (ours)	19.7 / 20.2	21.8 / 23.8 / 6.8	5.9 / 5.9	8.6 / 6.5 / 7.1	12.6

Proactive timing — ESTP-Bench

Model	EPT	IPT	CQ	Overall
EyeWO (benchmark's own model)	23.6	52.5	43.6	34.7
Qwen2-VL	15.4	39.3	10.4	21.3
MiniCPM-V	15.2	36.5	26.6	22.9
LLaVA-NeXT-Video	16.5	34.6	13.8	21.3
MMDuet	9.1	34.2	20.3	17.8
GROVE (ours)	22.7	34.4	41.0	28.6

---

How it works

Construction proceeds one window at a time. A VLM captions the window and emits a structured perceptual record; a segmenter decides at each window boundary whether the current activity continues or a new episode begins; on a cut, the closed episode is summarized and its moments extracted, and the episode is matched against existing patterns or seeds a new one. All four strata stay queryable throughout.

Retrieval exposes four scale-native skills — perception lookup, moment recall, episode replay, and pattern traversal — over a dual index that fuses BM25 with dense similarity via reciprocal rank fusion. Reactive QA lets the agent choose its own calls across multiple rounds; proactive assistance assembles one evidence block per screening unit.

---

Installation

git clone https://github.com/SitongGong/GROVE.git
cd GROVE
pip install -r requirements.txt

Set the environment before running anything:

export PYTHONPATH=$(pwd):$PYTHONPATH
export OPENAI_API_KEY=<your key>
export DATA_ROOT=<where benchmark videos and annotations live>

Local VLM captioning expects an OpenAI-compatible endpoint; pass --vllm_base_url to the construction scripts. No API keys are embedded in the code — machine-specific paths are placeholders (${DATA_ROOT}, ${GROVE_ROOT}, ${GROVE_WORK}) that you set or override with the corresponding CLI flags.

---

Quick start

# 1. build a memory from a video stream
python -m hypervideo.egolife_hyper_processing \
    --video_dir   ${DATA_ROOT}/EgoLife/A1_JAKE \
    --working_dir ./memory/a1_jake \
    --window_seconds 30 --interval_seconds 5

# 2. answer questions against it
python -m hypervideo.sp_agentic_qa_u1 \
    --mode day \
    --working_dir ./memory/a1_jake \
    --gt ${DATA_ROOT}/MM-Lifelong/day/test.json \
    --answer_model gpt-5.2 --max_rounds 8 --max_chapters 46 \
    --out_dir ./predictions/day

Construction writes hypergraph.json, structured_perception.jsonl and the text indices into --working_dir, checkpointing as it goes — rerunning with the same directory resumes rather than restarting.

---

Datasets

Download each benchmark from its release below, then point DATA_ROOT at the directory holding them (or pass explicit paths on the CLI).

Benchmark	What it tests	Download
MM-Lifelong	Day / week / month-scale reactive QA	🤗 MM-Lifelong/MM-Lifelong
OVO-Bench	Online video understanding (real-time / backward / forward)	🤗 JoeLeelyf/OVO-Bench
StreamingBench	Real-time and contextual streaming understanding	🤗 mjuicem/StreamingBench
ESTP-Bench	When to speak in an egocentric stream	🤗 SitongGong/ESTP-IT
EgoServe	Proactive service across ten sub-types	🤗 SitongGong/EgoServe

EgoServe is built on top of EgoLife, HoloAssist and CaptainCook4D; obtain the source videos from those releases.

---

Repository layout

hypervideo/                 # construction + retrieval (main package)
├── extractors/             #   moment / episode / pattern extraction stages
├── index/                  #   BM25 + dense indices
├── retrieval/              #   cross-stratum traversal
├── prompts/                #   captioning, segmentation, extraction prompts
├── skills/                 #   retrieval skill definitions
└── modules/                #   async pipeline, entity id, audio cues, profiling
videorag/                   # upstream VideoRAG utilities (third-party)
assets/                     # figures used in this README

Memory construction entry points

One script per source domain. Each consumes a video stream and writes the memory plus text indices to --working_dir.

Benchmark	Script
MM-Lifelong, EgoServe (EgoLife)	hypervideo/egolife_hyper_processing.py
EgoServe (HoloAssist)	hypervideo/holoassist_hyper_processing.py
EgoServe (CaptainCook4D)	hypervideo/captaincook4d_hyper_processing.py
OVO-Bench	hypervideo/ovobench_hyper_processing.py
StreamingBench	hypervideo/streamingbench_hyper_processing.py
ESTP-Bench	hypervideo/estp_async_processing.py
OVO forward track	hypervideo/forward_hyper_processing.py

Inference entry points

Benchmark / track	Script
MM-Lifelong day	hypervideo/sp_agentic_qa_u1.py
MM-Lifelong week	hypervideo/sp_agentic_qa_u2.py
MM-Lifelong month	hypervideo/sp_agentic_qa_month.py
OVO-Bench real-time & backward	hypervideo/agentic_solver_merge4rt.py
OVO-Bench forward	hypervideo/forward_hyper_retrieval.py
StreamingBench real-time	hypervideo/agentic_solver_sb3.py
StreamingBench contextual / SQA / proactive	hypervideo/streamingbench_json_eval.py
ESTP-Bench single-query	hypervideo/estp_streaming_inference.py
ESTP-Bench conversational	hypervideo/estp_cq_inference.py
EgoServe (EgoLife)	hypervideo/egoserve_service_engine_v4_clean.py
EgoServe (HoloAssist)	hypervideo/holoassist_service_inference_v10.py
EgoServe (CaptainCook4D)	hypervideo/captaincook4d_service_inference.py

---

Notes

• videorag/ is derived from an upstream project and retains its own header.
• Every LLM call is retried up to three times with exponential backoff, and its output passes through a permissive JSON repair step.
• Defaults in the scripts match the reported runs unless a flag overrides them.

Citation

If you find GROVE useful, please consider citing:

@article{gong2026grove,
  title   = {GROVE: Growing and Reasoning over Temporally Stratified Memory
             from Streaming Video Experience},
  author  = {Gong, Sitong and Kang, Caixin and Yan, Tianyu and Chen, Guo and
             Zheng, Bo and Zhang, Kaipeng and Zhuge, Yunzhi and Ruan, Xiang and
             Lu, Huchuan and Huang, Yifei},
  journal = {arXiv preprint arXiv:2608.02392},
  year    = {2026}
}