VQASynth 🎹

Spatial Reasoning is fundamental to interacting within and navigating physical environments for embodied AI applications like robotics. However, data samples suitable for learning these capabilities are rare in AI pretraining datasets. Don't be limited by what your model can do out-of-the-box, curate any image dataset from the Huggingface Hub for Spatial VQA with tools for 3D scene understanding.

VLMs trained using VQASynth 🎹

• estimate 3D distances between objects in an image
• describe distances colloquially, convert between common units of measurement
• answer queries about the orientation and spatial relationships between objects
• base responses on consistent references like floors and surfaces
• apply CoT "thinking" for more robust reasoning and better estimates

Run a Pipeline on Your Image Dataset

Environment

Before running the demo scripts, ensure you have the following installed:

• Python 3.10 or later
• Docker, Docker Compose V2
• NVIDIA Container Toolkit
• At least 24GB VRAM (A10 or larger) and 16GB RAM

Use Docker Compose to transform Image datasets from Huggingface Hub into VQA datasets describing spatial relations between objects. You can process different datasets after updating the config.yaml.

Then run the spatial VQA pipeline locally with Docker:

# Authenticate to push to hub
huggingface-cli login

# Run the pipeline
cd /path/to/VQASynth
bash run.sh

You can run the colab notebook (requires A100 runtime) or compose the pipeline modules in-process to turn one image into grounded spatial VQAs:

from PIL import Image
from vqasynth.localize import Localizer
from vqasynth.scene_fusion import SpatialSceneConstructor
from vqasynth.prompts import PromptGenerator

image = Image.open("warehouse.jpg").convert("RGB")

# Detect + segment task-relevant objects
masks, _, captions = Localizer(captioner_type="florence").run(image)

# Lift to 3D — VGGT emits per-object point clouds, depth, and intrinsics in one pass
pcd_filepaths, canonicalized, _, _ = SpatialSceneConstructor().run(
    "warehouse_0", image, masks, output_dir="./scenes"
)

# Generate spatial VQAs from the reconstructed 3D scene
qa_pairs = PromptGenerator().run(captions, pcd_filepaths, canonicalized)

# Example output:
#   "How close is the man in red hat walking from the wooden pallet with boxes?"
#   → "Approximately 60.13 centimeters."

The resulting Huggingface dataset is in the cache directory and you can push to hub with:

from vqasynth.datasets import Dataloader
Dataloader(cache_dir).push_to_hub(final_dataset, target_repo_name)

Description

Fusing semantic and metric data into templated VQA chat, Vision Language Models can be instruction-tuned with low-rank adapters to enhance their baseline spatial reasoning capabilities. VQASynth 🎹 provides an open-source reproduction of SpatialVLM, which describes a 3D scene reconstruction pipeline and prompt templates for enhancing the spatial reasoning abilities of VLMs including:

🪄 Object-grounded captions by point prompting with Molmo

📐 Improves metric depth estimation speed & accuracy by replacing DepthPro with VGGT.

🪢 SAM2 replaces SAM in the localization refinement stage.

🧠 Multimodal "Thinking" by CoT Reasoning

Datasets made using VQASynth 🎹

• SpaceOm
• SpaceThinker
• OpenSpaces_MC_R1
• OpenSpaces
• vqasynth_spacellava

Models trained using VQASynth 🎹

• SpaceOm - the best overall
• SpaceThinker-Qwen2.5VL-3B - the most accurate distance estimates
• SpaceQwen2.5-VL-3B-Instruct - 3B - the most popular
• SpaceLLaVA - 13B - the original
• SpaceLLaVA - 7B - the most interpretable

Examples from SpaceThinker:

Prompt: How far is the man in the red hat from the pallet of boxes in feet?	Prompt: How far is the Goal from the penalty taker in this image?	Prompt: Approximately how many meters apart are the chair and bookshelf?
Reasoning: To determine how far the man in the red hat is from the pallet of boxes, I need to consider the spatial arrangement in the warehouse. The man is walking on the floor, and there's a visible gap between him and the pallet. First, I'll estimate the size of the pallet. A standard pallet is typically around 4 feet wide and 8 feet long. Given that the man is standing near the edge of the pallet, he might be about 2–3 feet away from the front edge of the pallet. Next, I'll consider the distance from the pallet to the man. The man appears to be walking along the aisle, which seems to be about 6–7 feet wide. If he is standing at the edge of the pallet, he would likely be about 2–3 feet from the front edge of the pallet. Combining these observations, the man in the red hat is approximately 2–3 feet from the pallet of boxes.	Reasoning: First, I observe the scene where a player in a red uniform stands near the goal. The goal itself is a standard soccer goalpost, which typically extends 2.5 meters beyond the line of the goalposts. The player appears to be standing close to the goal, but not directly at the goal line. To estimate the distance, I consider that the player's position relative to the goal suggests he is likely standing closer than 2.5 meters away. Given the typical setup of a penalty kick scenario, the player would be positioned further back from the goal than the goal itself. By visual estimation, the player seems to be standing approximately 10–15 meters from the goal. This range aligns with the common distance for a penalty kick, considering the player's stance and the goal's structure. Thus, the distance from the goal to the penalty taker is likely around 10–15 meters.	Reasoning: To determine the distance between the chair and the bookshelf, I'll consider the typical dimensions of these objects. A standard office chair is around 60–70 cm tall, and a bookshelf can be anywhere from 1.2 to 1.8 meters tall. Given that the chair is likely placed on a desk or floor level, the height difference between the chair and the top of the bookshelf would be approximately 1 meter. Assuming the bookshelf is at least 1 meter high, and the chair is at ground level, the vertical distance between them is indeed 1 meter. The horizontal distance might vary depending on the layout, but given the room's dimensions and the placement of the chair relative to the bookshelf, the total distance between the chair and the top of the bookshelf is likely around 1 meter. Therefore, the chair and bookshelf are approximately 1 meter apart.
Answer: 2–3 feet	Answer: 10–15 meters	Answer: 1 meter

Synthetic spatial reasoning traces suffice to train VLMs capable of grounded, quantitative spatial reasoning—inferring accurate distances, interpreting 3D scene context, and formatting open-ended answers precisely by integrating visual cues, real-world object priors, and human-centric spatial logic.

Model Evaluation

Report VLM spatial reasoning on benchmarks: SpatialScore, OmniSpatial, SpaCE-10, MindCube

Notebooks

We've hosted some notebooks visualizing and experimenting with the techniques included in this repo.

Notebook	Description	Launch
Generate Spatial VQA Dataset	Augment an HF Image Dataset with Spatial VQA
Spatial Reasoning with Point Clouds	Visualize point clouds and evaluate spatial relationships
Evaluate SpaceThinker on QSpatial++	Assess SpaceThinker's quantitative spatial reasoning on the QSpatial++ benchmark
SpaceLLaVA Attention with TransformerLens	Visualize SpaceLLaVA-7B attention patterns using TransformerLens

Agent with VQASynth Tools

The VQASynth tool inventory — Florence-2 detection/OCR, DepthPro/VGGT metric depth, 3D distance measurement — is also exposed as an NOOA-based agent (experiments/nooa_agent/) that composes tool calls dynamically per prompt rather than following a pre-templated pipeline. Useful when the input question isn't known at pipeline-design time.

• Dynamic tool composition — the LLM decides which tools to call and in what order per question. detect_objects → metric_depth → distance_3d for a "how far apart" query; caption_scene → dense_region_captions for a scene inventory. No pre-coded question types.
• Dense reward annotation — plug into a robot-learning pipeline as a per-frame reward or CoT source. The agent satisfies lerobot's VlmClient protocol via SpatialAnnotatorVlmClient, so it drops into any annotation module (VQA, plan, ECoT) as a tool-grounded alternative to a raw VLM call.
• Traces grounded on tool calls — every annotate() streams a JSONL row in OpenAI-messages format (ready for Qwen2.5-VL / Qwen3-VL fine-tuning) preserving the full tool-call chain, not just the final answer. Reasoning stays auditable.

Quick example:

from PIL import Image
from experiments.nooa_agent.spatial_annotator import SpatialAnnotator, SceneContext
from experiments.nooa_agent.trace import TraceWriter
from nooa.unifiedllm.registry import get_llm_client

agent = SpatialAnnotator(llm=get_llm_client("gemini/gemini-2.5-pro"))
img = Image.open("warehouse.jpg").convert("RGB")

with TraceWriter("traces.jsonl") as writer:
    scene = SceneContext(agent, img, trace_writer=writer, image_ref="warehouse.jpg")
    result = await scene.annotate("How far apart are the two workers in the foreground?")

print(result.answer)            # "The two workers in the foreground are 2.04 meters apart."
print(result.confidence)        # "high"
print(result.tool_calls_used)   # 4

See experiments/nooa_agent/README.md for install, resource tiers (CPU with DepthPro / GPU with VGGT), the full tool inventory, and the lerobot integration path.

References

This project was inspired by or utilizes concepts discussed in the following research paper(s):

@article{chen2024spatialvlm,
  title = {SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities},
  author = {Chen, Boyuan and Xu, Zhuo and Kirmani, Sean and Ichter, Brian and Driess, Danny and Florence, Pete and Sadigh, Dorsa and Guibas, Leonidas and Xia, Fei},
  journal = {arXiv preprint arXiv:2401.12168},
  year = {2024},
  url = {https://arxiv.org/abs/2401.12168},
}
@inproceedings{wang2025vggt,
  title={VGGT: Visual Geometry Grounded Transformer},
  author={Wang, Jianyuan and Chen, Minghao and Karaev, Nikita and Vedaldi, Andrea and Rupprecht, Christian and Novotny, David},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2025}
}
@article{ravi2024sam2,
  title={SAM 2: Segment Anything in Images and Videos},
  author={Ravi, Nikhila and Gabeur, Valentin and Hu, Yuan-Ting and Hu, Ronghang and Ryali, Chaitanya and Ma, Tengyu and Khedr, Haitham and R{\"a}dle, Roman and Rolland, Chloe and Gustafson, Laura and Mintun, Eric and Pan, Junting and Alwala, Kalyan Vasudev and Carion, Nicolas and Wu, Chao-Yuan and Girshick, Ross and Doll{\'a}r, Piotr and Feichtenhofer, Christoph},
  journal={arXiv preprint arXiv:2408.00714},
  url={https://arxiv.org/abs/2408.00714},
  year={2024}
}