Updates

• [2026/06] 🎉 LocateAnything is accepted to ECCV 2026.
• [2026/06] 🔥 Release visual prompt fine-tuning script for LocateAnything, with LoRA fine-tuning for efficient adaptation.
• [2026/06] 🔥 Release batch inference with the optional la_flash runtime for efficient inference on A100, RTX 4090, and other non-Hopper/Blackwell GPUs.
• [2026/05] 🔥 Release LocateAnything, a generalist vision-language grounding model based on Eagle.

LocateAnything is a vision-language model for fast and high-quality visual grounding, enabling precise object localization, dense detection, and point-based localization across diverse domains in both Enterprise Intelligence and Physical AI. The model adopts a generalist design, supporting tasks such as referring expression grounding, multi-object detection, GUI element grounding, and text localization, with strong performance in complex and cluttered scenes.

• ⚡ Parallel Box Decoding (PBD) — atomic, single-step decoding of full bounding boxes / points.
• 🔁 Hybrid Inference — Fast Mode (MTP) by default, with seamless NTP fallback for stability.
• 📚 LocateAnything-Data — 138M language queries, 785M boxes, covering detection, GUI grounding, referring comprehension, OCR, layout, and pointing.
• 🏆 State-of-the-Art — 12.7 BPS on a single H100 (≈ 10× Qwen3-VL, 2.5× Rex-Omni), with SOTA accuracy on LVIS, M6Doc, ScreenSpot-Pro, and more.

Note: The currently released nvidia/LocateAnything-3B weights do not support visual prompt inference out of the box. Visual-prompt-capable weights will be released in a future version.

🎬 Visual Demo

🧠 Method

Vision-language models (VLMs) commonly formulate visual grounding and detection as a coordinate-token generation problem, serializing each 2D box into multiple 1D tokens that are learned and decoded largely independently. This token-by-token decoding mismatches the coupled structure of box geometry and creates a practical inference bottleneck due to strictly sequential generation.

We introduce LocateAnything, a unified generative grounding and detection framework based on Parallel Box Decoding (PBD). By decoding geometric elements such as bounding boxes and points as atomic units in a single step, LocateAnything preserves intra-box geometric coherence and unlocks substantial parallelism. We further develop a scalable data engine and curate LocateAnything-Data, a large-scale dataset with 138M+ training samples. Extensive evaluations show that LocateAnything advances the speed–accuracy frontier on diverse benchmarks.

Feature Summary

Parallel Box Decoding

_{Comparison of standard token decoding methods vs. Parallel Box Decoding (PBD). NTP generates coordinate values one by one; standard MTP produces irregular distributions; PBD generates a complete atomic box in a single parallel step.}

_{Architecture overview of LocateAnything with Parallel Box Decoding.}

On-Demand Inference: Corrected NTP Re-decoding

When parallel decoding encounters format irregularity (malformed syntax at category boundaries) or spatial ambiguity (intermediate coordinates between densely arranged objects), the compromised block is discarded and generation reverts to the last verified prefix. NTP then autoregressively generates tokens for the problematic block before switching back to MTP.

📚 LocateAnything-Data

138M diverse language queries · 785M boxes · 12M unique images

🚀 Installation

git clone https://github.com/NVlabs/Eagle.git eagle
cd eagle/Embodied
pip install -e .

Key dependencies (auto-installed): transformers==4.57.1, tokenizers==0.22.0, deepspeed==0.15.4, accelerate==1.5.2, timm>=1.0.11, liger_kernel==0.3.1, peft==0.12.0, decord.

git clone https://github.com/SandAI-org/MagiAttention.git
cd MagiAttention
git checkout v1.0.5
git submodule update --init --recursive
pip install -r requirements.txt
pip install --no-build-isolation .

⚡ Quick Start

import torch
from PIL import Image
from locateanything_worker import LocateAnythingWorker

worker = LocateAnythingWorker("nvidia/LocateAnything-3B")
img = Image.open("example.jpg").convert("RGB")

# Object Detection
print(worker.detect(img, ["person", "car", "bicycle"])["answer"])

# Phrase Grounding
print(worker.ground_multi(img, "people wearing red shirts")["answer"])

# Scene Text Detection
print(worker.detect_text(img)["answer"])

# GUI Grounding (point)
print(worker.ground_gui(img, "the search button", output_type="point")["answer"])

# Pointing
print(worker.point(img, "the traffic light")["answer"])

See locateanything_worker.py for the full worker API.

🚀 Batch Inference Release

The Hugging Face model repository now includes optional high-throughput inference utilities:

• batch_infer.py: JSONL/image-query batch inference CLI.
• batch_utils/: batched hybrid MTP/NTP scheduler and sampling runtime.
• kernel_utils/: LA Flash sparse range utilities implemented with FlashAttention varlen. This path does not build or ship a custom C++/CUDA extension.

LA_FLASH_ATTN=la_flash keeps LocateAnything's hybrid decoding path while running sparse range plans through FlashAttention. It is intended for inference/evaluation; training should continue to use the standard model code path.

hf download nvidia/LocateAnything-3B --local-dir LocateAnything-3B
cd LocateAnything-3B
export PYTHONPATH="$PWD:${PYTHONPATH:-}"

python batch_infer.py \
  --model . \
  --attn la_flash \
  --vision-attn flash_attention_2 \
  --scheduler pipeline \
  --batch-size 4 \
  --image /path/to/image.jpg \
  --query "person</c>car"

A100 4K probe, real 3840x2160 street image, query=vehicle, batch_size=4, raw PIL input, in_token_limit=25600, hybrid MTP inference:

The worker API can also use the released batch runtime when batch_utils/ and kernel_utils/ are on PYTHONPATH:

from PIL import Image
from locateanything_worker import LocateAnythingWorker

worker = LocateAnythingWorker(
    "nvidia/LocateAnything-3B",
    use_batch_runtime=True,
    attn="la_flash",
    vision_attn="flash_attention_2",
    scheduler="pipeline",
)

img1 = Image.open("street_1.jpg").convert("RGB")
img2 = Image.open("street_2.jpg").convert("RGB")

results = worker.detect_batch([
    (img1, ["person", "car"]),
    (img2, ["traffic light", "bus"]),
])

for result in results:
    print(result["answer"])

🧾 Output Format

The model outputs special tokens to represent bounding boxes and points:

• Bounding box: <ref>label</ref><box><x1><y1><x2><y2></box> — coordinates are integers in [0, 1000] (divide by 1000 for relative coordinates).
• Point: <box><x><y></box>
• No object: <box>none</box>

import re

def parse_boxes(answer: str, image_width: int, image_height: int):
    boxes = []
    for m in re.finditer(r"<box><(\d+)><(\d+)><(\d+)><(\d+)></box>", answer):
        x1, y1, x2, y2 = [int(g) for g in m.groups()]
        boxes.append({
            "x1": x1 / 1000 * image_width,  "y1": y1 / 1000 * image_height,
            "x2": x2 / 1000 * image_width,  "y2": y2 / 1000 * image_height,
        })
    return boxes

🏋️ Training (Continual SFT)

Full fine-tuning from a pretrained checkpoint — single command:

torchrun --nproc_per_node=8 \
  eaglevl/train/locany_finetune_magi_stream.py \
  --model_name_or_path nvidia/LocateAnything-3B \
  --meta_path "./locany_recipe/your_recipe.json" \
  --output_dir work_dirs/my_sft \
  --max_steps 25000 \
  --learning_rate 2e-5 \
  --bf16 True \
  --block_size 6 \
  --attn_implementation magi \
  --max_seq_length 16384 \
  --deepspeed "deepspeed_configs/zero_stage2_config.json"

For the complete training guide (all arguments, data recipe format, multi-node setup, streaming packing, checkpoint resume), see Training Documentation.

For data format details, annotation conventions, and recipe configuration, see Data Preparation.

Visual Prompt Fine-Tuning

We release visual prompt fine-tuning support for tasks where an image crop is used as the query instead of a category name. During training, datasets marked with visual_prompt=true automatically convert positive single-category detection prompts into cropped visual prompts from the source image. The source image remains the target image, and the crop is appended as an additional image placeholder.

Important: The public nvidia/LocateAnything-3B checkpoint does not currently support visual prompt inference. Use the released code to fine-tune on your own visual prompt data. Official visual-prompt-capable weights will be released in a future version.

Add visual_prompt: true to the datasets that should be converted into visual prompt training samples:

{
  "my_visual_prompt_data": {
    "annotation": "path/to/visual_prompt.jsonl",
    "root": "/data/images/",
    "repeat_time": 1.0,
    "data_augment": true,
    "visual_prompt": true
  }
}

LoRA Fine-Tuning

We also provide LoRA fine-tuning support for parameter-efficient adaptation. LoRA is useful when you want to adapt LocateAnything without updating all model parameters; by default, the released script enables LLM LoRA (USE_LLM_LORA=64), keeps the LLM and vision backbone frozen, and leaves the MLP projector trainable.

Launch the visual prompt LoRA fine-tuning script with:

export HF_TOKEN=your_hf_token
export META_PATH=./locany_recipe/visual_prompt_recipe.json

bash shell/locate-anything-lora-visual-prompt.sh 1 work_dirs/locany_lora_visual_prompt

Useful environment overrides:

• MODEL_PATH: base checkpoint or local model path, default nvidia/LocateAnything-3B.
• USE_LLM_LORA: LLM LoRA rank, default 64; set 0 to disable.
• USE_BACKBONE_LORA: vision-backbone LoRA rank, default 0; set a positive value to enable.
• FREEZE_LLM, FREEZE_BACKBONE, FREEZE_MLP: control which base modules are frozen.
• MAX_STEPS, LR, SAVE_STEPS, MAX_SEQ_LENGTH: standard training schedule and context-length controls.

📈 Evaluation

# COCO
bash evaluation/scripts/eval_coco.sh --model_path path/to/model --test_jsonl ... --image_root ... --output_dir ...

# LVIS
bash evaluation/scripts/eval_lvis.sh --model_path path/to/model --test_jsonl ... --image_root ... --output_dir ...

# Grounding (Dense200, DocLayNet, HumanRef, RefCOCOg, VisDrone, etc.)
bash evaluation/scripts/eval_grounding.sh --dataset Dense200 --eval_type box_eval --model_path ... --image_root ... --output_base ...

# ScreenSpot-Pro
bash evaluation/scripts/eval_sspro.sh --model_path ... --test_jsonl ... --image_root ... --output_dir ...

See the Evaluation Guide for setup and dataset preparation.

🏆 Results at a Glance

State-of-the-art accuracy at 10× the throughput. Full benchmark tables, ablation studies, and qualitative visualizations are documented separately in 📊 Detailed Results.

👉 For full per-benchmark tables, decoding-mode comparisons, box-ordering ablation, throughput scaling curves, and qualitative visualizations, see document/RESULTS.md.

📖 Citation

@article{wang2025locateanything,
  title   = {LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding},
  author  = {Shihao Wang and Shilong Liu and Yuanguo Kuang and Xinyu Wei and
             Yangzhou Liu and Zhiqi Li and Yunze Man and Guo Chen and
             Andrew Tao and Guilin Liu and Jan Kautz and Lei Zhang and Zhiding Yu},
  journal = {arXiv:2605.27365},
  year    = {2026},
}

🙏 Acknowledgement

We thank the Rex-Omni Team for their evaluation framework.

📜 License

• See LICENSE for the code of this repository.
• See LICENSE_MODEL for the models of LocateAnything.