🌐 Website   |   📑 Paper   |   🤗 Dataset   |   🤖 Models  

Introduction

Key Features

🎯 High-Quality Desktop Dataset

• Dense, expert-annotated screenshots with maximum annotation density
• Coverage of almost every visible element, including small icons and controls
• Fine-grained category information (menus, sidebars, etc.) for 50% of UI elements—fully open-source!

⚡ Efficient Model Training

• State-of-the-art performance with 700K datapoints vs 9M+ in prior work
• Two-stage training: supervised fine-tuning + reinforcement learning with fully open-source code
• Models at 3B and 7B scales for efficiency and accuracy

🌐 Cross-Platform Generalization

• Comprehensive evaluation on five challenging benchmarks
• Robust generalization across desktop, mobile, and web environments despite training only on desktop data

🚀 Quick Start

Installation & Setup

To install from PyPI (recommended):

# Create and activate environment
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip  # optional

# Install PyTorch (adjust for CUDA version) and Flash Attention (for faster inference)
pip install torch torchvision
pip install flash-attn --no-build-isolation

# Install GroundCUA package for utilities
pip install groundcua  # basic dependencies
pip install groundcua[all] # full dependencies (optional)

# Create and activate environment
conda create -n groundcua python=3.10 -y
conda activate groundcua

pip install --upgrade pip

# Clone repository
git clone https://github.com/ServiceNow/GroundCUA.git
cd GroundCUA

# Install PyTorch (adjust for your CUDA version)
pip install torch torchvision

# Install Flash Attention (recommended for faster inference)
pip install flash-attn --no-build-isolation

# Install in development mode
pip install -r requirements.txt

Quick GroundNext Model Inference

import torch
from transformers import Qwen2_5_VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from PIL import Image
from io import BytesIO
from urllib.request import urlopen

import groundcua

model_name = "ServiceNow/GroundNext-7B-V0"

# Load model and processor
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    device_map="auto",
    trust_remote_code=True
).eval()

processor = AutoProcessor.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)

# Configure generation
groundcua.set_generation_config(model)

# Load and prepare image
image = Image.open(BytesIO(urlopen(groundcua.SAMPLE_URL).read()))
image, (width, height) = groundcua.prepare_image(image)

# Create messages and generate
instruction = "Click on the 'File' button"
messages = groundcua.create_messages(instruction, image, width, height)

input_text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(text=[input_text], images=[image], videos=None, padding=True, return_tensors="pt")

generated_ids = model.generate(**inputs.to(model.device), max_new_tokens=groundcua.DEFAULT_MAX_NEW_TOKENS)
trimmed_ids = groundcua.trim_ids(generated_ids, inputs.input_ids)

response = processor.batch_decode(trimmed_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)
print(response[0])
# Expected output: <tool_call>{"name": "computer_use", "arguments": {"action": "left_click", "coordinate": [x, y]}}</tool_call>

Updates

• [Nov 11 2025] 🎉 We released our project webpage, the GroundCUA dataset, and the GroundNext-7B model!

Performance

Desktop Grounding Benchmarks

Cross-Platform Generalization

Performance numbers demonstrate strong cross-domain (desktop, mobile and web) generalization despite training only on desktop data.

Agentic Performance on OSWorld

GroundNext models also demonstrate strong agentic capabilities when integrated with reasoning models. When combined with OpenAI o3, GroundNext-3B achieves competitive performance on OSWorld, matching or exceeding much larger models.

Task categories: OS (operating system tasks), Office (productivity applications), Daily (common user tasks), Pro (professional software), Workflow (multi-step workflows).

Key Results

• Data Efficiency: Achieves SOTA with only 700K training examples vs 9M+ in prior work
• Cross-Domain Excellence: Strong performance across desktop, mobile, and web despite desktop-only training
• Fine-Grained Grounding: Superior performance on small UI elements and complex workflows

🎓 Training

Dataset

GroundCUA Dataset Overview

GroundCUA is a large-scale, human-annotated desktop grounding dataset with dense supervision:

• 📊 Scale: 56K annotated screenshots, 3.56M element annotations
• 🎯 Density: Maximum annotation density covering almost every visible UI element
• ✅ Quality: Human-verified annotations from trained experts
• 🖥️ Coverage: 87 desktop applications across 12 categories
• 📐 Resolution: High-resolution images (500K to 7M pixels)
• 🏷️ Categories: Fine-grained category information for 50% of elements

Dataset Access

Download the GroundCUA dataset:

pip install -U huggingface_hub
huggingface-cli download ServiceNow/GroundCUA --repo-type dataset --local-dir ./GroundCUA

📊 Evaluation

Supported Benchmarks

• ScreenSpot-Pro: Desktop element grounding
• ScreenSpot-v2: Web and mobile interface grounding
• MMBench-GUI: GUI understanding tasks
• OSWorld-G: Operating system grounding
• UI-Vision: Diverse desktop application grounding

Running Evaluations

cd eval/

# Evaluate on specific benchmark
python eval.py \
    --model_type qwen25vl \
    --model_name_or_path /path/to/trained/model \
    --benchmark screenspot \
    --data_path /path/to/benchmark/data \
    --output_dir results/

# Evaluate on all benchmarks
python eval.py \
    --model_type qwen25vl \
    --model_name_or_path /path/to/trained/model \
    --benchmark all \
    --task all \
    --language en

Evaluation Metrics

• Accuracy: Precision of GUI element localization
• Success Rate: Percentage of correctly grounded elements
• Cross-Domain Performance: Generalization to unseen platforms
• Fine-Grained Performance: Accuracy on small UI elements

Project Structure

GroundCUA/
├── README.md                    # This file
├── pyproject.toml              # Package configuration
├── PUBLISHING.md               # Guide for publishing to PyPI
├── assets/                      # Images and resources
├── groundcua/                  # Main package (pip installable)
│   ├── __init__.py             # Package initialization and utilities
│   └── version.py              # Version information
├── eval/                        # Evaluation framework
│   ├── eval.py                 # Main evaluation script
│   ├── data.py                 # Data loading utilities
│   ├── prompts.py              # Prompt processing
│   └── models/                 # Model implementations
└── training/                   # Training pipeline (documentation coming soon)

Acknowledgements

We thank the following projects and teams for their contributions to the open-source community:

• InfiGUI-G1 for the evaluation framework foundation
• LLaMA-Factory for the excellent SFT training framework
• verl for the robust RL infrastructure
• Qwen-2.5-VL for the foundation vision-language models
• OpenCUA for design inspiration of repository
• The computer use and GUI automation research community

Research Use and Disclaimer

GroundCUA is intended for research and educational purposes only.

Prohibited Uses

• The model, dataset, and code may not be used for any purpose that violates applicable laws or regulations
• Use for illegal, unethical, or harmful activities is strictly prohibited

Disclaimer

• The authors and contributors are not responsible for any illegal, unethical, or harmful use
• Users are solely responsible for ensuring compliance with applicable laws and regulations

Citation

If you use GroundCUA in your research, please cite our work:

@misc{feizi2025groundingcomputeruseagents,
      title={Grounding Computer Use Agents on Human Demonstrations}, 
      author={Aarash Feizi and Shravan Nayak and Xiangru Jian and Kevin Qinghong Lin and Kaixin Li and Rabiul Awal and Xing Han Lù and Johan Obando-Ceron and Juan A. Rodriguez and Nicolas Chapados and David Vazquez and Adriana Romero-Soriano and Reihaneh Rabbany and Perouz Taslakian and Christopher Pal and Spandana Gella and Sai Rajeswar},
      year={2025},
      eprint={2511.07332},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2511.07332}, 
}