[IEEE Robotics and Automation Letters 2026] SurgRAW: Multi-Agent Workflow with Chain-of-Thought Reasoning for Robotic Surgical Video Analysis

May 19, 2026 · View on GitHub

Chang Han Low, Ziyue Wang, Tianyi Zhang, Zhuo Zhu, Zhitao Zeng, Evangelos B. Mazomenos, Yueming Jin

📄 Published in IEEE Robotics and Automation Letters (2026)
🔗 Early Access: https://ieeexplore.ieee.org/document/11397309
📌 DOI: 10.1109/LRA.2026.3665443
🤗 Dataset: https://huggingface.co/datasets/FatNinjaaaaa/SurgCOTBench

🏆 News

🎉 Accepted to IEEE Robotics and Automation Letters (RA-L), 2026
📖 Early Access version available on IEEE Xplore
🚀 Full Code and COT released
🤗 SurgCOTBench dataset released on Hugging Face: https://huggingface.co/datasets/FatNinjaaaaa/SurgCOTBench

🔍 Overview

SurgRAW architecture

SurgRAW employs specialized prompts and a hierarchical orchestration system across five core surgical intelligence tasks:

Instrument Recognition
Action Recognition
Action Prediction
Patient Data Extraction
Outcome Assessment

Key Features

Chain-of-Thought Agents – Task-specific prompts guide VLM agents through structured reasoning, reducing hallucinations and improving explainability.
Hierarchical Orchestration – A Department Coordinator routes queries to visual-semantic or cognitive-inference agents, mirroring real surgical workflows.
Panel Discussion – An Action Evaluator cross-checks visual-semantic predictions using a knowledge graph and rubric-based evaluation for logical consistency.
Retrieval-Augmented Generation (RAG) – Cognitive-inference tasks are grounded in external medical knowledge for reliable, domain-specific responses.

📊 SurgCoTBench Dataset

We evaluate SurgRAW on SurgCoTBench — the first reasoning-based dataset covering the entire surgical workflow.

12 robotic procedures
2,277 frames
14,176 vision–query pairs
5 task categories aligned with the SurgRAW framework

Release Plan: SurgCoTBench and the corresponding Chain-of-Thought prompts will be made available with our paper.
You may also use SurgCoTBench or any dataset that includes the following columns in its .xlsx file:

image_path
question
ground_truth

📌 Current Status

This repository currently showcases:

The SurgRAW agentic framework architecture
Collaboration metrics

Dataset and full CoT prompt releases will follow publication. Collaborations are warmly welcomed.

⚙️ Setting Up the Environment

Follow these steps to set up the SurgRAW environment:

# 1️⃣ Create a new conda environment
conda create -n SurgRAW python=3.12 -y

# 2️⃣ Activate the environment
conda activate SurgRAW

# 3️⃣ Install required Python packages
pip install -r requirements.txt

Ensure requirements.txt is in the project root.
For GPU, install the CUDA-matching PyTorch wheels per the official PyTorch instructions.

🚀 Running SurgRAW

Run the orchestration pipeline on your .xlsx dataset using the provided script (which calls final_orchestrator under the hood).

python run_orchestration.py   --xlsx_file /path/to/your/input.xlsx   --log_dir /path/to/save/logs

Arguments

--xlsx_file – Path to the Excel file with columns: image_path, COT_Process, question_mcq, ground_truth (optional)
--log_dir – Directory where per-row logs (*.txt) will be written

Example

python run_orchestration.py   --xlsx_file data/SurgCoTBench_sample.xlsx   --log_dir logs/

Each row produces a dedicated log file named like:

<image_name>_<COT_FileNamingConvention>_SurgCOT.txt

🖼 Case Studies

Chain-of-Thought example

🎥 Video Demo

We provide a full demonstration of the SurgRAW multi-agent orchestration pipeline in action.

End-to-end execution of SurgRAW demonstrating structured Chain-of-Thought reasoning and multi-agent coordination.

📚 Citation

If you find this work useful, please cite our paper:

@ARTICLE{11397309,
  author={Low, Chang Han and Wang, Ziyue and Zhang, Tianyi and Zhuo, Zhu and Zeng, Zhitao and Mazomenos, Evangelos B. and Jin, Yueming},
  journal={IEEE Robotics and Automation Letters}, 
  title={SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Robotic Surgical Video Analysis}, 
  year={2026},
  volume={},
  number={},
  pages={1-8},
  keywords={Cognition;Surgery;Instruments;Videos;Robots;Visualization;Reliability;Robot kinematics;Biomedical imaging;Accuracy;Robotic Surgery;Surgical Data Science;Agentic AI;Datasets for Robotic Vision},
  doi={10.1109/LRA.2026.3665443}}

Acknowledgement

This paper was recommended for publication by Editor Jessica Burgner-Kahrs upon evaluation of the Associate Editor and Reviewers' comments. This work was supported by Ministry of Education Tier 2 grant, Singapore (T2EP20224-0028), and Ministry of Education Tier 1 grant, Singapore (23-0651-P0001).

Have fun with our work!