🔮 A Toolkit for Evaluating NEO-ov Models
May 27, 2026 · View on GitHub
🔮 A Toolkit for Evaluating NEO-ov Models
Comprehensive Evaluation for NEO-ov across Images, Videos, and Spatial Intelligence
🚀 Getting Started
Dependencies
- 🔥 PyTorch
2.8.0 - 🟢 CUDA
12.8 - 🤗 Transformers
4.57.3
Installation
# 1. Create conda environment
conda create -n neo-ov python=3.10 -y
conda activate neo-ov
# 2. Install PyTorch (CUDA 12.8)
pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128
# 3. Install transformers
pip install transformers==4.57.3
# 4. Install other dependencies
pip install -r requirements.txt
Evaluation
bash eval_image.sh # 🧠 Image Understanding
bash eval_video.sh # 🎬 Video Understanding
bash eval_si.sh # 🌐 Spatial Intelligence
💡 Tip: Edit the
DATASETSarray in each script to select specific benchmarks.
Built on top of VLMEvalKit. Refer to the original repository for detailed usage instructions.
🤖 Model Zoo
We release 2B and 9B NEO-ov in Supervised Fine-Tuning (SFT).
| Model Name | Model Weight |
|---|---|
| NEO-ov-2B-SFT |  NEO_ov-2B-SFT HF link |
| NEO-ov-9B-SFT | NEO_ov-9B-SFT HF link |
📊 Benchmark Results
This project evaluates NEO-ov models across 3 major benchmark categories:
| # | Category | Benchmarks | Description |
|---|---|---|---|
| 1 | 🧠 Image Understanding | 12 | Knowledge reasoning, general & OCR visual QA, hallucination detection |
| 2 | 🎬 Video Understanding | 6 | Video comprehension across varying lengths and complexity |
| 3 | 🌐 Spatial Intelligence | 12 | Spatial reasoning on images, multi-images, and videos |
🧠 Category 1: Image Understanding
| Model_NAME | Base_LLM_NAME | Knowledge | General VQA | OCR VQA | Hallucination | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MMMU-VAL | MMB-EN | RealWorldQA | MMStar | SEED-I | AI2D | DocVQA | ChartQA | TextVQA | OCRBench | POPE | HallusionBench | ||
| 🔻Vision Language Models (Instruct-2B) | |||||||||||||
| InternVL3.5 | Qwen3-1.7B | 53.0 | 78.2 | 62.0 | 62.7 | 75.3 | 78.8 | 89.4 | 80.7 | 76.5 | 83.6 | 87.2 | 48.6 |
| Qwen3-VL | Qwen3-1.7B | 53.4 | 78.4 | 63.9 | 58.3 | -- | 76.9 | 93.3 | 79.1 | -- | 85.8 | -- | 51.4 |
| NEO-ov | Qwen3-1.7B | 54.7 | 80.0 | 64.7 | 58.7 | 76.1 | 81.7 | 91.2 | 83.7 | 77.8 | 81.2 | 86.2 | 54.6 |
| 🔻Vision Language Models (Instruct-8B) | |||||||||||||
| InternVL3.5 | Qwen3-8B | 68.1 | 82.7 | 67.5 | 69.3 | 77.1 | 84.0 | 92.3 | 86.7 | 78.2 | 84 | 88.7 | 54.5 |
| Qwen3-VL | Qwen3-8B | 69.6 | 84.5 | 71.5 | 70.9 | -- | 85.7 | 96.1 | 89.6 | -- | 89.6 | -- | 61.1 |
| NEO-ov | Qwen3-8B | 68.1 | 85.1 | 68.8 | 67.2 | 76.5 | 85.4 | 91.8 | 86.1 | 78.6 | 81.5 | 89.0 | 59.8 |
🎬 Category 2: Video Understanding
| Model_NAME | Base_LLM_NAME | Video-MME | MVBench | LVBench | MLVU (M-Avg) | LongVideoBench | Video-MMMU |
|---|---|---|---|---|---|---|---|
| 🔻Vision Language Models (Instruct-2B) | |||||||
| InternVL3.5 | Qwen3-1.7B | 58.4 | 65.9 | 37.6 | 64.4 | 57.4 | 42.7 |
| Qwen3-VL | Qwen3-1.7B | 61.9 | 61.7 | 47.4 | 68.3 | 55.6 | 41.9 |
| NEO-ov | Qwen3-1.7B | 60.4 | 65.7 | 43.3 | 64.8 | 56.8 | 42.3 |
| 🔻Vision Language Models (Instruct-8B) | |||||||
| InternVL3.5 | Qwen3-8B | 66.0 | 72.1 | 45.9 | 70.2 | 62.1 | 54.9 |
| Qwen3-VL | Qwen3-8B | 71.4 | 68.7 | 58.0 | 78.1 | 63.6 | 65.3 |
| NEO-ov | Qwen3-8B | 67.4 | 70.7 | 46.4 | 69.3 | 63.5 | 51.6 |
🌐 Category 3: Spatial Intelligence
| Model_NAME | Base_LLM_NAME | VSI-Bench | MMSI | Mindcube-tiny | ViewSpatial | SITE | 3DSR | EmbSpatial | SPAR | MMSI-video | Omni_Manual cot | BLINK | MUIRBENCH |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 🔻Vision Language Models (Instruct-2B) | |||||||||||||
| InternVL3.5 | Qwen3-1.7B | 53.8 | 25.6 | 42.1 | 37.9 | 34.8 | 31.4 | 61.5 | 32.4 | 25.9 | 44.4 | 51.3 | 44 |
| Qwen3-VL | Qwen3-1.7B | 53.9 | 27.8 | 34.2 | 36.7 | 35.8 | 47.6 | 69.2 | 34.1 | 25.6 | 36.3 | 53.8 | 47.4 |
| NEO-ov | Qwen3-1.7B | 58.4 | 33.6 | 77.2 | 52.8 | 38.4 | 52.9 | 63.8 | 41.2 | 23.1 | 43.1 | 53.9 | 56.8 |
| 🔻Vision Language Models (Instruct-8B) | |||||||||||||
| InternVL3.5 | Qwen3-8B | 56.3 | 29.1 | 40.4 | 40 | 54.4 | 35.3 | 75.7 | 38.2 | 28 | 47.8 | 59.5 | 55.8 |
| Qwen3-VL | Qwen3-8B | 59.4 | 31.2 | 29.6 | 41.9 | 45.4 | 52.9 | 77.8 | 40.3 | 28.4 | 47 | 69.1 | 64.4 |
| NEO-ov | Qwen3-8B | 64.8 | 41.3 | 90 | 55.2 | 54.3 | 61.7 | 78.8 | 48.8 | 28.7 | 45 | 62.8 | 58.2 |
📊 Demonstration
# Demo
from vlmeval.config import supported_VLM
model = supported_VLM['NEOov-2B-image']()
# Forward Single Image
ret = model.generate(['assets/apple.jpg', 'What is in this image?'])
print(ret) # The image features a red apple with a leaf on it.
# Forward Multiple Images
ret = model.generate(['assets/apple.jpg', 'assets/apple.jpg', 'How many apples are there in the provided images? '])
print(ret) # There are two apples in the provided images.