The All-Seeing Project [Paper][Model][Dataset]

Introduction

We present the All-Seeing Project with:

All-Seeing 1B (AS-1B) dataset: we propose a new large-scale dataset (AS-1B) for open-world panoptic visual recognition and understanding, using an economical semi-automatic data engine that combines the power of off-the-shelf vision/language models and human feedback.

All-Seeing Model (ASM): we develop a unified vision-language foundation model (ASM) for open-world panoptic visual recognition and understanding. Aligning with LLMs, our ASM supports versatile image-text retrieval and generation tasks, demonstrating impressive zero-shot capability.

Figure 1: Overview and comparison of our All-Seeing project with other popular large foundation models.

Dataset Overview

AS-1B with over 1 billion regions annotated with semantic tags, question-answering pairs, and detailed captions. It covers a wide range of 3.5 million common and rare concepts in the real world, and has 132.2 billion tokens that describe the concepts and their attributes.

Some examples

Model Architecture

The All-Seeing model (ASM) is a unified framework for panoptic visual recognition and understanding, including image/region-text retrieval, image/region recognition, captioning, and question-answering.

Installation

pip install torch==2.0.0
pip install transformers==4.28.0
pip install pycocoevalcap
pip install mmeval==0.2.1

Model Zoo

Model	Download	Note
All-Seeing-Model-Pretrain	🤗 HF link	a unified vision-language foundation model
All-Seeing-Model-FT	🤗 HF link	a vision-language foundation model for region-level qa

Testing

For evaluation of region captioning, please download the evaluation data annotations and put them in ./data as the following structure.

The images can be downloaded from:

• COCO
• LVIS
• VG
• Flickr30K
• NoCaps

├── coco
│   ├── val2014
│   ├── val2017
│   ├── train2017
│   └── annotations
├── lvis
│   ├── val2017
│   └── annotations
├── flickr30k
│   └── images
├── nocaps
│   └── val
├── vg
│   ├── VG_100K
│   └── VG_100K_2
├── coco_karpathy_val.json
├── flickr30k_karpathy_test.json
├── nocaps_val.json
├── refcocog_test_coco_format.json
├── refcocog_val_coco_format.json
└── vg_test_coco_format.json

• Image/Region Captioning

# supported dataset: coco_caption, flickr30k, nocaps, vg, refcocog_val, refcocog_test
sh scripts/eval_caption.sh OpenGVLab/All-Seeing-Model-Pretrain ${DATASET_NAME}

• Region Recognition

# supported dataset: coco, lvis
sh scripts/eval_region_recognition.sh OpenGVLab/All-Seeing-Model-Pretrain ${DATASET_NAME}