M2KR Benchmark Datasets

December 19, 2024 · View on GitHub

We release the M2KR and M2KR-CN Benchmark datasets in Huggingface Dataset format.

We pre-process the datasets into a uniform format and write several task-specific prompting instructions for each dataset. The details of the instruction can be found in the paper. The M2KR benchmark contains three types of tasks:

Image to Text (I2T) retrieval

These tasks evaluate the ability of a retriever to find relevant documents associated with an input image.
Component tasks are WIT, IGLUE-en, KVQA, and CC3M.

Question to Text (Q2T) retrieval

This task is based on MSMARCO and is included to assess whether multi-modal retrievers retain their ability in text-only retrieval after any retraining for images.

Image & Question to Text (IQ2T) retrieval

This is the most challenging task which requires joint understanding of questions and images for accurate retrieval. It consists of these subtasks:
OVEN, LLaVA, OKVQA, Infoseek and E-VQA.

We show the dataset statistics in the following table:

Datasets #Examples #Passages
Train Val Test Train Val/Test
I2T Retrieval
WIT 2.8M 20,102 5,120 4.1M 40K
WIT_CN 281k 2,010 512 412k 4K
IGLUE - - 685 - 1K
KVQA 65K 13,365 5,120 16.3K 4,648
KVQA_CN 65K 13,365 5,120 16.3K 4,648
CC3M 595K - - 595K -
CC3M_CN 595K - - 595K -
Q2T Retrieval
MSMARCO 400K 6,980 5,120 8.8M 200K
MSMARCO_CN 99.4k 300 300 107k 5.34K
IQ2T Retrieval
OVEN 339K 20,000 5,120 10K 3,192
OVEN_CN 339K 20,000 5,120 10K 3,192
LLAVA 351K - 5,120 351K 6,006
LLAVA_CN 351K - 5,120 351K 6,006
OKVQA 9K 5,046 5,046 110K 110K
OKVQA_CN 9K 5,046 5,046 110K 110K
Infoseek 100K - 4,708 100K 100K
Infoseek_CN 100K - 4,708 100K 100K
E-VQA 212K 9,852 3,750 50K 50K
E-VQA_CN 212K 9,852 3,750 50K 50K

Huggingface Datasets

We release the M2KR on the huggingface BByrneLab/multi_task_multi_modal_knowledge_retrieval_benchmark_M2KR.

M2KR_CN on the huggingface BByrneLab/multi_task_multi_modal_knowledge_retrieval_benchmark_M2KR_CN

[NEW !] M2KR CN Dataset Detail

In the process of making the Chinese dataset, we use the Qwen2-7B-Instruct model for translation.

CN Dataset Source

For CC3M, E-VQA, KVQA, OVEN, LLAVA, OKVQA and Infoseek, we directly use the data in M2KR to translate and obtain the Chinese datasets.

For WIT, we first downsample WIT in M2KR to 10%, and then translate them to Chinese.

For MSMARCO, we directly use the mMARCO-zh in bge-m3-data

Translate Example

We use Qwen2-7B-Instruct to translate our dataset.

We build an OpenAI-compatible API service with VLLM to do translate, you can find useage in github repository of Qwen2.5

Due to the input length limit of large language model, we need to do some segmentation operations during the translation process.

import openai
openai.api_base = "http://localhost:8000/v1"
openai.api_key = "none"

from openai import ChatCompletion

def split_text_by_length(text, max_length):
    sentences = text.split('.')
    segments = []
    current_segment = ""

    for sentence in sentences:
        if len(current_segment) + len(sentence) + 1 > max_length:  # +1 for the space or punctuation
            segments.append(current_segment.strip())
            current_segment = sentence
        else:
            current_segment += " " + sentence

    if current_segment:
        segments.append(current_segment.strip())

    return segments

def split_text_by_length_line(text, max_length):
    sentences = text.split('\n')
    segments = []
    current_segment = ""

    for sentence in sentences:
        if len(current_segment) + len(sentence) + 1 > max_length:  # +1 for the space or punctuation
            segments.append(current_segment.strip())
            current_segment = sentence
        else:
            current_segment += " " + sentence

    if current_segment:
        segments.append(current_segment.strip())

    return segments

def translate_with_vllm(text, prompt):
    if len(text) < 12000:
        input_messages = [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": prompt.format(text)}
        ]

        chat_response = ChatCompletion.create(
            model="Qwen2-7B-Instruct",
            messages=input_messages,
            stream=False
        )
        return chat_response.choices[0].message.content
    else:
        texts = split_text_by_length(text, 12000)
        translate_result = ''
        for t in texts:
            if len(t) < 8000:
                input_messages = [
                    {"role": "system", "content": "You are a helpful assistant."},
                    {"role": "user", "content": prompt.format(t)}
                ]
                chat_response = ChatCompletion.create(
                    model="Qwen2-7B-Instruct",
                    messages=input_messages,
                    stream=False
                )
                translate_content = chat_response.choices[0].message.content
            else:
                translate_content = ''
                lines = split_text_by_length_line(t, 8000)
                for l in lines:
                    l = l.replace("\"", "")
                    input_messages = [
                    {"role": "system", "content": "You are a helpful assistant."},
                    {"role": "user", "content": prompt.format(t)}
                ]
                chat_response = ChatCompletion.create(
                    model="Qwen2-7B-Instruct",
                    messages=input_messages,
                    stream=False
                )
                translate_content = translate_content + " " + chat_response.choices[0].message.content
            translate_result = translate_result + " " + translate_content
        return translate_result

prompt = "将以下英文内容翻译成中文并仅返回给我翻译的内容：{}"
passage_en = 'olive oil is a healthy ingredient used liberally .'

passage_cn = translate_with_vllm(passage_en, prompt)
print(passage_cn)
# 橄榄油是一种健康的食材，使用量要丰富。

Example Use

The datasets are available for download and use with the Huggingface datasets library.

Datasets

from datasets import load_dataset
# EVQA datasets
EVQA_ds = load_dataset("BByrneLab/multi_task_multi_modal_knowledge_retrieval_benchmark_M2KR", "EVQA_data")
# WIT datasets
WIT_ds = load_dataset("BByrneLab/multi_task_multi_modal_knowledge_retrieval_benchmark_M2KR", "WIT_data")
# EVQA CN datasets
EVQA_CN_ds = load_dataset("BByrneLab/multi_task_multi_modal_knowledge_retrieval_benchmark_M2KR_CN", "EVQA_data")
# WIT CN datasets
WIT_CN_ds = load_dataset("BByrneLab/multi_task_multi_modal_knowledge_retrieval_benchmark_M2KR_CN", "WIT_data")

# ...

Each datasets contains the train/val/test split:

train_ds = WIT_ds['train']
val_ds = WIT_ds['valid']
test_ds = WIT_ds['test']

Passages

# EVQA passages
EVQA_passages = load_dataset("BByrneLab/multi_task_multi_modal_knowledge_retrieval_benchmark_M2KR", "EVQA_passages")
# WIT passages
WIT_passages = load_dataset("BByrneLab/multi_task_multi_modal_knowledge_retrieval_benchmark_M2KR", "WIT_passages")
# EVQA CN passages
EVQA_CN_passages = load_dataset("BByrneLab/multi_task_multi_modal_knowledge_retrieval_benchmark_M2KR", "EVQA_passages")
# WIT CN passages
WIT_CN_passages = load_dataset("BByrneLab/multi_task_multi_modal_knowledge_retrieval_benchmark_M2KR", "WIT_passages")
# ...

Each dataset contains the passages for the train/val/test split:

train_passages = WIT_passages['train_passages']
val_passages = WIT_passages['valid_passages']
test_passages = WIT_passages['test_passages']

Images

In the HF datasets, we only include the image path to the dataset without the root image directory (e.g., train2014/COCO_train2014_000000276336.jpg). The base image directory can be set using the image_root_dir argument in our provided example script to run PreFLMR (i.e., example_use_preflmr.py) for each datasets/tasks. You will need to change the image_root_dir to the correct path to the image directory according to the datasets/tasks.

In general, the image path structure should be <image_root_dir>/<img_path>, where <img_path> is provided with our HF datasets.

├── image_root_dir
│   ├── ...

We release the raw images used in M2KR benchmark, please see the M2kR Benchmark Images

WIT

The WIT dataset images can be downloaded with the instruction from the WIT Github page. The training images can be downloaded from Kaggle at a size around 275 GB. The validation and test images can be downloaded directly from the github page.

The downloaded image directory should contains: <image_id>.jpg after unzipping. Your image_root_dir should be the path to the directory containing the unzipped images.

IGLUE

Following the instruction from the IGLUE Github page, the IGLUE-WIT images can be downloaded from their hosted server. You will only need the WIT-en split images.

The downloaded image directory should contains: <image_id>.jpg after unzipping.

KVQA

The KVQA dataset images can be downloaded from the KVQA Project page at a size around 25GB. You will only need the KVQAimgs.tar.gz file.

The downloaded image directory should contains: <image_id>.jpg after untaring.

CC3M

We use the downsampled 595K version of CC3M from LLaVA. The images can be downloaded from the LLaVA-CC3M-Pretrain-595K. The images can be found in the images.zip in their HF repository.

The downloaded image directory should have the following structure after unzipping: <image_id>.jpg. The image_id starts with GCC_train_.

OVEN

The OVEN dataset images can be downloaded from the OVEN Github page with their provided script.

You will need to download 6 shards of image tar files: shard00-05.tar. The downloaded image directory should have the following structure after unzipping: 00/<image_id>.jpg, 01/<image_id>.jpg, ..., 05/<image_id>.jpg.

├── image_root_dir
│   ├── 00
│   │   ├── oven_00xxxxx.jpg
│   │   ├── ...
│   ├── 01
│   │   ├── oven_01xxxxx.jpg
│   │   ├── ...
│   ├── ...
│   ├── 05
│   │   ├── oven_05xxxxx.jpg
│   │   ├── ...

LLaVA

The LLaVA-Instruct-150K images are from MSCOCO, which can be downloaded here: train2014. You may refer to the MSCOCO website.

The downloaded image directory should have the following structure after unzipping: train2014/<image_id>.jpg

|── image_root_dir
│   ├── train2014
│   │   ├── COCO_train2014_000000276336.jpg
│   │   ├── ....

OKVQA

The preparation of the OKVQA dataset images can be found in the OKVQA Project page. You will need to downlod the train2014 and val2014 images from the MSCOCO website.

The downloaded image directory should have the following structure after unzipping: train2014/<image_id>.jpg, val2014/<image_id>.jpg

|── image_root_dir
│   ├── train2014
│   │   ├── ...
│   ├── val2014
│   │   ├── ...

Infoseek

Infoseek is obtained from downsampling of the OVEN dataset.
You may use the full OVEN images for Infoseek. However, the img_path provided in our HF removes the 00/, 01/, ..., 05/ prefix from the OVEN images. You may create a folder that contains all the OVEN images with symlinks.

E-VQA

To prepare the images for E-VQA, please refer to the E-VQA Github page. You will need to download the iNaturalist 2021 and Google Landmarks Dataset V2 datasets.

You may expect the following structure after unzipping the downloaded images:

|── image_root_dir
│   ├── inat
│   │   ├── train
│   │   │   ├── ...
│   │   ├── val
│   │   │   ├── ...
│   ├── google-landmark
│   │   ├── train
│   │   │   ├── ...

Reproduce PreFLMR results

To reproduce the PreFLMR results, you can use the M2KR HF datasets and the PreFLMR models. You will need to change the image_root_dir to the correct path to the image directory.

Evaluate the PreFLMR models on a single dataset

python example_use_preflmr.py \
            --use_gpu --run_indexing \
            --index_root_path "." \
            --index_name EVQA_PreFLMR_ViT-G \
            --experiment_name EVQA \
            --indexing_batch_size 64 \
            --image_root_dir /rds/project/rds-hirYTW1FQIw/shared_space/vqa_data/KBVQA_data/EVQA/eval_image/ \
            --dataset_hf_path BByrneLab/multi_task_multi_modal_knowledge_retrieval_benchmark_M2KR \
            --dataset EVQA \
            --use_split test \
            --nbits 8 \
            --Ks 1 5 10 20 50 100 500 \
            --checkpoint_path LinWeizheDragon/PreFLMR_ViT-G \
            --image_processor_name laion/CLIP-ViT-bigG-14-laion2B-39B-b160k \
            --query_batch_size 8 \
            --compute_pseudo_recall \

By changing the --dataset, --experiment_name and image_root_dir, you can reproduce the results for different datasets in the PreFLMR paper.

Evaluate the PreFLMR models on all M2KR benchmarks

Change the image root paths in examples/evaluate_all.sh and execute:

cd examples
bash evaluate_all.sh

Obtain the report by:

python report.py

Ideally, you will obtain the following report:

ModelWIT Recall@10IGLUE Recall@1KVQA Recall@5MSMARCO Recall@5OVEN Recall@5LLaVA Recall@1EVQA Recall@5EVQA Pseudo Recall@5OKVQA Recall@5OKVQA Pseudo Recall@5Infoseek Recall@5Infoseek Pseudo Recall@5
PreFLMR_ViT-G0.6190.7180.4190.7830.6430.7260.6250.7210.3020.6740.3920.577
PreFLMR_ViT-L0.6050.6990.4400.7790.6080.7290.6090.7080.3140.6900.3740.578
PreFLMR_ViT-B0.4270.5740.2940.7860.4680.6730.5500.6630.2720.6580.2600.496