PathBench

May 25, 2025 · View on GitHub

This is the official repo for PathBench: Advancing the Benchmark of Large Multimodal Models for Pathology Image Understanding at Patch and Whole Slide Level (technical report will be released soon !)

Abstract

Rapid advancements in large multimodal models (LMMs) have significantly enhanced their applications in pathology, particularly in image classification, pathology image description, and whole slide image (WSI) classification. In pathology, WSIs represent gigapixel-scale images composed of thousands of image patches. Therefore, both patch-level and WSI-level evaluations are essential and inherently interconnected for assessing LMM capabilities. In this work, we propose PathBench, which comprises three subsets at both patch and WSI levels, to refine and enhance the validation of LMMs. At the patch-level, evaluations using existing multi-choice Q&A datasets reveal that some LMMs can predict answers without genuine image analysis. To address this, we introduce PatchVQA, a large-scale visual question answering (VQA) dataset containing 5,382 images and 6,335 multiple-choice questions designed with distractor options to prevent shortcut learning. These new questions are rigorously validated by professional pathologists to ensure reliable model assessments. At the WSI-level, current efforts primarily focus on image classification tasks and lack diverse validation datasets for multimodal models. To address this, we generate a detailed WSI report dataset through an innovative approach that integrates detailed patch descriptions generated by foundational models into comprehensive WSI reports. These are then combined with physician-written reports corresponding to TCGA WSIs, resulting in WSICap, a detailed report dataset containing 7,000 samples. Based on WSICap, we further develop a WSI-level VQA dataset, WSIVQA, to serve as a validation set for WSI LMMs. Using these PathBench subsets, we conduct extensive experiments to benchmark the performance of state-of-the-art LMMs at both the patch and WSI levels.

Data Acquisition

For PatchVQA subset:

Access images can from PathMMU and the questions from PatchVQA at PatchVQA.

Example of the PatchVQA:

 {
    "No": "1",
    "img": "3a147c020474daa571a9b2df8405b9d33ce16913dab323d05431585ce073ce50.png",
    "question": "What is the most likely morphological classification of the bacteria present in the micrograph?",
    "options": [
        "A) Spirochetes",
        "B) Bacilli",
        "C) Vibrio",
        "D) Filamentous bacteria",
        "E) Cocci",
        "F) Coccobacilli"
    ],
    "answer": "F) Coccobacilli"
}

For WSICap subset:

Step1:

Access and download the JSON file containing WSI IDs, file IDs, and processed reports from the WSICap.

Data example:

{
    "wsi_id": "TCGA-A3-A8OV-01Z-00-DX1.ABA46896-E96D-4208-B0E3-4712F04175D4",
    "file_id": "d343ae83-8a87-4d10-8db6-d022087bf03f",
    "report": "The examination of the renal tissue reveals a primary diagnosis of **clear cell renal cell carcinoma (CCRCC), Fuhrman nuclear grade 2**. The tumor is confined to the kidney but extends focally to the renal parenchymal margin, indicating a positive margin status (R1). Lymphovascular invasion, sarcomatoid features, and tumor necrosis are not identified. \n\nFurther detailed histopathological analysis shows preserved renal architecture with some tubules exhibiting cellular stress markers such as eosinophilic cytoplasm, prominent nuclei, and occasional vacuolated cytoplasm. Chronic interstitial fibrosis and tubular atrophy are noted, consistent with chronic kidney disease. Tubular changes, including nuclear pleomorphism and hyperchromatic nuclei, suggest potential early neoplastic changes.\n\nInterstitial inflammation and fibrosis point towards chronic pyelonephritis or interstitial nephritis. Glomerular pathology includes hypercellularity, capillary loop thickening, and mesangial expansion, indicative of proliferative glomerulonephritis or focal segmental glomerulosclerosis. Vascular changes like arteriosclerosis and vascular calcifications suggest underlying hypertension or cardiovascular issues.\n\nNeoplastic processes other than CCRCC, including oncocytomas and potential renal spindle cell neoplasms, are indicated by cellular pleomorphism and disorganized architecture. The combined observations highlight significant chronic kidney disease, tubular injury, and neoplastic processes."
}

Step2:

Employ the GDC Data Transfer Tool to download the whole-slide images (.svs files) referenced in the JSON file. Detailed instructions for using this tool can be found on the GDC's documentation page: https://docs.gdc.cancer.gov/Data_Transfer_Tool/Users_Guide/Getting_Started/.

A simple approach is to use the file_id field and download the file using gdc-client download <file_id>.

For WSIVQA subset

Step1:

Access and download the JSON file containing WSI IDs, file IDs, and processed reports from the WSICap.

Data example:

{
    "wsi_id": "TCGA-BQ-5892-01Z-00-DX1.f3a4da33-2b60-4b8e-b623-7b6237106a4c",
    "file_id": "0f54bf67-2014-49f3-b14e-e7783e01d900",
    "QAs": [
        {
            "question": "Which staining result is observed in the high-grade unclassified renal cell carcinoma?",
            "options": [
                "Positive for CD10 and CA-IX",
                "Negative for CD10 and CA-IX",
                "Positive for CD20 and CA-IX",
                "Negative for CD20 and CA-IX"
            ],
            "answer_index": "0"
        },
        {
            "question": "What features are present in the renal cell carcinoma?",
            "options": [
                "Papillary and solid features only",
                "Clear cell and papillary features only",
                "Solid, papillary, and clear cell features",
                "Solid and clear cell features only"
            ],
            "answer_index": "2"
        },
        {
            "question": "What is the condition of the renal vein in this case?",
            "options": [
                "No invasion",
                "Invasion present"
            ],
            "answer_index": "0"
        }
    ],
    "MIZero_CLIP_prompts": [
        {
            "prompts": [
                "Prompt A: The staining result observed in the high-grade unclassified renal cell carcinoma is positive for CD10 and CA-IX.",
                "Prompt B: The staining result observed in the high-grade unclassified renal cell carcinoma is negative for CD10 and CA-IX.",
                "Prompt C: The staining result observed in the high-grade unclassified renal cell carcinoma is positive for CD20 and CA-IX.",
                "Prompt D: The staining result observed in the high-grade unclassified renal cell carcinoma is negative for CD20 and CA-IX."
            ],
            "answer_index": "0"
        },
        {
            "prompts": [
                "Prompt A: The renal cell carcinoma shows papillary and solid features only.",
                "Prompt B: The renal cell carcinoma shows clear cell and papillary features only.",
                "Prompt C: The renal cell carcinoma shows solid, papillary, and clear cell features."
            ],
            "answer_index": "2"
        },
        {
            "prompts": [
                "Prompt A: The renal vein shows no invasion.",
                "Prompt B: The renal vein shows invasion."
            ],
            "answer_index": "0"
        }
    ]
}

where answer_index represents the index of the correct option within the options array, and MIZero_CLIP_prompts contains the question prompts that we have converted into a format suitable for MI-Zero to compute image similarity using CLIP.

Step2:

A simple approach is to use the file_id field and download the file using gdc-client download <file_id>.