initialize model

Oncology FM Evaluation Framework by kaiko.ai

Installation • How To Use • Quick Start • Documentation • Datasets • Leaderboard
Contribute • Acknowledgements

eva is an evaluation framework for oncology foundation models (FMs) by kaiko.ai. Check out the documentation for more information.

Highlights:

• 🔬 Standardized benchmarking for oncology foundation models
• 🧠 Supports patch-level/slide-level/scan-level classification, 2D/3D semantic segmentation, and VQA tasks
• ⚡ Offline + online evaluation modes
• 📦 Built-in support for popular medical datasets and models
• 🔁 Robust evaluation with multi-run statistics
• 🧩 Fully configurable via YAML or Python

🛠 Installation

Simple installation from PyPI:

# to install the core version only
pip install kaiko-eva

# to install the expanded `vision` version
pip install 'kaiko-eva[vision]'

# to install the expanded `language` version
pip install 'kaiko-eva[language]'

# to install the expanded `multimodal` version
pip install 'kaiko-eva[multimodal]'

# to install everything
pip install 'kaiko-eva[all]'

To install the latest version of the main branch:

pip install "kaiko-eva[all] @ git+https://github.com/kaiko-ai/eva.git"

You can verify that the installation was successful by executing:

eva --version

🧩 How To Use

eva can be used directly from the terminal as a CLI tool as follows:

eva {fit,predict,predict_fit} --config url/or/path/to/the/config.yaml 

eva uses jsonargparse to make it easily configurable by automatically generating command line interfaces (CLIs), which allows to call any Python object from the command line. Moreover, the configuration structure is always in sync with the code. Thus, eva can be used either directly from Python or as a CLI tool (recommended).

For more information, please refer to the documentation.

# main.py
# execute with: `python main.py`

from torch import nn

from eva import core
from eva.vision import datasets, transforms

# initialize trainer
trainer = core.Trainer(max_steps=100)

# initialize model
model = core.HeadModule(
  backbone=nn.Flatten(),
  head=nn.Linear(150528, 4),
  criterion=nn.CrossEntropyLoss(),
)

# initialize data
data = core.DataModule(
  datasets=core.DatasetsSchema(
    train=datasets.BACH(
      root="data/bach",
      split="train",
      download=True,
      transforms=transforms.ResizeAndCrop(),
    ),
  ),
  dataloaders=core.DataloadersSchema(
    train=core.DataLoader(batch_size=32),
  ),
)

# perform fit
pipeline = core.Interface()
pipeline.fit(trainer, model=model, data=data)

# main.yaml
# execute with: `eva fit --config main.yaml`

---
trainer:
  class_path: eva.Trainer
  init_args:
    max_steps: 100
model:
  class_path: eva.HeadModule
  init_args:
    backbone: torch.nn.Flatten
    head:
      class_path: torch.nn.Linear
      init_args:
        in_features: 150528
        out_features: 4
    criterion: torch.nn.CrossEntropyLoss
data:
  class_path: eva.DataModule
  init_args:
    datasets:
      train:
        class_path: eva.vision.datasets.BACH
        init_args:
          root: ./data/bach
          split: train
          download: true
          transforms: eva.vision.transforms.ResizeAndCrop
    dataloaders:
      train:
        batch_size: 32

⚡️ Quick Start

Offline classification DINO ViT-S/16 on the BACH dataset:

# set the model architecture
DOWNLOAD_DATA=true \
MODEL_NAME=universal/vit_small_patch16_224_dino \
\
# execute the offline evaluation pipeline with the BACH dataset config
eva predict_fit \
    --config https://raw.githubusercontent.com/kaiko-ai/ \
             eva/main/configs/vision/pathology/ \
             offline/classification/bach.yaml

Online segmentation of DINO ViT-S/16 on the MoNuSAC dataset with the ConvDecoderWithImage decoder:

# define the model backbone
DOWNLOAD_DATA=true \
MODEL_NAME=universal/vit_small_patch16_224_dino \
\
# execute online segmentation training for MoNuSAC Dataset
eva fit \
    --config https://raw.githubusercontent.com/kaiko-ai/ \
             eva/main/configs/vision/pathology/ \
             online/segmentation/monusac.yaml

The results of 5 different runs will be saved to ./logs by default, or to OUTPUT_ROOT if specified. For more examples, take a look at the configs and tutorials.

🏆 Leaderboards

The following table shows the FMs we have evaluated with eva. For more detailed information about the evaluation process, please refer to our documentation.

Pathology

Radiology

🤝 Contributing

eva is an open source project and welcomes contributions of all kinds. Please checkout the developer and contributing guide for help on how to do so.

All contributors must follow the code of conduct.

⚙️ Acknowledgements

Our codebase is built using multiple opensource contributions

📖 Citation

If you find this repository useful, please consider giving a star ⭐ and adding the following citation:

@inproceedings{kaiko.ai2024eva,
    title={eva: Evaluation framework for pathology foundation models},
    author={kaiko.ai and Ioannis Gatopoulos and Nicolas K{\"a}nzig and Roman Moser and Sebastian Ot{\'a}lora},
    booktitle={Medical Imaging with Deep Learning},
    year={2024},
    url={https://openreview.net/forum?id=FNBQOPj18N}
}