README.md

Models Genesis is a self-supervised learning framework that pre-trains 3D models on unlabeled medical images. It learns transferable visual representations from CT and MRI volumes with no manual annotation, and those representations fine-tune for downstream segmentation and classification.

This line of work is described in the literature as self-supervised learning, representation learning, feature learning, and transfer learning. Models Genesis is an early example of what would later be called a foundation model for medical imaging.

Pre-training uses image restoration proxy tasks: non-linear intensity transformation, local pixel shuffling, out-painting, and in-painting. The model learns anatomy by reconstructing deliberately distorted volumes. In the self-supervised learning literature this places Models Genesis in the generative or restorative family, as distinct from contrastive methods.

The original name, Generic Autodidactic Models, describes the same idea: models created ex nihilo (with no manual labeling), self-taught (learned by self-supervision), and generic (serving as source models for application-specific target models). Models Genesis is a primary source of transfer learning for 3D medical imaging, particularly where annotated data is limited.

Paper

This repository provides the official implementation of training Models Genesis as well as the usage of the pre-trained Models Genesis in the following paper:

Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis
Zongwei Zhou¹, Vatsal Sodha¹, Md Mahfuzur Rahman Siddiquee¹,
Ruibin Feng¹, Nima Tajbakhsh¹, Michael B. Gotway², and Jianming Liang^1
1 Arizona State University, ² Mayo Clinic
International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), 2019
Young Scientist Award
paper | code | slides | poster | talk (YouTube, YouKu) | blog

Models Genesis
Zongwei Zhou¹, Vatsal Sodha¹, Jiaxuan Pang¹, Michael B. Gotway², and Jianming Liang^1
1 Arizona State University, ² Mayo Clinic
Medical Image Analysis (MedIA)
MedIA Best Paper Award
paper | code | slides | graphical abstract

Available implementation

• keras/
• pytorch/

★ Models Genesis, incorporated with nnU-Net, ranked #1 in liver/tumor and hippocampus segmentation on the Medical Segmentation Decathlon (as of 2021; the live leaderboard now loads results client-side and may not display historical standings).

• competition/

Major results from our work

1. Models Genesis outperform 3D models trained from scratch
2. Models Genesis top any 2D approaches, including ImageNet models and degraded 2D Models Genesis
3. Models Genesis (2D) offer performances equivalent to supervised pre-trained models

The par plots presented below are produced by Matlab code in figures/plotsuperbar.m and the helper functions in figures/superbar. Credit to superbar by Scott Lowe.

Note that learning from scratch simply in 3D may not necessarily yield performance better than ImageNet-based transfer learning in 2D

Citation

If you use this code or use our pre-trained weights for your research, please cite our papers:

@InProceedings{zhou2019models,
  author="Zhou, Zongwei and Sodha, Vatsal and Rahman Siddiquee, Md Mahfuzur and Feng, Ruibin and Tajbakhsh, Nima and Gotway, Michael B. and Liang, Jianming",
  title="Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis",
  booktitle="Medical Image Computing and Computer Assisted Intervention -- MICCAI 2019",
  year="2019",
  publisher="Springer International Publishing",
  address="Cham",
  pages="384--393",
  isbn="978-3-030-32251-9",
  url="https://link.springer.com/chapter/10.1007/978-3-030-32251-9_42"
}

@article{zhou2021models,
  title="Models Genesis",
  author="Zhou, Zongwei and Sodha, Vatsal and Pang, Jiaxuan and Gotway, Michael B and Liang, Jianming",
  journal="Medical Image Analysis",
  volume = "67",
  pages = "101840",
  year = "2021",
  issn = "1361-8415",
  doi = "10.1016/j.media.2020.101840",
  url = "https://www.sciencedirect.com/science/article/abs/pii/S1361841520302048",
}

@phdthesis{zhou2021towards,
  title={Towards Annotation-Efficient Deep Learning for Computer-Aided Diagnosis},
  author={Zhou, Zongwei},
  year={2021},
  school={Arizona State University}
}

Acknowledgement

This research has been supported partially by ASU and Mayo Clinic through a Seed Grant and an Innovation Grant, and partially by the National Institutes of Health (NIH) under Award Number R01HL128785. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. This work has utilized the GPUs provided partially by the ASU Research Computing and partially by the Extreme Science and Engineering Discovery Environment (XSEDE) funded by the National Science Foundation (NSF) under grant number ACI-1548562. This is a patent-pending technology.