LightlyTrain - SOTA Pretraining, Fine-tuning and Distillation

Train LTDETR detection models with DINOv2 or DINOv3 backbones.

COCO Results

*Picodet models are in preview and we report preliminary results.

Models are trained on the COCO 2017 dataset and evaluated on the validation set with single-scale testing. Latency is measured with TensorRT on a NVIDIA T4 GPU with batch size 1. All models are optimized using tensorrt==10.13.3.9.

Usage

import lightly_train

if __name__ == "__main__":
    # Train an object detection model with a DINOv3 backbone
    lightly_train.train_object_detection(
        out="out/my_experiment",
        model="dinov3/vitt16-ltdetr-coco",
        data={
            "path": "my_data_dir",
            "train": "images/train",
            "val": "images/val",
            "names": {
                0: "person",
                1: "bicycle",
                2: "car",
                # ...
            },
        },
    )

    # Load model and run inference
    model = lightly_train.load_model("out/my_experiment/exported_models/exported_best.pt")
    # Or use one of the models provided by LightlyTrain
    # model = lightly_train.load_model("dinov3/vitt16-ltdetr-coco")
    results = model.predict("image.jpg")
    results["labels"]   # Class labels, tensor of shape (num_boxes,)
    results["bboxes"]   # Bounding boxes in (xmin, ymin, xmax, ymax) absolute pixel
                        # coordinates of the original image. Tensor of shape (num_boxes, 4).
    results["scores"]   # Confidence scores, tensor of shape (num_boxes,)

Train state-of-the-art panoptic segmentation models with DINOv3 backbones using the EoMT method from CVPR 2025.

COCO Results

Tiny models are trained for 48 epochs, small and base models for 24 epochs and large models for 12 epochs on the COCO 2017 dataset and evaluated on the validation set with single-scale testing. Avg. Latency is measured on a single NVIDIA T4 GPU with batch size 1. All models are optimized using torch.compile.

Usage

import lightly_train

if __name__ == "__main__":
    # Train an panoptic segmentation model with a DINOv3 backbone
    lightly_train.train_panoptic_segmentation(
        out="out/my_experiment",
        model="dinov3/vitb16-eomt-panoptic-coco",
        data={
            "train": {
                "images": "images/train",
                "masks": "annotations/train",
                "annotations": "annotations/train.json",
            },
            "val": {
                "images": "images/val",
                "masks": "annotations/val",
                "annotations": "annotations/val.json",
            },
        },
    )

    model = lightly_train.load_model("out/my_experiment/exported_models/exported_best.pt")
    results = model.predict("image.jpg")
    results["masks"]    # Masks with (class_label, segment_id) for each pixel, tensor of
                        # shape (height, width, 2). Height and width correspond to the
                        # original image size.
    results["segment_ids"]    # Segment ids, tensor of shape (num_segments,).
    results["scores"]   # Confidence scores, tensor of shape (num_segments,)

Train state-of-the-art instance segmentation models with DINOv3 backbones using the EoMT method from CVPR 2025.

COCO Results

Tiny models are trained for 48 epochs, while all other models are trained for 12 epochs on the COCO 2017 dataset and evaluated on the validation set with single-scale testing. Average latency is measured on a single NVIDIA T4 GPU with batch size 1. All models are optimized using torch.compile.

Usage

import lightly_train

if __name__ == "__main__":
    # Train an instance segmentation model with a DINOv3 backbone
    lightly_train.train_instance_segmentation(
        out="out/my_experiment",
        model="dinov3/vitb16-eomt-inst-coco",
        data={
            "path": "my_data_dir",
            "train": "images/train",
            "val": "images/val",
            "names": {
                0: "background",
                1: "vehicle",
                2: "pedestrian",
                # ...
            },
        },
    )

    model = lightly_train.load_model("out/my_experiment/exported_models/exported_best.pt")
    results = model.predict("image.jpg")
    results["labels"]   # Class labels, tensor of shape (num_instances,)
    results["masks"]    # Binary masks, tensor of shape (num_instances, height, width).
                        # Height and width correspond to the original image size.
    results["scores"]   # Confidence scores, tensor of shape (num_instances,)

Train state-of-the-art semantic segmentation models with DINOv2 or DINOv3 backbones using the EoMT method from CVPR 2025.

COCO-Stuff Results

Models are trained for 12 epochs with num_queries=200 on the COCO-Stuff dataset and evaluated on the validation set with single-scale testing. Average latency is measured on a single NVIDIA T4 GPU with batch size 1. All models optimized using torch.compile.

Cityscapes Results

Average latency is measured on a single NVIDIA T4 GPU with batch size 1. All models are optimized using torch.compile.

Usage

import lightly_train

if __name__ == "__main__":
    # Train a semantic segmentation model with a DINOv3 backbone
    lightly_train.train_semantic_segmentation(
        out="out/my_experiment",
        model="dinov3/vits16-eomt",
        data={
            "train": {
                "images": "my_data_dir/train/images",
                "masks": "my_data_dir/train/masks",
            },
            "val": {
                "images": "my_data_dir/val/images",
                "masks": "my_data_dir/val/masks",
            },
            "classes": {
                0: "background",
                1: "road",
                2: "building",
                # ...
            },
        },
    )

    # Load model and run inference
    model = lightly_train.load_model("out/my_experiment/exported_models/exported_best.pt")
    # Or use one of the models provided by LightlyTrain
    # model = lightly_train.load_model("dinov3/vits16-eomt")
    masks = model.predict("image.jpg")
    # Masks is a tensor of shape (height, width) with class labels as values.
    # It has the same height and width as the input image.

Train multiclass or multilabel image classification models with any backbone.

Usage

import lightly_train

if __name__ == "__main__":
    # Train an image classification model with a DINOv3 backbone
    lightly_train.train_image_classification(
        out="out/my_experiment",
        model="dinov3/vitt16",
        data={
            "train": "my_data_dir/train/",
            "val": "my_data_dir/val/",
            "classes": {
                0: "cat",
                1: "car",
                2: "dog",
                # ...
            },
        },
    )

    model = lightly_train.load_model("out/my_experiment/exported_models/exported_best.pt")
    results = model.predict("image.jpg", topk=1, threshold=0.5)
    results["labels"]   # Class labels, tensor of shape (topk,)
    results["scores"]   # Confidence scores, tensor of shape (topk,)

Pretrain any model architecture with unlabeled data by distilling the knowledge from DINOv2 or DINOv3 foundation models into your model. On the COCO dataset, YOLOv8-s models pretrained with LightlyTrain achieve high performance across all tested label fractions. These improvements hold for other architectures like YOLOv11, RT-DETR, and Faster R-CNN. See our announcement post for more benchmarks and details.

Usage

import lightly_train

if __name__ == "__main__":
    # Distill the knowledge from a DINOv3 teacher into a YOLOv8 model
    lightly_train.pretrain(
        out="out/my_experiment",
        data="my_data_dir",
        model="ultralytics/yolov8s",
        method="distillation",
        method_args={
            "teacher": "dinov3/vitb16",
        },
    )

    # Load model for fine-tuning
    model = YOLO("out/my_experiment/exported_models/exported_last.pt")
    model.train(data="coco8.yaml")

With LightlyTrain you can train your very own foundation model like DINOv2 on your data.

ImageNet-1K Results

Models are pretrained on ImageNet-1k for 100 epochs and evaluated with a k-NN classifier on the ImageNet validation set.

Usage

import lightly_train

if __name__ == "__main__":
    # Pretrain a DINOv2 vision foundation model
    lightly_train.pretrain(
        out="out/my_experiment",
        data="my_data_dir",
        model="dinov2/vitb14",
        method="dinov2",
    )

LightlyTrain provides simple commands to autolabel your unlabeled data using DINOv2 or DINOv3 pretrained models. This allows you to efficiently boost performance of your smaller models by leveraging all your unlabeled images.

ADE20K Results

The better results with auto-labeling were achieved by fine-tuning a ViT-H+ on the ADE20K dataset, which reaches 0.595 validation mIoU. This model was then used to autolabel 100k images from the SUN397 dataset. Using these labels, we subsequently fine-tuned the smaller models, and then used the ADE20k dataset for validation.

Usage

import lightly_train

if __name__ == "__main__":
    # Autolabel your data with a DINOv3 semantic segmentation model
    lightly_train.predict_semantic_segmentation(
        out="out/my_autolabeled_data",
        data="my_data_dir",
        model="dinov3/vitb16-eomt-coco",
        # Or use one of your own model checkpoints
        # model="out/my_experiment/exported_models/exported_best.pt",
    )

    # The autolabeled masks will be saved in this format:
    # out/my_autolabeled_data
    # ├── <image name>.png
    # ├── <image name>.png
    # └── …