Lightning Pose development roadmap

General enhancements

• introduce jaxtyping (#407)
• multi-GPU training for unsupervised models (#207)
• multi-GPU training for supervised models (#206)
• introduce visibility column in labeled data to match COCO format; treat occluded and unlabeled points differently (#440)
• support for inference with mixed precision
• read COCO-style JSON label files (need to address visibility handling first)
• export predictions in COCO-style JSON format
• support training across multiple datasets; see #284

Video reading enhancements

• look into using pynvvideocodec for accelerated inference and Windows support
• add OpenCV video reader option for inference for native Windows compatability

Losses and backbones

• incorporate transformer backbones (#84, #106)
• compute non-temporal unsupervised losses on labeled data

Multi-view support for non-mirrored setups

• unsupervised losses for multi-view (#187)
• context frames for multi-view (#126)
• implement supervised datasets/dataloaders that work with multiple views (#115)

Single-view dynamic crop (small animals in large frames)

• context frames for dynamic crop (#250)
• unsupervised losses for dynamic crop (#250)
• implement dynamic cropping pipeline with detector model and pose estimator (#250)
• split CLI crop command into create_bbox and smooth_bbox commands for modularity (#420)

Multi-view dynamic crop

• perform view-specific dynamic cropping, re-assemble views after pose estimation stage
• context frames for multi-view dynamic crop
• unsupervised losses for multi-view dynamic crop

Multi-animal

• single-view, supervised
• single-view, context
• single-view, unsupervised losses
• multi-view, supervised
• multi-view, context
• multi-view, unsupervised losses