OccamVTS: Distilling Vision Models to 1% Parameters for Time Series Forecasting

August 1, 2025 · View on GitHub

framework

Datasets

Download the pre-processed datasets from the THUML Time-Series-Library GitHub repository (https://github.com/thuml/Time-Series-Library). Place all downloaded files in the ./dataset folder at the project root.

Requirements

To set up the environment, install Python 3.8 with Pytorch 1.4.4. Use the following commands for convenience:

conda create -n OccamVTS python=3.8
conda activate OccamVTS
pip install -r requirements.txt

Project Structure

OccamVTS/
│───README.md                 # Project documentation
│───requirements.txt          # Python dependencies
│───run.py                    # Main entry point for training and testing
│───dataset/                  # Dataset directory
│   │───ETT/                  # ETT datasets
│   │───Weather/              # Weather dataset
│   │───Electricity/          # Electricity dataset
│   │───Traffic/              # Traffic dataset
│   └───...
│───scripts/                  # Training and evaluation scripts
│   │───full-shot/           # Full-shot learning scripts
│   │───few-shot/            # Few-shot learning scripts
│   │───zero-shot/           # Zero-shot learning scripts
│───src/                      # Source code
│   │───OccamVTS/             # OccamVTS model implementation
│   │   │───model.py         # Main model architecture
│   │   │───vlm_custom.py    # Custom LVM implementations
│   │   │───vlm_manager.py   # LVM manager for different types
│   │   └───distill.py       # Knowledge distillation framework
│───utils/                    # Utility functions
│───layers/                   # Custom layers
│───exp/                      # Experiment configurations
│───logs/                     # Training logs
└───...

Quick Start

Run the following scripts for different forecasting tasks:

# Few-shot Forecasting
bash ./scripts/few_shot_forecast/OccamVTS_ETTh1.sh

# Zero-shot Forecasting
bash ./scripts/zero_shot_forecast/ETT_script/OccamVTS_ETTh1_ETTh2.sh

# Full-shot Forecasting
bash ./scripts/long_term_forecast/OccamVTS_ETTh1.sh

Note:

  • Make sure you have downloaded the datasets and placed them in the correct directory
  • The default parameters provided in scripts are a good starting point, but you need to adjust them based on your specific dataset and requirements.

Here are the basic parameter configurations and tuning suggestions:

Model Architecture Parameters

  • image_size: Input image size (default: 56)
    • Balances computational efficiency and visual information preservation
    • Try 112 for more detailed visual patterns
    • Try 28 for faster processing with less detail
  • d_model: Dimension of hidden model (default: 128) [Most Important]
    • Core backbone dimension that affects model capacity
    • For complex patterns: try 256 or 512
    • For simpler data: 64 or 128 is sufficient
  • d_fusion: Dimension of fusion layer (default: 256)
    • Controls multimodal feature integration capacity
    • Increase for more complex multimodal interactions
    • Should be proportional to d_model
  • num_workers: Number of data loader workers (default: 32)
    • Adjust based on your CPU cores
    • Typically set to 2-4x number of GPUs
    • Reduce if CPU becomes bottleneck
  • e_layers: Number of encoder layers (default: 2)
    • More layers for complex temporal patterns
    • Fewer layers for faster training
    • Range: 1-4 layers typically sufficient
  • d_layers: Number of decoder layers (default: 1)
    • Usually kept smaller than encoder
    • Increase for more complex output mappings
  • dropout: Dropout rate (default: 0.1)
    • Increase if observing overfitting
    • Range: 0.1-0.3 for most cases
    • Higher values may hurt performance on large datasets
  • memory_size: Maximum capacity of memory bank (default: 100)
    • Stores historical patterns for retrieval
    • Increase for datasets with many recurring patterns
    • Balance between memory usage and pattern diversity
  • top_k: Top-K selection for memory retrieval (default: 5)
    • Number of most relevant memories to retrieve
    • Increase for more diverse pattern matching
    • Keep small (3-10) for efficiency
  • teacher_hidden_size: Hidden size of teacher model
    • CLIP: 512
    • MAE-Base: 768
    • MAE-Large: 1024
    • MAE-Huge: 1280
    • EfficientNet-B3: 1536
    • ResNet101: 2048
  • student_hidden_size: Hidden size of student model (default: 128)
    • Applies to Tiny-ViT, EfficientNet-B0, MobileNet-V3
    • Keep smaller than teacher for effective distillation

Training Parameters

  • batch_size: Training batch size (default: 32)
    • Increase if GPU memory allows (64, 128)
    • Decrease if encountering OOM errors
    • Larger batches generally improve stability
  • learning_rate: Initial learning rate (default: 0.001)
    • Uses AdamW optimizer
    • Try range: 0.0001 - 0.01
    • Start with 0.001 as moderate baseline
    • Reduce if training unstable
  • train_epochs: Number of training epochs (default: 10)
    • Maximum epochs before stopping
    • May stop earlier due to early stopping
    • Increase for complex datasets
  • patience: Early stopping patience (default: 5)
    • Epochs to wait before stopping if no improvement
    • Increase for noisy datasets
    • Decrease for faster experimentation
  • loss: Loss function (default: MSE)
    • Mean Squared Error for regression tasks
    • Suitable for time series forecasting
  • seq_len: Input sequence length (default: 512)
    • Length of historical context
    • Longer sequences capture more patterns
    • Balance with computational cost
  • pred_len: Prediction length
    • Options: 96, 192, 336, 720
    • Choose based on forecasting horizon needs
    • Longer predictions are more challenging
  • c_out: Output dimension (dataset-specific)
    • ETTh1/h2/m1/m2: 7
    • Weather: 21
    • Electricity: 321
    • Traffic: 862
  • periodicity: Dataset periodicity (dataset-specific)
    • ETTh1/h2, Electricity, Traffic: 24 (hourly data with daily patterns)
    • ETTm1/m2: 96 (15-minute data with daily patterns)
    • Weather: 144 (10-minute data with daily patterns)
    • Critical for capturing temporal patterns
  • norm_const: Normalization coefficient (default: 0.4)
    • Stabilizes training dynamics
    • Range: 0.1 - 1.0
    • Adjust based on data scale
  • patch_len: Patch length (default: 16)
    • Length of each patch for embedding
    • Larger patches capture longer local patterns
    • Must divide seq_len evenly
  • padding: Padding length (default: 8)
    • Ensures proper patch coverage
    • Usually half of patch_len
  • stride: Stride length (default: 8)
    • Overlap between patches
    • Smaller stride = more patches = higher computation
  • num_queries: Number of learnable queries (default: 8)
    • For temporal memory mechanism
    • More queries capture more diverse patterns
    • Balance with computational cost
  • n_heads: Number of attention heads (default: 4)
    • Multi-head attention for pattern learning
    • Typical range: 4-8
    • Should divide d_model evenly

Distillation Parameters

  • init_feature_w: Initial feature distillation loss weight (default: 0.01) [Learnable]
    • Controls feature-level knowledge transfer
    • Start small to avoid overwhelming main task
    • Will be optimized during training
  • init_fcst_w: Initial prediction task loss weight (default: 1.0) [Learnable]
    • Main forecasting task weight
    • Usually kept as highest weight
    • Ensures primary objective is maintained
  • init_recon_w: Initial reconstruction loss weight (default: 0.5) [Learnable]
    • Balances reconstruction quality
    • Helps preserve input information
    • Adjust based on reconstruction importance
  • init_att_w: Initial attention distillation loss weight (default: 0.01) [Learnable]
    • Transfers attention patterns from teacher
    • Start small as attention patterns are high-level
    • Increases model interpretability
  • init_temperature: Initial distillation temperature (default: 4) [Learnable]
    • Softens teacher model outputs
    • Higher values = softer probability distributions
    • Range: 3-10 typically
  • distill_lr_ratio: Learning rate ratio for distillation parameters (default: 0.1)
    • Distillation parameters learn at 0.1x main learning rate
    • Ensures stable weight adaptation
    • Prevents rapid weight changes
  • num_alignment_scales: Number of multi-scale feature alignments (default: 3)
    • Hierarchical feature matching levels
    • More scales = better multi-resolution transfer
    • Computational cost increases linearly
  • feature_alignment_dropout: Dropout for feature alignment (default: 0.1)
    • Prevents overfitting in alignment layers
    • Similar principles as main dropout
    • Keep consistent with main model dropout
  • loss_momentum: Momentum for loss balancer (default: 0.9)
    • Smooth updates of loss weights
    • Higher values = more stable weight evolution
    • Range: 0.8-0.95
  • weight_regularization: Weight regularization coefficient (default: 0.001)
    • Prevents weight degradation
    • Keeps weights in reasonable range
    • Increase if weights become too extreme

Additional Notes:

  • All distillation weights are learnable and optimized during training
  • Enable adaptive balancing for automatic weight tuning
  • Monitor weight evolution to ensure balanced learning