📄 Abstract

Remote Sensing (RS) data encapsulates rich multi-dimensional information essential for Earth observation. Its vast volume, diverse sources, and temporal continuity make it particularly well-suited for developing large Visual Foundation Models (VFMs). These models serve as powerful feature extractors, leveraging extensive RS data for pretraining and subsequent fine-tuning in various geoscientific applications. However, existing VFMs in the RS domain often concentrate on specific image characteristics, neglecting the full season-aware potential of RS data. To bridge this gap, we introduce SeaMo, a novel VFM that effectively integrates multimodal and multi-seasonal RS information. SeaMo leverages a masked image modeling framework to fully exploit the spatial, spectral, and seasonal dimensions of RS data. Specifically, we employ unaligned spatial region selection to capture spatial heterogeneity, incorporate multi-source inputs for enhanced multimodal integration, and introduce temporal-multimodal fusion blocks to assimilate seasonal variations effectively. By explicitly modeling the complex, season-dependent attributes of RS data, SeaMo enhances generalization, robustness, and adaptability across geoscientific tasks. Extensive experiments and ablation studies demonstrate its superior performance, underscoring its potential as a foundational model for Earth observation.

---

👀 Overview

---

📋 Table of Contents

• Abstract
• Overview
• Preparation
• Pretraining SeaMo
• A Quick Start on How to Use SeaMo
• Downstream Tasks
• Acknowledgements
• Citation
• Licensing
• Contact Information

---

🛠️ Preparation

Install Python dependencies by running:

pip install -r requirements.txt

---

🚀 Pretraining SeaMo

The pretraining experiments were conducted on four NVIDIA GeForce RTX 4090 GPUs.
Our pretraining implementation is primarily based on SSL4EO-S12.

The pre-trained model can be downloaded 🤗here. This checkpoint includes the full pre-training setup, comprising both the Encoder with the TM Block and the Decoder. It corresponds exactly to the implementation in models_pretrain_seamo.py, specifically the code at lines 532–542.

We also provide the backbone model 🤗here, which contains only the Encoder. This makes it convenient to use SeaMo for feature extraction. It corresponds exactly to the implementation in models_feature_seamo.py, specifically the code at lines 375–388. A quick guide on how to use SeaMo will be provided in the following sections.

Pretrained Model Download

Model	Download Link	Description
checkpoint-199.pth	Download	The Complete Pretrained Model
SeaMo.pth	Download	The Backbone Model (Only Encoder)

Pretrained Dataset SSL4EO-S12

Dataset	Download Link	Task Description
SSL4EO-S12	Download	Sentinel 1 & 2 Pretraining Dataset

---

📚 A Quick Start on How to Use SeaMo

We provide several ways to load the pre-trained SeaMo model:

• models_cls_seamo.py can be directly used for classification tasks. It loads SeaMo with a classification head that outputs a single token prediction. You only need to set the desired value for num_classes.
• models_feature_seamo.py can be used for feature extraction. It loads SeaMo and outputs the feature maps extracted by the model.

Since SeaMo is pre-trained on spectral images with a setting of 128×128 and 12 channels, and SAR images with a setting of 128×128 and 2 channels, certain adjustments are required when adapting it to inputs with different image sizes or channel numbers. Below are specific guidelines to help you make these modifications.

Image Size Modifications

SeaMo follows the Vision Transformer paradigm, which can handle sequences of arbitrary length. When changing the spatial resolution of input images, only the positional encoding needs adjustment. Here are two practical approaches:

1. Checkpoint-Level Interpolation (Fixed Image Sizes)

If the target image size is known and fixed, you can interpolate the positional encodings once and save a new checkpoint using convert_model.py:

python convert_model.py \
  --src SeaMo.pth \
  --dst SeaMo_img256.pth \
  --img_size 256 \
  --patch_size 8

The result 🤗SeaMo_img256.pth can be loaded directly, e.g. in models_feature_convert_seamo.py:

import torch
from models_feature_convert_seamo import vit_base_patch8_256

img   = torch.randn(1, 12, 256, 256).cuda()
model = vit_base_patch8_256().cuda()

checkpoint = torch.load("SeaMo_img256.pth", map_location="cuda")
msg = model.load_state_dict(checkpoint, strict=False)
print(msg)

out = model(img)
print(out.shape)

Note: convert_model.py interpolates only the positional-encoding weights. The converted checkpoint must be loaded into a network definition with the matching img_size (e.g. vit_base_patch8_256 for 256 × 256, vit_base_patch8_128 for 128 × 128).

2. On-the-Fly Interpolation (Flexible Image Sizes)

For a network that handles any resolution, use models that interpolate positional encodings in the forward pass. Both models_feature_seamo.py and models_feature_seamo.py already contain this logic (see lines 142–169 in models_feature_seamo.py):

import torch
from models_feature_seamo import vit_base_patch8

img = torch.randn(1, 12, 196, 196).cuda()
model = vit_base_patch8().cuda()

pretrained_dict = torch.load('SeaMo.pth', map_location='cuda')
checkpoint_model = pretrained_dict
state_dict = model.state_dict()

msg = model.load_state_dict(checkpoint_model, strict=False)
print(msg)
output = model(img)
print(output.shape)

Note: With this approach you do not need to pre-convert the weights, and the same vit_base_patch8 backbone will work for any spatial resolution.

Image Channel Modifications

In all model files, the PatchEmbed class uses a dictionary-based projection layer design to handle inputs with different channel configurations. By default, it includes projection layers for 12-channel spectral images and 2-channel SAR images:

self.proj_dict = nn.ModuleDict({
    'channel_12': nn.Conv2d(12, embed_dim, kernel_size=patch_size, stride=patch_size),
    'channel_2': nn.Conv2d(2, embed_dim, kernel_size=patch_size, stride=patch_size)
})

The forward function dynamically adds new projection layers for unseen channel numbers:

if channel_type is None:
    channel_type = f'channel_{C}'
    self.proj_dict[channel_type] = nn.Conv2d(
        C, out_channels=self.embed_dim,
        kernel_size=self.patch_size, stride=self.patch_size
    ).cuda()

This means that when the input image does not have 12 or 2 channels, the model will automatically create and register a new projection layer, without requiring any manual changes to the architecture.

Note: While this mechanism allows architectural compatibility with arbitrary input channels, we recommend fine-tuning the model when using different channel configurations. Directly using a pre-trained backbone for feature extraction without fine-tuning may lead to suboptimal results.

Image Normalization Recommendation

SeaMo uses channel-wise normalization with fixed mean and standard deviation values from SSL4EO-S12. To ensure consistency and optimal performance, we recommend applying the same normalization scheme during downstream tasks:

Sentinel-2 (12-channel MSI) Normalization

S2A_MEAN = [752.40087073, 884.29673756, 1144.16202635, 1297.47289228, 
            1624.90992062, 2194.6423161, 2422.21248945, 2517.76053101, 
            2581.64687018, 2645.51888987, 2368.51236873, 1805.06846033]

S2A_STD  = [1108.02887453, 1155.15170768, 1183.6292542, 1368.11351514, 
            1370.265037, 1355.55390699, 1416.51487101, 1474.78900051, 
            1439.3086061, 1582.28010962, 1455.52084939, 1343.48379601]

Sentinel-1 (2-channel SAR) Normalization

S1_MEAN = [-12.54847273, -20.19237134]
S1_STD  = [5.25697717, 5.91150917]

Note: Using these normalization statistics is important to ensure compatibility with the pre-training distribution. Applying different normalization schemes may lead to degraded performance in fine-tuning or inference.

---

🗺️ Downstream Tasks

Downstream Dataset

The table below compiles the official download links for every dataset used to fine-tune our pre-trained model on a variety of multimodal tasks, ranging from spectral and SAR scene classification to multimodal land-cover classification and semantic segmentation.

Dataset	Download Link	Task Description
EuroSAT-MS	Download	MSI Scene Classification Tasks
fMoW-S2	Download	MSI Scene Classification Tasks
SegMunich	Download	MSI Semantic Land Cover Segmentation Tasks
OSCD	Download	MSI Urban Change Detection Tasks
EuroSAR-SAR	Download	SAR Scene Classification Tasks
GLH-Water	Download	High-resolution RGB Water Detection Tasks
BigEarthNet	Download	MSI & SAR Multi-label Land Cover Classification Tasks
DFC2020	Download	MSI & SAR Land Cover Segmentation Tasks
S1S2-Water	Download	MSI & SAR Water Detection Tasks

Downstream Fine-Tune Implementation

For downstream tasks, please refer to the codebase provided by SpectralGPT and SSL4EO-S12.

---

🙏 Acknowledgements

Pretrain and downstream classification codes are inspired by MAE and SSL4EO-S12. Downstream pixel-level codes are inspired by SpectralGPT.

---

📝 Citation

If you find our project helpful, please cite our paper:

Li, Xuyang, Chenyu Li, Gemine Vivone, and Danfeng Hong. "SeaMo: A season-aware multimodal foundation model for remote sensing." Information Fusion 125 (2026): 103334.

@article{li2026seamo,
  title={SeaMo: A season-aware multimodal foundation model for remote sensing},
  author={Li, Xuyang and Li, Chenyu and Vivone, Gemine and Hong, Danfeng},
  journal={Information Fusion},
  volume={125},
  pages={103334},
  year={2026},
  publisher={Elsevier}
}

---

📜 Licensing

Copyright © 2026 Danfeng Hong

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

---

📧 Contact Information

Danfeng Hong: hongdanfeng1989@gmail.com
School of Automation, Southeast University, 211189 Nanjing, China.

---