SkySense++

This repository is the official implementation of the paper "SkySense++: A Semantic-Enhanced Multi-Modal Remote Sensing Foundation Model Beyond SkySense for Earth Observation".

📢 Latest Updates

🔥🔥🔥 Last Updated on 2025.09.15 🔥🔥🔥

• [2025.09.15] Add a 🌍 project page.
• [2025.08.04] Our work has been published in Nature Machine Intelligence.
• [2025.03.23] Code for preprocessing/pretraining/application and model weights for models have been uploaded.
• [2025.03.14] updated optical images of JL-16 dataset in Huggingface.
• [2025.03.12] updated sentinel-1 images and labels of JL-16 dataset in Zenodo.
• [2025.03.09] created repo in Zenodo, datasets are uploading.
• [2024.11.13] updated details of pretrain and evaluation data.

Pretrain Data

RS-Semantic Dataset

We conduct semantic-enhanced pretraining on the RS-Semantic dataset, which consists of 13 datasets with pixel-level annotations. Below are the specifics of these datasets. (also see in Zenodo).

Dataset	Modalities	GSD(m)	Size	Categories	Download Link
Five Billion Pixels	Gaofen-2	4	6800x7200	24	Download
Potsdam	Airborne	0.05	6000x6000	5	Download
Vaihingen	Airborne	0.05	2494x2064	5	Download
Deepglobe	WorldView	0.5	2448x2448	6	Download
iSAID	Multiple Sensors	-	800x800 to 4000x13000	15	Download
LoveDA	Spaceborne	0.3	1024x1024	7	Download
DynamicEarthNet	WorldView	0.3	1024x1024	7	Download
	Sentinel-2*	10	32x32
	Sentinel-1*	10	32x33
Pastis-MM	WorldView	0.3	1024x1024	18	Download
	Sentinel-2*	10	32x32
	Sentinel-1*	10	32x33
C2Seg-AB	Sentinel-2*	10	128x128	13	Download
	Sentinel-1*	10	128x128
FLAIR	Spot-5	0.2	512x512	12	Download
	Sentinel-2*	10	40x40
DFC20	Sentinel-2	10	256x256	9	Download
	Sentinel-1	10	256x256
S2-naip	NAIP	1	512x512	32	Download
	Sentinel-2*	10	64x64
	Sentinel-1*	10	64x64
JL-16	Jilin-1	0.72	512x512	16	Download
	Sentinel-1*	10	40x40

* for time-series data.

RS-Representation Dataset

The pretraining list is in the Zenodo- rep_data_list.tar. The download and process scripts are in tools/pretraining_data_builder.

EO Benchmark

We evaluate our SkySense++ on 12 typical Earth Observation (EO) tasks across 7 domains: agriculture, forestry, oceanography, atmosphere, biology, land surveying, and disaster management. The detailed information about the datasets used for evaluation is as follows.

Domain	Task type	Dataset	Modalities	GSD	Image size	Download Link	Notes
Agriculture	Crop classification	Germany	Sentinel-2*	10	24x24	Download
Foresetry	Tree species classification	TreeSatAI-Time-Series	Airborne,	0.2	304x304	Download
			Sentinel-2*	10	6x6
			Sentinel-1*	10	6x6
	Deforestation segmentation	Atlantic	Sentinel-2	10	512x512	Download
Oceanography	Oil spill segmentation	SOS	Sentinel-1	10	256x256	Download
Atmosphere	Air pollution regression	3pollution	Sentinel-2	10	200x200	Download
			Sentinel-5P	2600	120x120
Biology	Wildlife detection	Kenya	Airborne	-	3068x4603	Download
Land surveying	LULC mapping	C2Seg-BW	Gaofen-6	10	256x256	Download
			Gaofen-3	10	256x256
	Change detection	dsifn-cd	GoogleEarth	0.3	512x512	Download
Disaster management	Flood monitoring	Flood-3i	Airborne	0.05	256 × 256	Download
		C2SMSFloods	Sentinel-2, Sentinel-1	10	512x512	Download
	Wildfire monitoring	CABUAR	Sentinel-2	10	5490 × 5490	Download
	Landslide mapping	GVLM	GoogleEarth	0.3	1748x1748 ~ 10808x7424	Download
	Building damage assessment	xBD	WorldView	0.3	1024x1024	Download

* for time-series data.

Implementation Code

Structure

This project mainly contains the following parts.

./
├── antmmf/                             # antmmf framework code
├── configs/                   
│   ├── eval_skysense_pp_flood3i.yml    # eval cfg on flood3i                
│   └── pretrain_skysensepp.yml         # pretrain cfg
├── finetune/                           # finetuning code
│   ├── configs/                        # finetuning configs
│   ├── mmseg/                          # mmseg library
│   ├── requirements/                   # mmseg install requirements folder
│   ├── requirements.txt                # mmseg install requirements
│   ├── setup.py                        # mmseg setup file
│   └── tools/                          # mmseg utils
├── lib/                                # model implementation
│   ├── datasets/                       # datasets for evaluation
│   ├── evaluation/                     # evaluation code
│   ├── models/                         # model architecture
│   ├── predictors/                     # inference code
│   ├── task/                           # task code
│   ├── trainer/                        # trainer code
│   ├── utils/                          # library code
│   └── __init__.py                     # packages init file
├── pretrain/                           # pretrain ckpts
├── tools/                              # tools ckpts
│   ├── pretraining_data_builder        # pretraining dataset builder
│   ├── run_1shot_flood3i.sh            # datasets for evaluation
│   ├── run_ft_atlantic.sh              # run ft script
│   ├── run_pretrain.sh                 # run pretrain script
│   └── run.py                          # Program entry point
└── readme.md                           # project readme

Environment

Each machine for implementating the pretraining or fintuning are with Alibaba Group Enterprise Linux(7.2) and Python 3.8.10. The pretraining and finetuning code are implemented on severs with Intel(R) Xeon(R) Platinum 8369B CPU @ 2.90GHz and Nvidia A100 GPUS.

Pretraining

To run our pretraining code, please install dependency packages. (Instalazation takes about 14 minutes on a node with Intel(R) Xeon(R) Platinum 8369B CPU @ 2.90GHz and 8 A100 GPUs.)

torch==1.13.1
atorch==0.1.3
torchvision==0.14.1
mmcv-full==1.7.1
mmsegmentation==0.30.0
mmcls==0.25.0
timm==0.6.13
gdal==3.4.0
scikit-image==0.19.3

Step1. Install the above packages and clone antmmf framework:

git clone https://github.com/alipay/Ant-Multi-Modal-Framework.git antmmf/

Step2. Download the pretraining datasets in Zenodo and orgnize them as follows:

pretrain_datasets
├── dynamic-mm                          # multi-modal dynamic-mm datasets
│   ├── images_hr                       # hr images
│   ├── images_s2                       # sentinel-2 images
│   ├── images_s1                       # sentinel-1 images
│   ├── labels                          # segmentation annotations
│   ├── dynamic-mm_train.json           # train list file
│   └── dynamic-mm_val.json             # val list file
├── fbp                                 # single-modal fbp datasets
│   ├── images                          # input gaofen-2 images
│   ├── labels                          # segmentation annotations
│   ├── fbp_train.json                  # train list file
│   └── fbp_val.json                    # val list file
└── ......                       

The <dataset>_<train/val>.json is used to read information for training and validation, with a unified organizational format:

[
  {
    "hr_path": "dataset_name/images_hr/<img_name>.png", // hr info c,h,w
    "s2_path": ["dataset_name/images_s2/<img_name>_20240101.npz", "dataset_name/images_s2/<img_name>_20240103.npz"], // s2 c,h,w
    "s1_path": ["dataset_name/images_s1/<img_name>_20240104.npz", "dataset_name/images_s1/<img_name>_20240108.npz"], // s1 c,h,w
    "target_path": "dataset_name/labels/<img_name>.png", // annotation info
    "type": "dataset_name", // dataset_name
    "classes": [            // Included categories
            0,
            2,
            4,
            5
        ]
  },
  {
    ...
  }
]

Step3. Download the pretraining weights of SkySense here and move it to pretrain/

Step4. Run the pretrain code on 4 nodes (each node with 8 A100 gpus):

bash tools/run_pretrain.sh <node_rank:0-3> <master_ip_address>

For example, if the ip adress of master node is 192.168.112.10, the command for node 1 is:

bash tools/run_pretrain.sh 1 192.168.112.10

Downstream 1-shot application

Requirments

To run our code, please install dependency packages. ( Instalazation takes about 10 minutes on a sever with Intel(R) Xeon(R) Platinum 8369B CPU @ 2.90GHz and 2 A100 GPUs.)

torch==1.13.1
atorch==0.1.3
torchvision==0.14.1
mmcv-full==1.7.1
mmcls==0.25.0
mmsegmentation==0.30.0
timm==0.6.13
gdal==3.4.0
scikit-image==0.19.3

Run steps

step1. Clone antmmf framework. and install the above packages:

git clone https://github.com/alipay/Ant-Multi-Modal-Framework.git antmmf/

step1. Download the flood-3i dataset (Images.zip/Semantic_mask.zip at here, val.txt at here. Testing dataset should be organized as follows:

eval_datasets/
└── flood3i/
    ├── Images/
    │   ├── 10165_0_2.jpg
    │   ├── 10165_1_0.jpg
    │   └── ...
    ├── Semantic_mask/
    │   ├── 10165_lab_0_2.png
    │   ├── 10165_lab_1_0.png
    │   └── ...
    └── val.txt

step2. Using the above pretraining wieights or download the pretrained model weights here.

step3. Run the script for evaluating 1-shot performance on flood-3i:

bash tools/run_1shot.sh <gpu_idx> flood-3i(dataset_name)

Downstream finetuning application

Requirments

We build our fine-tuning application code on the openmmlab framework.

To run our code, please install dependency packages. ( Instalazation takes about 10 minutes on a sever with Intel(R) Xeon(R) Platinum 8369B CPU @ 2.90GHz and 2 A100 GPUs.)

torch==1.13.1
torchvision==0.14.1
mmcv-full==2.1.0
mmpretrain==1.2.0
mmsegmentation==1.2.2
mmdetection==3.3.0
timm==0.6.13
gdal==3.4.0
scikit-image==0.19.3

Run steps

Step1. Install the mmsegmentation framework under the instrction in here

Step2. Download the evaluation datsets. We take Atlantic dataset for deforestation segmentation as an example. Download the Atlantic dataset at here. Spliting json files of evaluation framwork here.

../rs_datasets/deforestation_atlantic/
--
├── deforestation_atlantic_test.json
├── deforestation_atlantic_train.json
├── deforestation_atlantic_val.json
├── Test/
│   ├── image/
│   └── label/
├── Training/
│   ├── image/
│   └── label/
└── Validation/
    ├── image/
    └── label/

Step3. Use your pretrained model weights or download the model weights: here

Step4. Run the finetuning script. We take the Atlantic dataset as an example:

bash tools/run_finetune.sh configs/atlantic.py

Acknowledgments

This projects are mainly built on the following projects:

• antmmf
• mmcv
• mmpretrain
• mmsegmentation
• mmdetection
• Painter

License

The pre-trained model weight and pre-training code are only available for the non-commercial research. For any commercial use or cooperation, please contact Yansheng Li at Wuhan University (e-mail: yansheng.li@whu.edu.cn).

Citation

If you find our repo useful, please consider giving a star and citation:

@article{wu2025semantic,
  author       = {Wu, Kang and Zhang, Yingying and Ru, Lixiang and Dang, Bo and Lao, Jiangwei and Yu, Lei and Luo, Junwei and Zhu, Zifan and Sun, Yue and Zhang, Jiahao and Zhu, Qi and Wang, Jian and Yang, Ming and Chen, Jingdong and Zhang, Yongjun and Li, Yansheng},
  title        = {A semantic‑enhanced multi‑modal remote sensing foundation model for Earth observation},
  journal      = {Nature Machine Intelligence},
  year         = {2025},
  doi          = {10.1038/s42256-025-01078-8},
  url          = {https://doi.org/10.1038/s42256-025-01078-8}
}

@inproceedings{guo2024skysense,
    author    = {Guo, Xin and Lao, Jiangwei and Dang, Bo and Zhang, Yingying and Yu, Lei and Ru, Lixiang and Zhong, Liheng and Huang, Ziyuan and Wu, Kang and Hu, Dingxiang and He, Huimei and Wang, Jian and Chen, Jingdong and Yang, Ming and Zhang, Yongjun and Li, Yansheng},
    title     = {SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {27672-27683}
}

@inproceedings{zhu2025skysenseo,
  title={Skysense-o: Towards open-world remote sensing interpretation with vision-centric visual-language modeling},
  author={Zhu, Qi and Lao, Jiangwei and Ji, Deyi and Luo, Junwei and Wu, Kang and Zhang, Yingying and Ru, Lixiang and Wang, Jian and Chen, Jingdong and Yang, Ming and others},
  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
  pages={14733--14744},
  year={2025}
}

@article{luo2024skysensegpt,
  title={Skysensegpt: A fine-grained instruction tuning dataset and model for remote sensing vision-language understanding},
  author={Luo, Junwei and Pang, Zhen and Zhang, Yongjun and Wang, Tingzhu and Wang, Linlin and Dang, Bo and Lao, Jiangwei and Wang, Jian and Chen, Jingdong and Tan, Yihua and others},
  journal={arXiv preprint arXiv:2406.10100},
  year={2024}
}

Star History