GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Data 🌍🛰️

Welcome to the official repository for the NeurIPS 2025 paper "GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Data"!

📄 Read our paper: arXiv Preprint

📖 About This Project

GeoLink is a multimodal framework that empowers remote sensing foundation models by integrating OpenStreetMap (OSM) data in both pretraining stage and downstream tasks. Most traditional remote sensing models have focused solely on imagery, but GeoLink bridges the critical modality gap between RS and OSM data, enabling synergy for geospatial intelligence.

🚀 Key characteristics:

• Finegrained fusion: GeoLink can fusion RS and OSM data at entity-patch level, and obtain hybrid RS-OSM patch encodings for pixel-level tasks like semantic segmentation. Example can be found in Using Multimodal GeoLink.
• Versatile Applications: Supports both unimodal and multimodal fine-grained encodings for diverse tasks from land cover classification to urban function zone mapping.

Our extensive experiments demonstrate that incorporating OSM data significantly boosts RS image encoder performance and enhances adaptability to complex geographic scenarios.

🏋️‍♂️ Pre-trained Models

We provide two pre-trained models for different use cases:

1. Unimodal GeoLink(ViT) 🔍

• Input: Remote sensing images only. Pretrained by GeoLink and we extract the image encoder out for unimodal application. Using cases can be found in Using Unimodal GeoLink (ViT)
• Output: Remote sensing encodings
• Use case: Standard RS interpretation tasks
• Download: PKU Disk Link or Google Drive

2. Multimodal GeoLink 🌐

• Input: Remote sensing images + OSM data. Using canses can be found in Using Multimodal GeoLink.
• Output: Fused RS-OSM features
• Use case: Comprehensive geographic tasks requiring multimodal understanding
• Download: PKU Disk Link or Google Drive

⚙️ Installation & Dependencies

First, install the required dependencies:

pip install torch==1.10.0+cu111 torchvision==0.11.0+cu111
pip install torch-cluster==1.6.0 torch_geometric==2.4.0 torch-scatter==2.0.9 torch-sparse==0.6.12 torch-spline-conv==1.2.1
pip install timm==0.6.12 transformers==4.46.3

🛠️ Quick Start

Here are some using examples of GeoLink. There are some example data in ./exapmle_data dir, including ufz_example to demonstrate how to use GeoLink to extract multimodal encodings and perform downstream semantic segmentation tasks, as well as osm2graph_example to illustrate how to convert OSM data into standard GeoLink input. The following examples are from ./using_example.ipynb.

Using Unimodal GeolInk (ViT)

If you only need remote sensing image processing capabilities, use GeoLink pretrained unimodal ViT model. After download the checkpoint, you can use timm ViT to load it directly:

import timm
import torch

# Load the pre-trained model
model_path = 'geolink_vit_large_patch16_224.pth'  # path to ViT checkpoint
checkpoint = torch.load(model_path, map_location='cpu')
config = checkpoint['model_config']

# Create and initialize the model
model = timm.create_model(
    config['architecture'], 
    pretrained=False, 
    num_classes=config['num_classes'],  # 0
    global_pool=config['global_pool']   # ''
)
model.load_state_dict(checkpoint['model_state_dict'])

# Using sinusoidal position embedding with no back-propagation
model.pos_embed.requires_grad = config['pos_embed_requires_grad']
# Then you can use it to generate embeddings or add the task-specific decoder for a given task.

Using Multimodal GeoLink

To use the full multimodal GeoLink model for generating RS-OSM fusion embeddings. After download the checkpoint, you should load GeoLink model:

import timm
import torch
from model import *
from dataset import *

'''
    You can use the following code to obtain the fusion embeddings.
    The default output embeedings are from the 7/11/15th layers of ViT-L and the hybrid RS-OSM embeddings.
    The hybrid RS-OSM embeddings are generated by the integration of features from last(23th) layer of ViT-L
    and OSM embeddings (detail structure are shown in the right of Fig 8b in the paper).
    Such embeddings can be applied to downstream tasks through a task-specific decoder, like UperNet for semantic segmentation.
'''

ckpt_fp = r'geolink_mutimodal_vit_large_patch16_224.pth' # path to multimodal geolink checkpoint dir
checkpoint = torch.load(ckpt_fp, map_location='cpu')
config = checkpoint['model_config']
img_encoder = timm.create_model(
    config['architecture'], 
    pretrained=False, 
    num_classes=config['num_classes'], 
    global_pool=config['global_pool']
)

osm_encoder = OSMHeteroGAT()
geolink = GeoLink(img_encoder, osm_encoder)
msg = geolink.load_state_dict(checkpoint['model_state_dict'])
print(msg)

multi_encoder = GeoLink_Fusion_Embedding(geolink, output_layers=[7, 11, 15, 23])
# 23 means the fusion embedding with shape of [256,14,14]

data_root = r'./example_data/ufz_example'
samples = os.listdir(os.path.join(data_root, 'graph'))

trainset = DownstreamDataset(data_root, file_names=samples)
trainloader = torch_geometric.data.DataLoader(trainset, batch_size=2,
                             pin_memory=True, num_workers=1, drop_last=True, shuffle=True)
train_iter = iter(trainloader)
# get a batch
img, graph = next(train_iter)
result = multi_encoder(img, graph) #list size [4, batch_size, embed_dim, 14, 14]

Using Multimodal GeoLink for semantic segmentation tasks

To use the full multimodal GeoLink model for RS-OSM fusion:

import timm
import torch
from model import *
from dataset import *

ckpt_fp = r'geolink_mutimodal_vit_large_patch16_224.pth' # path to multimodal geolink checkpoint dir
checkpoint = torch.load(ckpt_fp, map_location='cpu')
config = checkpoint['model_config']
img_encoder = timm.create_model(
    config['architecture'], 
    pretrained=False, 
    num_classes=config['num_classes'], 
    global_pool=config['global_pool']
)

osm_encoder = OSMHeteroGAT()
geolink = GeoLink(img_encoder, osm_encoder)
msg = geolink.load_state_dict(checkpoint['model_state_dict'])
print(msg)

multi_encoder = GeoLink_Fusion_Embedding(geolink, output_layers=[7, 11, 15, 23])
model = SegUPerNet(encoder=multi_encoder, num_classes=9, channels=512) # using UperNet as the decoder

data_root = r'./example_data/ufz_example'
samples = os.listdir(os.path.join(data_root, 'graph'))

trainset = DownstreamDataset_UFZ(data_root, file_names=samples)
trainloader = torch_geometric.data.DataLoader(trainset, batch_size=2,
                             pin_memory=True, num_workers=1, drop_last=True, shuffle=True)
train_iter = iter(trainloader)
# get a batch
img, graph, label = next(train_iter)
result = model(img, graph) #list [batch_size, cls_num, 224, 224]

Formatting your own OSM data to standard input

You can use your own OSM data and format to the GeoLink standard input. The OSM data corresponding to a remote sensing image area should be stored as vector types in three separate files: polygon, polyline, and point (at least one vector type must be present). The OSM data can be Shapefile/Geojson/Geopackage/..., and it should has two main columns, OSM Tags and Geometry.

• OSM Tags: record the OSM tags in format 'tag:value' which joined by ';'
• Geometry: record the geometry information, and it can be POLYGON/LINESTRING/POINT

Index	OSM Tags	Geometry
0	leisure:golf_course	POLYGON ((-104.88896 39.38870, -104.88824 39.3...))
1	landuse:grass;golf:tee	POLYGON ((-104.88803 39.38820, -104.88802 39.3...))
2	landuse:grass;golf:green	POLYGON ((-104.88855 39.38835, -104.88847 39.3...))
3	landuse:grass;golf:tee	POLYGON ((-104.88786 39.38859, -104.88786 39.3...))
4	building:building	POLYGON ((-104.88830 39.38886, -104.88842 39.3...))

The following code shows how to format the OSM data the GeoLink standard input. Please ensure your environment can connect to Hugging Face, as it requires using CLIP's BERT model for tag encoding (alternatively, you can download the parameters locally for loading).

import geopandas as gpd
import pickle
import os
from prepare_data import *

example_path = r'./example_data/osm2graph_example/' # path to the osm example dir
tagw_path = r'all_tags30_frequency1.json'

osm_process = OSM2Graph(tagw_path, 'cuda:1')
name = 'a472140250_FI_21' # example name

# obatin the geographic boundingbox of the give RS image
# here, the boundingbox is saved in the meta file
# you can also calculate it from the original RS data like TIFF, tools are prepared in the prepare_data/utils.py
# note that the geographic coodinate system should be the same between RS and OSM data

with open(os.path.join(example_path, name, name+'.pickle'), 'rb') as file:
    meta_data = pickle.load(file)
bbox = meta_data['bbox']
north, south, east, west = bbox[3], bbox[1], bbox[2], bbox[0]
width = east - west
height = north - south

# OSM2Graph can handle situations where one or two vector types are missing
# If none of the three vector types are available, then just use RS image.
try:
    polygon_file = gpd.read_file(os.path.join(example_path, name, name + '_polygon.geojson'))
except:
    polygon_file = None
try:
    line_file = gpd.read_file(os.path.join(example_path, name, name + '_line.geojson'))
except:
    line_file = None
try:
    point_file = gpd.read_file(os.path.join(example_path, name, name + '_point.geojson'))
except:
    point_file = None

data = osm_process.process(polygon_file, line_file, point_file, north, south, east, west)

---

This repository contains the official implementation of GeoLink. If you find our work helpful, please consider citing our paper! 📚

@misc{bai2025geolinkempoweringremotesensing,
      title={GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Data}, 
      author={Lubian Bai and Xiuyuan Zhang and Siqi Zhang and Zepeng Zhang and Haoyu Wang and Wei Qin and Shihong Du},
      year={2025},
      eprint={2509.26016},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2509.26016}, 
}