REMSA: Foundation Model Selection for Remote Sensing via a Constraint-Aware Agent

July 7, 2026 · View on GitHub

Alt text

This repository contains the code of the paper REMSA: Foundation Model Selection for Remote Sensing via a Constraint-Aware Agent. The work introduces RS-FMD, the first structured database of Remote Sensing Foundation Models (RSFMs), and REMSA, the first LLM-based agent designed to automatically select suitable foundation models for a given remote sensing task.

This work has been done at the Remote Sensing Image Analysis group and the BIFOLD by Binger Chen, Tacettin Emre Bök, Behnood Rasti, Volker Markl, and Begüm Demir.

If you use this code, please cite our paper given below:

B. Chen, T. E. Bök, B. Rasti, V. Markl, B. Demir, "REMSA: Foundation Model Selection for Remote Sensing via a Constraint-Aware Agent", arXiv preprint arXiv:2511.17442 (2025).

@misc{chen2026remsafoundationmodelselection,
      title={REMSA: Foundation Model Selection for Remote Sensing via a Constraint-Aware Agent}, 
      author={Binger Chen and Tacettin Emre Bök and Behnood Rasti and Volker Markl and Begüm Demir},
      year={2026},
      eprint={2511.17442},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2511.17442}, 
}

Overview

REMSA REMSA automates RS foundation model selection by combining:

  • A structured database (RS-FMD) containing metadata for 160+ RSFMs
  • A modular agent architecture with:
    • query interpretation
    • metadata-grounded retrieval
    • LLM-based candidate ranking
    • clarification loops
    • transparent explanations

The system supports diverse RS tasks and modalities and provides reproducible, user-tailored model recommendations.

Environment Setup

This project uses uv for Python environment and dependency management.

Install dependencies from the project root:

uv sync
source .venv/bin/activate

This creates a local virtual environment in .venv/ and installs all required dependencies defined in pyproject.toml and locked in uv.lock.

Configuration

REMSA is fully configurable via the config.py file. The following parameters can be adjusted:

  • LLM backend selection
  • LLM inference settings
  • Database and vector index paths
  • Output formatting and thresholds

Adjust these parameters to suit your runtime environment and model preferences.

Running the Agent

To launch the agent REMSA:

python main.py

You will be prompted to provide a natural-language query describing your RS application, data modality, constraints, and requirements.

REMSA then interprets your input, retrieves suitable candidate models from RS-FMD, ranks them, and outputs recommendations with explanations.

Data Structure

RS-FMD Model Metadata

  • Located in model_metadata/
  • Contains structured JSON model metadata extracted from papers, repositories, and model cards
  • Supports 160+ Remote Sensing Foundation Models

Internal Runtime Data

  • data/: processed database used at runtime
  • data/queries.txt: example queries used for experiments

You may extend these files with additional models or metadata.

RS-FMD: Generating Foundation Model Metadata

To extract structured metadata for a new foundation model from its documentation:

# Navigate to the directory
cd file_to_db

# Run the script with your configuration and file path
python src/run.py --config FoundationModels.yaml --file_path path_to_your_pdf

The resulting JSON metadata file will be stored under:

/file_to_db/model_metadata

Building a Vector Database

To build the vectorstore from your .jsonl metadata file:

python build_vectorstore.py

Configurable parameters in config.py include:

  • FMD_JSONL_PATH: Path to your .jsonl file.
  • EMBEDDING_MODEL_NAME: Name of the embedding model to use.
  • VECTOR_INDEX_PATH: Path where the generated vector database will be stored.

Authors

Binger Chen https://hu.berlin/binger_chen

Tacettin Emre Bök https://rsim.berlin/team/members/tacettin-bok

Behnood Rasti https://rsim.berlin/team/members/behnood-rasti

Volker Markl https://www.bifold.berlin/people/prof-dr-volker-markl.html

Begüm Demir https://rsim.berlin/team/members/begum-demir

For questions, requests and concerns, please contact Binger Chen

License

The code in this repository is licensed under the MIT License.
See the LICENSE file for more details.