Target Location Datasets

We release the following training & evaluation datasets with target locations annotations on Huggingface. These datasets will automatically be downloaded for training and inference, although you will still need to download the raw satellite images and iNaturalist files in the following sections.

• Quad-modal: Sat-Text-Image-Sound pairing modalities
  • CLIP Training
  • Search Evaluation
• Tri-modal: Sat-Text-Image pairing modalities
  • CLIP Training
  • Search Evaluation

Satellite Images and Sound Data

For convenience, you may directly download the satellite images and sound zip files from the links below. Alternatively, you may run the scripts from taxabind_avs/scripts/ that download data the Taxabind datasets. Note that you should download the partial dataset if you only want to run evals.

• Satellite Images: Partial Eval, Full Train
• Sound + Ground Images: Partial Eval, Full Train

iNaturalist Ground Images

Download the following datasets from the iNaturalist 2021 Challenge using the links below. Note that you should download the partial dataset if you only want to run evals.

• Partial iNat Dataset: Partial Images+Json
• Full iNat Dataset: Train Images, Train Json, Val Images, Val Json

Dataset Organization

You must download the datasets from the above links, and organize them as follows.

Note: You will need to download at least the partial datset from above if you want to only perform evals. If you would like to train the satellite image or sound encoder, please download the full dataset. The total partial dataset size for evals is ~2GB, while full dataset size for training is around ~350GB.

├── avs_bench_ds
│   ├── inat21
│   │   ├── train
│   │   │   ├── 00000_Animalia_Annelida_Clitellata_Haplotaxida_...jpg
│   │   │   └── ...
│   │   ├── val
│   │   │   ├── 00000_Animalia_Annelida_Clitellata_Haplotaxida_...jpg
│   │   │   └── ...
│   │   ├── train.json
│   │   └── val.json
│   ├── sat_jpg
│   │   ├── train_512px
│   │   │   ├── 0_43.83486_-71.22231.jpg
│   │   │   └── ...
│   │   └── test_512px
│   │       ├── 2686843_-21.93073_114.12239.jpg
│   │       └── ...
│   └── sound_mp3
│       ├── train
│       │   ├── sounds
│       |   │   ├── 100002768.mp3
│       |   │   └── ...
│       │   └── images
│       |       ├── 100002768.jpg
│       |       └── ...
│       └── test
│           ├── sounds
│           │   ├── 100010745.mp3
│           │   └── ...
│           └── images
│               ├── 100010745.jpg
│               └── ...

Code Structure

The structure of our codebase is as follows:

• eval/ evaluation shellscripts to evaluate Search-TTA.
• planner/ planner framework scripts of Search-TTA.
• train/ training models, logs, and gifs.
• inference/ trained model, inference logs and gifs.
• maps/ training/eval envs and score maps.
• taxabind_avs/
  • satbind/ training & TTA scripts for satellite image encoder.
  • soundbind/ training scripts for sound encoder.
  • scripts/ scripts to download inat raw datasets.

SatBind

To train the satellite image encoder, follow the steps below. It automatically downloads the tri_modal dataset from Huggingface, and trains the satellite image encoder to align to the same representation space as BioCLIP's ground image encoder. Note that you should adjust the avs_ds_dir parameter in config_sat.py to match your downloaded dataset directories. We offer the finetuned sat encoder checkpoint here.

cd taxabind_avs/satbind   # Adjust config_sat.py
python model_sat.py
tensorboard --logdir=lightning_logs

SoundBind

To train the sound encoder, follow the steps below. It automatically downloads the quad_modal dataset from Huggingface, and trains the sound encoder to align to the same representation space as BioCLIP's ground image encoder. Note that you should adjust the avs_ds_dir parameter in config_sound.py to match your downloaded dataset directories. We offer the finetuned sound encoder checkpoint here.

cd taxabind_avs/soundbind   # Adjust config_sound.py
python model_sound.py
tensorboard --logdir=lightning_logs

RL Planner

To train the RL planner, follow the steps below. The planner is trained using the score maps and envs from maps/GT_GPT4o, which are generated by converting point locations to segmentation score masks using GPT4o. We offer the trained model checkpoint here.

# Adjust planner/parameter.py
python -m planner.driver    
tensorboard --logdir=train/logs

Custom Score Maps

Instead of running on the planners on CLIP-generated score maps, you may run them on your custom score maps. Please refer to eval/eval_lisa.sh for an example on how to load your custom data. You can download the score maps for LISA here.

Custom Target Positions

Instead of retrieving targets from AVS-Bench, you can override the target positions by setting the TARGETS_SET_DIR parameter in test_parameter.py. This loads maps with target positions marked with grey squares (value of 208). See an example in maps/example/gt_masks_val_with_tgts (only targets, not mask, are loaded). Note that you need to set LOAD_AVS_BENCH to False in order to use this feature.

Custom Planners

Instead of using our Reinforcement Learning (RL) or Information Surfing (IS) based planner, you may use your own custom planners. For this, initialize another script that is similar to test_worker.py or test_info_surfing.py, and interface it with the Ray framework in test_driver.py and search environment in env.py.