Affect-aware Cross-Domain Recommendation for Art Therapy via Music Preference Elicitation

August 1, 2025 · View on GitHub

CC BY-NC license Python Pytorch Black

Overview

The official PyTorch implementation of the paper paper "Affect-aware Cross-Domain Recommendation for Art Therapy via Music Preference Elicitation" published in the Proceedings of the 19th ACM Conference on Recommender Systems (RecSys ’25) Read on ACM, Arxiv, ORBilu

Our three Affect Awre Cross-Domain VA RecSys engeines from music preferences, Mozart, Haydn and Salieri are named after the three iconic composers whose contributions shaped the history of Western music. Wolfgang Amadeus Mozart, Joseph Haydn and Antonio Salieri.

Proposed architectures. From left to right: Affect-aware contrastive alignment (Mozart), Affective Space Search (Haydn), and Multi-Modal alignment with LLM and VLM (Salieri)

Joint embeddings

📂 Project Structure

mozart-crossmodal/
├── 📁 data/                         # Datasets and precomputed embeddings  
│   ├── 📁 music/                    # Music-related data  
│   │   ├── 🎵 audio/                 # MP3 audio files  
│   │   ├── 🎵 heal_audio/            # MP3 audio files selected for preference elicitation
│   │   ├── 📊 features/              # Precomputed acoustic features (CSVs)  
│   │   ├── 📜 filtered_songs.csv     # Metadata (song_id, valence, arousal)  
│   │   ├── 📜 music_embeddings_258D_normalized.csv  # 258D embeddings (normalized)  
│   │   └── 📜 music_features_with_embeddings.csv    # Combined features & embeddings  
│   ├── 📁 paintings/                 # Painting-related data  
│   │   ├── 🖼️ images/                # Painting image files (JPG)  
│   │   ├── 🖼️ heal_paintings/        # Painting image files (JPG) selected for art therapy
│   │   ├── 📊 features/              # Precomputed painting features  
│   │   ├── 📜 painting_data.csv      # Metadata (ID, valence, arousal)  
│   │   └── 📜 painting_embeddings_258D_normalized.csv  # 258D embeddings (normalized)  
│   │   └── 📜 resnet_similarity_matrix.csv  # 63 × 63 painting similarity matrix filtered by expert  
│   ├── 📜 similarity_matrix.csv                    # 909 × 4105 similarity matrix from contrastive alignement 
│   ├── 📜 heal_similarity_matrix_haydn.csv           # 239 × 63 similarity matrix filtered by expert 
│   ├── 📜 haydn_similarity_matrix.csv                # 909 × 4105 similarity matrix  from V-A vectors
│   ├── 📜 heal_similarity_matrix_mozart.csv        # 239 × 63 similarity matrix filtered by expert 
│   ├── 📜 salieri_similarity_matrix.csv                # 909 × 4105 similarity matrix from Salieri alighnement 
│   ├── 📜 heal_salieri_similarity_matrix.csv           # 239 × 63 similarity matrix filtered by expert 
│   └── 📜 joint_embeddings.csv                     # 128D joint embeddings (music-art alignment)  
│  
├── 📁 feature_extraction/             # Feature extraction scripts  
│   ├── 📁 music/  
│   │   ├── 🎼 feature_extraction_music.py   # Extracts MERT & acoustic features  
│   │   ├── 🎼 reduce_normalize_music.py    # Dimensionality reduction & normalization  
│   ├── 📁 painting/  
│   │   ├── 🎨 feature_extraction_painting.py  # ResNet-based feature extraction  
│   │   ├── 🎨 reduce_normalize_painting.py   # Dimensionality reduction & normalization  
│   ├── 📁 salieri/                         # Salieri feature extraction for cross-modal alignment 
│   │   ├── 🤖 music_salieri_features.py    # Extracts Salieri (GPT-4o) features for music 
│   │   ├── 🤖 painting_vlm_features.py # Extracts VLM (GPT-4V) features for paintings  
│   │   ├── 🤖 multi_modal_music.py  # Combines MERT + VLM features, reduces to 256D
│   │   ├── 🤖 multi_modal_painting.py  # Combines ResNet + Salieri features, reduces to 256D
│   │   ├── 🤖 similarity_computation.py # Computes cross-modal similarity matrix (S_LV) 
├── 📁 flask/                          # Backend API using Flask  
│   ├── 🧠 mozart_engine.py            # Core recommendation engine  
│   ├── 🌐 mozart.py                   # API server for recommendations  
│   ├── ⚙️ engine.py                   # Generic engine class  
│   ├── 🔍 haydn_engine.py               # Haydn baseline  
│   ├── 🔍 haydn.py                      # Haydn API  
│   ├── 🤖 salieri_engine.py               # Large Language Model (Salieri) based RecSys engine  
│   ├── 🤖 salieri.py                      # API endpoint for Salieri-based tasks  
│   ├── 🚀 start.sh                        # Starts the Flask server  
│   ├── 🛑 stop.sh                         # Stops the Flask server  
│   ├── 🔄 restart.sh                      # Restarts the Flask server  
│   ├── 📟 status.sh                       # Checks server status  
│   ├── 📖 README.md                       # Flask setup and usage 
├── 🔥 contrastive_alignment.py  # Contrastive learning for joint embeddings  
├── 🔄 run_feature_pipeline.py   # Orchestrates feature extraction    
├── 📖 README.md                 # Project overview, setup & usage  
├── 📦 requirements.txt          #  Dependencies  
├── 📜 LICENSE                   # Open-source license (CC BY-NC)  
└── 🔒 .gitignore                # Excludes temp files & datasets  
└── ⚙️ .env                      # Store private keys here (OpenAI Key) 
├── 📁 figs/                     # figures   
├── 📁 app/                      # Web application for user study

Setup Instructions

Install Requirements

To install the required dependencies run:

pip install -r requirements.txt

Download Data

Download the data.zip file containing all pre-trained models and features using the following gdown link: run

 gdown "https://drive.google.com/uc?export=download&id=14aehmvmf-MAwRUZqOOBS5xsHHNBR0uPC"

Extract the data.zip file in the project root (mozart-crossmodal/): run

unzip -q data.zip

Usage

All trained models and features are provided via the download. If you want to extract features from scratch, run:

python3 run_feature_pipeline.py

To train the contrastive alignment model, run:

 python3 contrastive_alignment.py

See flask instructions for setting up the services. For furhter usage please see LICENCE