Introduction

We introduce Charm , a novel tokenization approach that preserves Composition, High-resolution, Aspect Ratio, and Multi-scale information simultaneously. By preserving critical aesthetic information, Charm achieves significant performance improvement across different image aesthetic assessment datasets.

The capacity of Vision transformers (ViTs) to handle variable-sized inputs is often constrained by computational complexity and batch processing limitations. Consequently, ViTs are typically trained on small, fixed-size images obtained through downscaling or cropping. While reducing computational burden, these methods result in significant information loss, negatively affecting tasks like image aesthetic assessment. We introduce Charm , a novel tokenization approach that preserves Composition, High-resolution, Aspect Ratio, and Multi-scale information simultaneously. Charm prioritizes high-resolution details in specific regions while downscaling others, enabling shorter fixed-size input sequences for ViTs while incorporating essential information. Charm is designed to be compatible with pre-trained ViTs and their learned positional embeddings. By providing multiscale input and introducing variety to input tokens, Charm improves ViT performance and generalizability for image aesthetic assessment. We avoid cropping or changing the aspect ratio to further preserve information. Extensive experiments demonstrate significant performance improvements on various image aesthetic and quality assessment datasets (up to 8.1 %) using a lightweight ViT backbone.

Performance improvement across various datasets

Charm not only improves the performance of image aesthetic/quality assessment models but also accelerates their convergence.

Performance improvement across different IAA and IQA datasets by replacing the standard tokenization with Charm. Dinov2-small is employed for all experiments except for *, which shows the ViT-small backbone. The epoch at which each model achieves its highest validation performance across different datasets. Charm generally leads to faster convergence.

Posters

CVPR poster
VSS poster

Video Presentation

here

Standard Installation

Clone the repository locally and install with:

git clone https://github.com/FBehrad/Charm.git
pip install -r requirements.txt

Model checkpoints

Pretrained models are available at HuggingFace 🤗.

Quick inference

:globe_with_meridians: Google Colab: Colab

:desktop_computer: On your system: Please read this page.

Training

Configure the values in the config file and execute train.py to start training.

Citation

If you find this repo useful in your research, please star ⭐ this repository and consider citing 📝:

@InProceedings{Behrad_2025_CVPR,
    author    = {Behrad, Fatemeh and Tuytelaars, Tinne and Wagemans, Johan},
    title     = {Charm: The Missing Piece in ViT Fine-Tuning for Image Aesthetic Assessment},
    booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
    month     = {June},
    year      = {2025},
    pages     = {7815-7824}
}

Acknowledgments

Funded by the European Union (ERC AdG, GRAPPA, 101053925, awarded to Johan Wagemans) and the Research Foundation-Flanders (FWO, 1159925N, awarded to Fatemeh Behrad).

Follow me on LinkedIn