📊 Scaffold-based Metrics for Molecular Generators

Purpose: This repository provides an implementation of recall-based metrics — RS, SED and ASER — for evaluating molecular generators. These metrics help assess the diversity, novelty, and structural relevance of generated compounds.

🚀 Features

Implementation of three key recall metrics:

▪ scaffold Recovery Score (RS):

quantifies the ability of the generator to produce molecules containing scaffolds that occur in biologically active compounds. It is calculated as the ratio of the number of unique active scaffolds in the Output Set (OS) (UAs_OS) to the number of unique active scaffolds in the Recall Set (RS) (UAS_RS). RS value ranges between 0 (no unique active scaffolds from RS were found in the OS) and 1 (all unique active scaffolds from RS were found in the OS).

$$RS = \frac{|\text{Unique Active scaffolds in the OS}|} {|\text{Unique Active scaffolds in the RS}|}$$

• The RS metric is a key measure that quantifies the ability of a molecular generator to reproduce biologically active scaffolds. As such, RS is analogous to the recall metric used to evaluate classification tasks in machine learning. RS is effective for a generate-then-select methodology. Selecting the bioactive compounds out of a large set of generated molecules is hard, but if they are not generated at all, this is impossible.

▪ SEt scaffold Diversity(SED):

reflects the structural diversity of the generated set. It is calculated as the percentage of unique scaffolds (Us_OS) in the Output Set relative to the total number of compounds in the Output Set (c_OS). A value of 1 indicates that every scaffold in the OS is unique, while 0 means no valid scaffolds were generated.

$$SED = \frac{|\text{Unique scaffolds in the OS}|} {|\text c_{OS}|}$$

• SED enables striking a balance between exploration and exploitation. A high SED value indicates that the generator explores diverse regions of chemical space by discovering novel scaffolds, which is essential for expanding the chemical space. On the other hand, a low SED value suggests a focus on exploiting similar scaffolds, refining specific chemical structures. Adjusting the generator parameters based on the SED value enables fine-tuning the balance between exploring diverse molecular structures and exploiting specific scaffolds.

▪ Absolute SEt Scaffold Recall (ASER):

indicates how effectively the generator explores the target region of chemical space. It is calculated as the ratio of the number of active scaffolds in the OS (As_OS) to the total number of compounds in the Output Set (c_OS). The ASER metric starts at 0, indicating that no generated compounds in the output set (OS) contain an active scaffold. Higher ASER values reflect an increasing frequency of active scaffolds within the generated compounds. Importantly, the metric can exceed 1 because some compounds may contain multiple active scaffolds. In this sense, it is akin to the precision metric that is used to evaluate classification tasks in machine learning.

$$ASER = \frac{|\text{Active scaffolds in the OS}|} {|\text c_{OS}|}$$

• A higher ASER value means the generator is better at making molecules similar to those known to be biologically active. This suggests that the generator is exploring the right chemical space for drug discovery — beyond what it was originally trained on.

✔ Supports multiple molecular generators: Molpher, REINVENT, DrugEx, GB_GA, and AddCarbon.
✔ Works with Murcko and CSK scaffold analysis.
✔ Normalization and visualization tools for result interpretation.

In this analysis, we prepared data for two different receptors (Glucocorticoid receptor, Leukocyte elastase) and two different splits (dissimilarity split and similarity split)

---

📥 Installation

To use the scaffold-based metrics, clone this repository and install the required dependencies:

git clone https://github.com/filvaleriia/scaffold-based-metrics.git
cd scaffold-based-metrics
conda env create -f environment.yml
conda activate scaffold-based-metrics-env

Or install the following packages manually:

• rdkit
• pandas
• JupyterLab
• scikit-learn
• scikit-learn-extra
• seaborn
• matplotlib

📂 Data Requirements

All required data are available on Zenodo: 10.5281/zenodo.18678356
After download, place the entire data/ folder into the root directory of this repository.

⚠️ The folder has a size of ~9 GB.
All scripts are configured to work with this folder automatically.

Folder structure

• data/input_recall_sets/ – Input Sets and Recall Sets for Glucocorticoid receptor and Leukocyte elastase
• data/nuclear_receptor/ – original ChEMBL31 data for Glucocorticoid receptor, used in preprocessing for Input and Recall Sets
• data/protease/ – original ChEMBL31 data for Leukocyte elastase, used in preprocessing for Input and Recall Sets
• data/information_about_clusters/ – statistics for 5 clusters across 2 different splits
• data/results/ – results for 5 different generators (some with different settings), organized as follows: s

data/results/
├── Glucocorticoid_receptor/
├── Leukocyte_elastase/
      ├── csk_scaffolds/
      ├── murcko_scaffolds/
          ├── dis/
          ├── sim/
              ├── Molpher/
              ├── DrugEx_GT_epsilon_0.6/
              ...

🛠 Usage

1️⃣ Running Metric Calculations

Modify the calculate_metrics function in your notebook or script to specify:

• Scaffold type: csk or murcko
• Cluster type: dis or sim
• Target receptor: Glucocorticoid_receptor or Leukocyte_elastase
• Molecular generator: Molpher, REINVENT, DrugEx_GT_epsilon_0.1, DrugEx_GT_epsilon_0.6, DrugEx_RNN_epsilon_0.1, DrugEx_RNN_epsilon_0.6,GB_GA_mut_r_0.01, GB_GA_mut_r_0.5, GB_GA_log_p_mut_r_0.01,GB_GA_log_p_mut_r_0.5, addcarbon.

To use these metrics as explained below, you need to save your OutputSet to the folder. The Output must have only one column with smiles, without header. /data/output_sets/{receptor_name}/{generator_name}/cOS_{generator_name}_{type_cluster}_{number_of_cluster}_one_column.csv.

After that, you can simply use the generator name. The results can be found in the folder /data/results/{receptor_name}/{generator_name}/

Example usage in a jupyter notebook:

from src import metrics

type_cluster = 'dis' #options: 'dis'|'sim' 
type_scaffold = 'csk' #options: 'csk'|'murcko'
generator = 'Molpher' #options: 'Molpher' | 'REINVENT' | 'DrugEx_GT_epsilon_0.1' | 'DrugEx_GT_epsilon_0.6' | 'DrugEx_RNN_epsilon_0.1' | 'DrugEx_RNN_epsilon_0.6' | 'GB_GA_mut_r_0.01' | 'GB_GA_mut_r_0.5' | 'GB_GA_log_p_mut_r_0.01' |'GB_GA_log_p_mut_r_0.5' | 'addcarbon'
receptor = 'Glucocorticoid_receptor' #options: 'Glucocorticoid_receptor'|'Leukocyte_elastase'
ncpus = 4
data_folder = '' # Path to your data folder. No need to set this if your data is in the repository root (default is data_folder = '').

mt = metrics.Metrics(type_cluster, type_scaffold, generator, receptor,  ncpus)
result = mt.calculate_metrics()

Example usage in bash (see also run_calculation_metrics.sh):

python3 src/metrics.py --type_cluster dis --type_scaffold csk --generator Molpher --receptor Glucocorticoid_receptor --data_folder data_folder --num_cpu 3 

The second option how use generator it define the path to folder where Recall Set and Output Set or you can look for examples run_calculation_metrics_define_path.sh. The results can be found in the folder /data/results/{receptor_name}/{generator_name}/

python3 src/metrics_define_path.py \
    --type_cluster sim \
    --type_scaffold csk \
    --generator Molpher_125k \
    --receptor Glucocorticoid_receptor \
    --recall_set_path /recall_metrics/data/input_recall_sets/Glucocorticoid_receptor \
    --output_set_path /recall_metrics/data/output_sets/Glucocorticoid_receptor/Molpher_125k \
    --data_folder ../
    --ncpus 1

The last option is to run it with your own data. For this purpose, the script src/metrics_own_data.py exists or you can look for examples run_own_single_metrics_calculation.sh, the results will be stored in the results folder in the main directory:

python3 src/metrics_own_data.py \
    --type_scaffold csk \
    --generator Molpher_125k \
    --recall_set_path /recall_metrics/data/input_recall_sets/Glucocorticoid_receptor/cRS_Glucocorticoid_receptor_dis_0.csv \
    --output_set_path /recall_metrics/data/output_sets/Glucocorticoid_receptor/Molpher_62.5k/cOS_Molpher_62.5k_dis_0_one_column.csv \
    --data_folder ../
    --ncpus 1

🔬⚙️ Project overview

In this project, we applied three metrics to compare different molecular generators as well as different generator settings in the optimization process.
To demonstrate the target-independence of these metrics, we evaluated five molecular generators on two pharmacological targets:

1. Molpher – atom-based molecular generator using an evolutionary strategy
2. DrugEx – neural network–based generator. Two architectures and two different epsilon parameter settings (balancing exploration and exploitation) were applied:
   • DrugEx RNN – recurrent neural network–based generator
   • DrugEx GT – graph transformer–based generator
3. REINVENT – neural network–based generator using a pretrained model
4. GB_GA – atom-based generator using a genetic algorithm. Two different mutation_rate parameters were tested (1% and 50%).
   • GB_GA – generator with a custom scoring function, checking only molecules with an SA score lower than 6.
   • GB_GA_log_p – generator with the logP scoring function, based on the original paper.
5. AddCarbon – baseline generator for result verification

To cover different structural aspects, we employed two scaffold definitions: CSKs and Murcko scaffolds (RDKit definition).

For data preparation, we retrieved compounds from ChEMBL31 for the Glucocorticoid receptor (GR) and Leukocyte elastase (LE), selecting only active molecules using activity thresholds (<=100 nM for GR, <=1000 nM for LE).
The compounds were converted to CSKs and clustered into five groups. From these clusters, we constructed an Input Set (IS) and a Recall Set (RS) using two approaches:

• dissimilarity split (a)
• similarity split (b)

The rationale was to explore metric limitations:

• Similarity split → IS and RS are structurally close (upper limit of the metric).
• Dissimilarity split → IS and RS are structurally distant (lower limit of the metric).

In total, we created 10 different sets (5 for dissimilarity and 5 for similarity). Each generator was run with the IS to produce corresponding Output Sets (OS).
Metrics were calculated for each OS using the RS. To avoid bias from outliers or unusual generator behavior, the results were averaged across the five runs.

Finally, we compared the average metric values across the two split strategies, the two targets, and the two scaffold definitions. For visualization, we prepared normalized heatmaps.

---

📏 Metrics Interpretation Thresholds

To facilitate interpretation of differences between generators, we defined effect-size thresholds for each metric and scaffold type.
These thresholds categorize the absolute difference Δ between generators into four qualitative levels:

• Trivial (≤ 25%)
• Small (25–50%)
• Moderate (50–75%)
• Large (> 75%)

Thresholds were derived from bootstrap-based pairwise comparisons of generator performance.

Threshold values

Metric	Scaffold	Trivial Δ	Small Δ	Moderate Δ	Large Δ
RS	CSK	Δ ≤ 0.072	0.072 < Δ ≤ 0.189	0.189 < Δ ≤ 0.332	Δ > 0.332
RS	Murcko	Δ ≤ 0.044	0.044 < Δ ≤ 0.136	0.136 < Δ ≤ 0.212	Δ > 0.212
SED	CSK	Δ ≤ 0.045	0.045 < Δ ≤ 0.090	0.090 < Δ ≤ 0.159	Δ > 0.159
SED	Murcko	Δ ≤ 0.076	0.076 < Δ ≤ 0.175	0.175 < Δ ≤ 0.299	Δ > 0.299
ASER	CSK	Δ ≤ 0.004	0.004 < Δ ≤ 0.011	0.011 < Δ ≤ 0.018	Δ > 0.018
ASER	Murcko	Δ ≤ 0.001	0.001 < Δ ≤ 0.004	0.004 < Δ ≤ 0.007	Δ > 0.007

These thresholds allow quantitative comparison of generators beyond raw metric values.

---

📊 Results

Here we summarize the main outcomes of our study.

Combined results in a single overview plot

The heatmaps summarize the performance of all generators across two receptors, two scaffold definitions (CSK and Murcko), and two data splits (dissimilarity and similarity). Each column represents one experimental setting.

Colors indicate the magnitude of performance differences relative to the best generator in that column, categorized using metric-specific interpretation thresholds (Trivial, Small, Moderate, Large). This enables comparison across metrics with different scales.

Within each column:

• Bold values mark the best-performing generator
• Color intensity reflects how far other generators are from the best result
• ASER values are reported in units of ×10⁻²

Overall, the DrugEx Graph Transformer (ε = 0.6) achieved the strongest performance across most settings, followed by GB_GA (mutation rate = 50%) and DrugEx RNN (ε = 0.6).
The AddCarbon baseline consistently showed the weakest results.

Similarity splits generally produced higher scores than dissimilarity splits, reflecting the greater structural overlap between Input and Recall Sets.