Benchmarking 3D Structure-Based Generative Drug Design

January 26, 2026 · View on GitHub

Are We There Yet? Offical code base of Journal of Medicinal Chemistry paper "Structure-Based Generation of 3D Small-Molecule Drugs: Are We There Yet?" (2025). A comprehensive benchmarking study for 3D structure-based generative models, focusing on the chemical plausibility of generated structures.

This repository provides novel metrics and tools to evaluate the quality of molecules generated by structure-based drug design (SBDD) models, with emphasis on ring system and scaffold coverage.

Table of Contents

Overview

This benchmarking framework evaluates generative models for drug design using two key metrics:

  1. Ring System Frequency - Measures how often generated ring systems appear in known drug-like molecules (ChEMBL database)
  2. BM Scaffold Coverage - Evaluates whether Bemis-Murcko scaffolds exist in reference databases

These metrics help assess whether generated molecules are chemically plausible.

Installation

# Clone the repository
git clone https://github.com/Kartinaa/Benchmarking_gene_model.git
cd Benchmarking_gene_model

# Create and activate conda environment
conda env create -f environment.yml
conda activate cheminfo

Dependencies

PackageVersion
Python3.11.9
RDKit2023.9.6
useful-rdkit-utils0.56
pandaslatest
jobliblatest

Project Structure

Benchmarking_gene_model/
├── utils/                      # Core utility modules
│   ├── filtering_functions.py  # Molecular filtering (ring validation)
│   └── ring_system_calculator.py # Ring system frequency analysis
├── data/                       # Reference datasets
│   └── three_proteins/         # Protein-specific benchmarking data
├── 3D_generative_SBDD/         # 3D SBDD model outputs
├── Decompdiff/                 # DecompDiff model outputs
├── Decomopt/                   # DecompOpt model outputs
├── Molsnapper/                 # MolSnapper model outputs
├── PMDM/                       # PMDM model outputs
├── Pocket2mol/                 # Pocket2Mol model outputs
├── Targetdiff/                 # TargetDiff model outputs
├── BMScaffoldChecker.py        # BM scaffold validation tool
├── BM_scaffold_checker_ZINC22.py # Large-scale scaffold checker
├── BM_scaffolds_occurance.ipynb # BM scaffold analysis notebook
├── Picture_drawing.ipynb       # Figure generation
└── environment.yml             # Conda environment specification

Quick Start

1. Analyze Ring System Frequencies

# Run ring system analysis on a SMILES file
python utils/ring_system_calculator.py -i molecules.smi -o results.csv -v

2. Check BM Scaffold Coverage

from BMScaffoldChecker import BMScaffoldChecker

# Initialize with reference scaffolds
checker = BMScaffoldChecker("path/to/scaffolds.smi")

# Check a file
percentage, matched, total = checker.check_file("generated_molecules.smi")
print(f"Scaffold coverage: {percentage:.2f}% ({matched}/{total})")

Core Tools

Ring System Calculator (utils/ring_system_calculator.py)

Analyzes ring system frequencies using the ZINC database:

# Basic usage
python utils/ring_system_calculator.py -i input.smi -o output.csv

# With verbose output
python utils/ring_system_calculator.py -i input.smi -o output.csv -v

# Debug mode
python utils/ring_system_calculator.py -i input.smi -o output.csv -vv

BM Scaffold Checker (BMScaffoldChecker.py)

Checks molecules against a reference set of Bemis-Murcko scaffolds:

from BMScaffoldChecker import BMScaffoldChecker

# Initialize (uses parallel loading)
checker = BMScaffoldChecker("scaffolds.smi", n_jobs=8)

# Check individual SMILES
is_known = checker.check_smiles("CCO")

# Check entire file
pct, matched, total = checker.check_file("molecules.smi")

Large-Scale Scaffold Checker (BM_scaffold_checker_ZINC22.py)

Optimized for very large scaffold files (e.g., BM scaffold from ZINC22):

# Build cache from large scaffold file
python BM_scaffold_checker_ZINC22.py scaffolds.smi --cache scaffolds.pkl.gz -v

# Check a single file
python BM_scaffold_checker_ZINC22.py scaffolds.smi --cache scaffolds.pkl.gz --once query.smi

# Interactive mode
python BM_scaffold_checker_ZINC22.py scaffolds.smi --cache scaffolds.pkl.gz

Metrics

Ring System Frequency Metric

Evaluates whether ring systems in generated molecules appear in known drug-like compounds:

  • min_freq = -1: Novel ring system (not in ZINC database)
  • min_freq > 100: Common ring system (appears frequently)
  • 0 < min_freq ≤ 100: Rare but known ring system

For detailed information, see:

BM Scaffold Coverage Metric

Measures what percentage of generated molecule scaffolds exist in reference databases:

  • Higher coverage suggests more chemically plausible molecules
  • See BM_scaffolds_occurance.ipynb for examples

Data Availability

DatasetDescriptionLocation
Ring System FrequenciesZINC20/ZINC22 drug-like moleculesdata/ folder
BM Scaffolds (ZINC20 and 22)Bemis-Murcko scaffoldsHugging Face

Generated Molecule Datasets

Each algorithm folder contains:

  • Generated molecule SMILES files
  • Ring_system.ipynb - Analysis notebook with all metrics applied

Reproducing Results

Apply Metrics to Generated Molecules

  1. Ring System Analysis: Open Ring_system.ipynb in any algorithm folder
  2. BM Scaffold Analysis: See BM_scaffolds_occurance.ipynb
  3. Generate Figures: Run Picture_drawing.ipynb

Apply Metrics to Your Own Molecules

import pandas as pd
from rdkit import Chem
import useful_rdkit_utils as uru

# Load your molecules
df = pd.read_csv("your_molecules.smi", sep="\t", names=["SMILES", "Name"])
df["mol"] = df["SMILES"].apply(Chem.MolFromSmiles)

# You can decided if further standardization of SMILES is needed, just keep consistent.

# Compute ring system frequencies
ring_lookup = uru.RingSystemLookup()
df["ring_info"] = df["mol"].apply(ring_lookup.process_mol)
df[["min_ring", "min_freq"]] = df["ring_info"].apply(
    uru.get_min_ring_frequency
).tolist()

Contact

  • Author: Bo Yang
  • Email: yang2531@purdue.edu
  • Issues: Please create an issue on GitHub for bugs or feature requests

License

This project is licensed under the terms specified in the LICENSE file.