DrugGEN: Target Specific De Novo Design of Drug Candidate Molecules with Graph Transformer-based Generative Adversarial Networks

Discovering novel drug candidate molecules is a fundamental step in drug development. Generative deep learning models can sample new molecular structures from learned probability distributions; however, their practical use in drug discovery hinges on generating compounds tailored to a specific target molecule. Here we introduce DrugGEN, an end-to-end generative system for the de novo design of drug candidate molecules that interact with a selected protein. The proposed method represents molecules as graphs and processes them using a generative adversarial network that comprises graph transformer layers. Trained on large datasets of drug-like compounds and target-specific bioactive molecules, DrugGEN designed candidate inhibitors for AKT1, a kinase crucial in many cancers. Docking and molecular dynamics simulations suggest that the generated compounds effectively bind to AKT1, and attention maps provide insights into the model’s reasoning. Furthermore, selected de novo molecules were synthesized and shown to inhibit AKT1 at low micromolar concentrations in the context of in vitro enzymatic assays. These results demonstrate the potential of DrugGEN for designing target-specific molecules. Using the open-access DrugGEN codebase, researchers can retrain the model for other druggable proteins, provided a dataset of known bioactive molecules is available.

Given a random molecule z, the generator G (below) creates annotation and adjacency matrices of a supposed molecule. G processes the input by passing it through a multi-layer perceptron (MLP). The input is then fed to the graph transformer encoder module. In the graph transformer setting, Q, K and V are the variables representing the annotation matrix of the molecule. After the final products are created in the attention mechanism, both the annotation and adjacency matrices are forwarded to layer normalization and then summed with the initial matrices to create a residual connection. These matrices are fed to separate feedforward layers, and finally, given to the discriminator network D together with real molecules.

• DrugGEN is the default model. The input of the generator is the real molecules (ChEMBL) dataset (to ease the learning process) and the discriminator compares the generated molecules with the real inhibitors of the given target protein.
• DrugGEN-NoTarget is the non-target-specific version of DrugGEN. This model only focuses on learning the chemical properties from the ChEMBL training dataset.

The DrugGEN repository is organized as follows:

data/

• Contains raw dataset files and processed graph data for model training
• encoders/ - Contains encoder files for molecule representation
• decoders/ - Contains decoder files for molecule representation
• Format of raw dataset files should be text files containing SMILES strings only

src/

Core implementation of the DrugGEN framework:

• data/ - Data processing utilities
  • dataset.py - Handles dataset creation and loading
  • utils.py - Data processing helper functions
• model/ - Model architecture components
  • models.py - Implementation of Generator and Discriminator networks
  • layers.py - Contains transformer encoder implementation
  • loss.py - Loss functions for model training
• util/ - Utility functions
  • utils.py - Performance metrics and helper functions
  • smiles_cor.py - SMILES processing utilities

assets/

• Graphics and figures used in documentation
• Contains model architecture diagrams and visualization resources
• Includes images of generated molecules and model animations

results/

• Contains evaluation results and generated molecules
• generated_molecules/ - Storage for molecules produced by the model
• docking/ - Results from molecular docking analyses
• evaluate.py - Script for evaluating model performance

experiments/

• Directory for storing experimental artifacts
• logs/ - Model training logs and performance metrics
• models/ - Saved model checkpoints and weights
• samples/ - Molecule samples generated during training
• inference/ - Molecules generated in inference mode
• results/ - Experimental results and analyses

Scripts:

• train.py - Main script for training the DrugGEN model
• inference.py - Script for generating molecules using trained models
• setup.sh - Script for downloading and setting up required resources
• environment.yml - Conda environment specification

The DrugGEN model requires two types of data for training: general compound data and target-specific bioactivity data. Both datasets were carefully curated to ensure high-quality training.

Compound Data

The general compound dataset provides the model with knowledge about valid molecular structures and drug-like properties:

• Source: ChEMBL v29 compound dataset
• Size: 1,588,865 stable organic molecules
• Composition: Molecules with a maximum of 45 atoms
• Atom types: C, O, N, F, Ca, K, Br, B, S, P, Cl, and As
• Purpose: Teaches the GAN module about valid chemical space and molecular structures

Bioactivity Data

The target-specific dataset enables the model to learn the characteristics of molecules that interact with the selected protein targets:

• Target: Human AKT1 protein (CHEMBL4282)

  • Sources:
    • ChEMBL bioactivity database (potent inhibitors with pChEMBL ≥ 6, equivalent to IC50 ≤ 1 µM)
    • DrugBank database (known AKT-interacting drug molecules)
  • Size: 2,607 bioactive compounds
  • Filtering: Molecules larger than 45 heavy atoms were excluded
  • Purpose: Guides the model to generate molecules with potential activity against AKT1
    
    

• Target: Human CDK2 protein (CHEMBL301)

  • Sources:
    • ChEMBL bioactivity database (potent inhibitors with pChEMBL ≥ 6, equivalent to IC50 ≤ 1 µM)
    • DrugBank database (known CDK2-interacting drug molecules)
  • Size: 1,817 bioactive compounds
  • Filtering: Molecules larger than 38 heavy atoms were excluded
  • Purpose: Guides the model to generate molecules with potential activity against CDK2

Data Processing

Both datasets undergo extensive preprocessing to convert SMILES strings into graph representations suitable for the model. This includes:

• Conversion to molecular graphs
• Feature extraction and normalization
• Encoding of atom and bond types
• Size standardization

For more details on dataset construction and preprocessing methodology, please refer to our paper.

System Requirements

• Operating System: Ubuntu 20.04 or compatible Linux distribution
• Python: Version 3.9 or higher
• Hardware:
  • CPU: Supports CPU-only operation
  • GPU: Recommended for faster training and inference (CUDA compatible)
• RAM: Minimum 8GB, 16GB+ recommended for larger datasets

Installation

1. Clone the repository:

   git clone https://github.com/HUBioDataLab/DrugGEN.git
   cd DrugGEN
   

2. Set up and activate the environment:

   conda env create -f environment.yml
   conda activate druggen
   

3. Run the setup script:

   bash setup.sh
   

   This script will:

   • Download all necessary resources from our Google Drive repository
   • Create required directories if they don't exist
   • Organize downloaded files in their appropriate locations:
     • Dataset files and SMILES files → data/
     • Encoder/decoder files → data/encoders/ and data/decoders/
     • Model weights → experiments/models/
     • SMILES correction files → data/

Now you're ready to start using DrugGEN for molecule generation or model training. Refer to the subsequent sections for specific usage instructions.

Note: The first time you run training or inference, it may take longer than expected as the system needs to create and process the dataset files. Subsequent runs will be faster as they use the cached processed data.

You can use the following commands to train different variants of the DrugGEN model. Select the appropriate example based on your target protein or use case:

python train.py --submodel="[MODEL_TYPE]" \
                --raw_file="data/[GENERAL_DATASET].smi" \
                --drug_raw_file="data/[TARGET_DATASET].smi" \
                --max_atom=[MAX_ATOM_NUM]

python train.py --submodel="DrugGEN" \
                --raw_file="data/chembl_train.smi" \
                --drug_raw_file="data/akt_train.smi" \
                --max_atom=45

python train.py --submodel="DrugGEN" \
                --raw_file="data/chembl_train.smi" \
                --drug_raw_file="data/cdk2_train.smi" \
                --max_atom=38

python train.py --submodel="NoTarget" \
                --raw_file="data/chembl_train.smi" \
                --max_atom=45

Detailed Explanation of Arguments

Below is a comprehensive list of arguments that can be used to customize the training process:

Note: The first time you run inference, it may take longer than expected as the system needs to create and process the dataset files. Subsequent runs will be faster as they use the cached processed data.

Using the Hugging Face Interface (Recommended)

For ease of use, we provide a Hugging Face Space with a user-friendly interface for generating molecules using our pre-trained models.

Local Generation Using Pre-trained Models

Use the following commands to generate molecules with trained models. Select the appropriate example based on your target protein or use case:

python inference.py --submodel="[MODEL_TYPE]" \
                    --inference_model="experiments/models/[MODEL_NAME]" \
                    --inf_smiles="data/[TEST_DATASET].smi" \
                    --train_smiles="data/[TRAIN_DATASET].smi" \
                    --train_drug_smiles="data/[TARGET_DATASET].smi" \
                    --sample_num=[NUMBER_OF_MOLECULES] \
                    --max_atom=[MAX_ATOM_NUM]

python inference.py --submodel="DrugGEN" \
                    --inference_model="experiments/models/DrugGEN-akt1" \
                    --inf_smiles="data/chembl_test.smi" \
                    --train_smiles="data/chembl_train.smi" \
                    --train_drug_smiles="data/akt_train.smi" \
                    --sample_num=1000 \
                    --max_atom=45

python inference.py --submodel="DrugGEN" \
                    --inference_model="experiments/models/DrugGEN-cdk2" \
                    --inf_smiles="data/chembl_test.smi" \
                    --train_smiles="data/chembl_train.smi" \
                    --train_drug_smiles="data/cdk2_train.smi" \
                    --sample_num=1000 \
                    --max_atom=38

python inference.py --submodel="NoTarget" \
                    --inference_model="experiments/models/NoTarget" \
                    --inf_smiles="data/chembl_test.smi" \
                    --train_smiles="data/chembl_train.smi" \
                    --train_drug_smiles="data/akt_train.smi" \
                    --sample_num=1000 \
                    --max_atom=45

Output location:

The generated molecules in SMILES format will be saved to:

experiments/inference/[MODEL_NAME]/inference_drugs.csv

During processing, the model also creates an intermediate file:

experiments/inference/[MODEL_NAME]/inference_drugs.txt

Inference Parameters

The inference process can be customized with various arguments to control how molecules are generated and evaluated:

To evaluate the bioactivity of generated molecules against the AKT1 and CDK2 proteins, we utilize DEEPScreen, a deep learning-based virtual screening tool. Follow these steps to reproduce our bioactivity predictions:

Setting up DEEPScreen

1. Download the DEEPScreen model: Download the pre-trained model from this link

2. Extract the model files:

# Extract the downloaded file
unzip DEEPScreen2.1.zip

Running Predictions

Execute the following commands to predict bioactivity of your generated molecules:

# Navigate to the DEEPScreen directory
cd DEEPScreen2.1/chembl_31

# Run prediction for AKT target
python 8_Prediction.py AKT AKT

Output

Prediction results will be saved in the following location:

DEEPScreen2.1/prediction_files/prediction_output/

These results include bioactivity scores that indicate the likelihood of interaction between the generated molecules and the AKT1 target protein. Higher scores suggest stronger potential binding affinity.

The system is trained to design effective inhibitory molecules against the AKT1 protein, which has critical importance for developing treatments against various types of cancer. SMILES notations of the de novo generated molecules from DrugGEN models, along with their deep learning-based bioactivity predictions (DeepScreen), docking and MD analyses, and filtering outcomes, can be accessed under the results folder. The structural representations of the final selected molecules are depicted in the figure below.

Fig. 2. Promising de novo molecules to effectively target AKT1 protein (generated by DrugGEN model), selected via expert curation from the dataset of molecules with sufficiently low binding free energies (< -8 kcal/mol) in the molecular docking experiment.

• 15/09/2025: Peer-reviewed DrugGEN article is published.
• 12/03/2025: DrugGEN v2.0 is released.
• 26/07/2024: DrugGEN pre-print is updated for v1.5 release.
• 04/06/2024: DrugGEN v1.5 is released.
• 30/01/2024: DrugGEN v1.0 is released.
• 15/02/2023: Our pre-print is shared here.
• 01/01/2023: DrugGEN v0.1 is released.

@article{DrugGEN2025,
  title={Target Specific De Novo Design of Drug Candidate Molecules with Graph Transformer-based Generative Adversarial Networks},
  author={Ünlü, Atabey and Çevrim, Elif and Yiğit, Melih Gökay and Sarıgün, Ahmet and Çelikbilek, Hayriye and Bayram, Osman and Kahraman, Deniz Cansen and Olğaç, Abdurrahman and Rifaioğlu, Ahmet Sureyya and Banoğlu, Erden and Doğan, Tunca},
  journal={Nature Machine Intelligence},
  pages={1--17},
  year={2025},
  publisher={Nature Publishing Group},
  doi={10.1038/s42256-025-01082-y}
}

Ünlü, A., Çevrim, E., Yigit, M.G., Sarıgün, A., Çelikbilek, H., Bayram, O., Kahraman, D.C., Olğaç, A., Rifaioğlu, A., Banoglu, E., Doğan, T. (2025). Target Specific De Novo Design of Drug Candidate Molecules with Graph Transformer-based Generative Adversarial Networks. Nature Machine Intelligence, 7(9), 1524-1540 DOI:10.1038/s42256-025-01082-y.

For the static v2.0 of the repository, you can refer to the following DOI: 10.5281/zenodo.15014579.

In each file, we indicate whether a function or script is imported from another source. Here are some excellent sources from which we benefit from:

• Molecule generation GAN schematic was inspired from MolGAN.
• MOSES was used for performance calculation (MOSES Script are directly embedded to our code due to current installation issues related to the MOSES repo).
• PyG was used to construct the custom dataset.
• Graph Transformer Encoder architecture was adopted from Dwivedi & Bresson (2021) and Vignac et al. (2022) and modified.

Copyright (C) 2026 HUBioDataLab

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.