Reproducing the SynCraft Experiments

May 25, 2026 · View on GitHub

This guide describes how to reproduce each numerical result reported in the SynCraft paper. The repository is code-only; the inference outputs, test sets, and analysis tables are released separately on the paper's data archive.

Data archive

Most reproducer scripts expect a sibling data_release/ directory laid out as:

data_release/
├── cliff_pair_corpus/cliff_pairs.json
├── test_sets/
│   ├── P2M_unsynthesizable.json
│   ├── ResGen_unsynthesizable.json
│   └── RIPK1_7YDX_unsynthesizable.json
├── multi_llm_inference/                        # P2M/ResGen × {Gemini, GPT-5.2, DSv4-Pro, DSv4-Flash}
├── plip_retention/
│   ├── ripk1_pairs.csv
│   ├── ripk1_vina_per_source.csv
│   ├── mpro_47_sources.csv
│   └── mpro_pairs_47subset.csv
└── mpro_skill_demo/
    ├── 47_sources.csv
    ├── mpro_vina_paired.csv
    └── summary.json

Every reproducer script reads $DATA_ROOT (default: ../data_release sibling of the SynCraft-Core checkout). Override with DATA_ROOT=/path/to/data_release bash scripts/reproduce_<...>.sh. See the data archive's own README.md for per-file schemas and reproduction recipes.

Index of experiments

ExperimentScriptWhat it measures
Main result (Table 1)scripts/reproduce_table1.shSynCraft rescue rate on P2M and ResGen unsynthesizable molecules
RAG ablationscripts/reproduce_rag_ablation.shRandom-RAG vs similarity-RAG + LLM-as-judge alignment scoring
Edit-path auditscripts/reproduce_edit_path_audit.shUniqueness of graph-matching ground truth (LLM-direct, n=100)
Baseline coveragescripts/reproduce_baseline_coverage.shSynCraft on LIGAN / AR / DiffSBDD subsets
Binding-mode retentionscripts/reproduce_binding_mode_retention.shPer-source PLIP recall + Jaccard, source vs edited poses

What's in this repo vs. the data archive

This repository (code-only, MIT-licensed):

  • src/ — Python modules used by every experiment
  • scripts/ — runnable shell wrappers + dock_batch.py
  • assets/ — input fixtures (unsolved.json, reasoning.json, RIPK1.txt)
  • vina/ — pre-prepared receptor files for the five demo targets (7L12, 7YDX, 2YAC, 5mo4, 8F2B)
  • skill/ — Anthropic Agent Skill packaging (PyPI-publishable)
  • notebooks/ — analysis notebooks

Paper data archive (DOI in paper):

  • Cliff-pair RAG corpus
  • Unsynthesizable test sets (P2M, ResGen, RIPK1)
  • Multi-LLM `.jsonl$ \text{inference} \text{outputs} (\text{P2M} / \text{ResGen} \times \text{four} \text{backends})
  • \text{Per}-\text{target} \text{PLIP} \text{retention} + \text{Vina} \text{CSVs} (\text{RIPK1}, \text{Mpro})
  • \text{Mpro} \text{skill} \text{demo} \text{end}-\text{to}-\text{end} \text{trace}

\text{Setup} (\text{one}-\text{time})

$``bash git clone https://github.com/catalystforyou/SynCraft-Core.git cd SynCraft-Core conda create -n syncraft python=3.12 -y conda activate syncraft conda install -c conda-forge rdkit openbabel biopython pdb2pqr -y pip install vina meeko plip openai litellm loguru pydantic httpx


## API access

Two reasoning-LLM backends are first-class supported.

**Gemini (closed API):**
```bash
export GEMINI_API_KEY=...

DeepSeek-V4 (MIT open-weight; self-hosted via vLLM/SGLang):

export DSV4_BASE_URL=http://localhost:8443/ds-eval
export DSV4_API_KEY=...
export NO_PROXY=127.0.0.1,localhost

DeepSeek-V4 weights are public on HuggingFace (deepseek-ai/DeepSeek-V4-Pro, V4-Flash), so the pipeline remains reproducible long-term, independent of any commercial API.

Run any experiment

# Main result
bash scripts/reproduce_table1.sh

# RAG ablation (random-RAG + LLM-as-judge)
bash scripts/reproduce_rag_ablation.sh

# Edit-path audit on n=100 cliff pairs
N=100 bash scripts/reproduce_edit_path_audit.sh

# Additional generative-model baselines (LIGAN, AR, DiffSBDD)
bash scripts/reproduce_baseline_coverage.sh

# Binding-mode retention (per target)
# FAST mode (default) — recomputes per-source retention statistics from the shipped pair CSVs:
bash scripts/reproduce_binding_mode_retention.sh 7YDX      # RIPK1
bash scripts/reproduce_binding_mode_retention.sh 7L12      # Mpro
# FULL mode — re-dock + PLIP from your own pose-pair candidate CSV:
MODE=full bash scripts/reproduce_binding_mode_retention.sh 7YDX

Citation

@article{li2026syncraft,
  title  = {SynCraft: Guiding Large Language Models to Predict Edit Sequences for Molecular Synthesizability Optimization},
  author = {Li, Junren and {others} and Lai, Luhua},
  journal= {Nature Machine Intelligence},
  year   = {2026},
}