VDJdb Motif Clustering

This repository now keeps two clustering pipelines side by side:

• tcrnet/: the R Markdown workflow based on VDJtools/TCRNET
• redcea/: the new Python package based on TCRemP + REDCEA-style shared-background clustering

The goal is to keep the TCRNET pipeline runnable as before while letting the new implementation evolve independently.

Repository Layout

vdjdb-motifs/
  tcrnet/   TCRNET clustering workflow
  redcea/   installable Python package
  results/  generated tabular outputs
  figures/  generated PDF figures
  scripts/  run and setup helpers

Simple Commands

If you just want one command per pipeline:

./scripts/setup.sh tcrnet
./scripts/run_tcrnet.sh

./scripts/setup.sh redcea
./scripts/run_redcea.sh

run_redcea.sh launches TRA and TRB in parallel. setup.sh redcea requires a working conda installation because it creates and updates the vdjdb-redcea environment automatically.

Thread count is configurable for both pipelines:

TCRNET_NPROC=16 ./scripts/run_tcrnet.sh
REDCEA_NPROC=16 ./scripts/run_redcea.sh

If you want to prepare everything in one go:

./scripts/setup.sh all

Assumptions:

• vdjdb_release/vdjdb.slim.txt already exists before running either pipeline.
• the repository is run in a Unix-like environment.

Required command-line tools:

• shared: bash
• TCRNET: Rscript, python, VDJtools, wget, unzip
• REDCEA: conda, curl, tar

Notes on these requirements:

• scripts/setup.sh tcrnet installs the needed R packages automatically, but it does not install VDJtools itself.
• scripts/run_tcrnet.sh now also runs a small Python post-processing step that generates per-epitope HTML visualizations.
• scripts/setup.sh redcea creates and updates the vdjdb-redcea conda environment automatically.

Background Data

• tcrnet backgrounds are public and downloaded automatically from Zenodo by scripts/fetch_tcrnet_backgrounds.sh.
• redcea needs two prepared background files:
  • AIRR repertoire table
  • embedding parquet file
• The repository now includes a downloader script for redcea.
• By default it downloads https://zenodo.org/records/19520535/files/redcea_bg.gz.
• You can still pass a different Zenodo archive URL as the first argument if needed.
• scripts/setup.sh redcea <url> passes that URL through to the REDCEA background fetch step.
• scripts/install_redcea.sh creates a separate conda environment and installs this repository's redcea package together with its declared git-based dependencies.

REDCEA Zenodo Files

If you want redcea to start quickly from downloaded assets, the default Zenodo bundle is:

• https://zenodo.org/records/19520535/files/redcea_bg.gz

Required per chain:

• tra_background_100k.tsv
• tra_background_100k_embeddings.parquet
• trb_background_100k.tsv
• trb_background_100k_embeddings.parquet

What these do:

• *_background_100k.tsv is the actual background repertoire input passed to --background-airr.
• *_background_100k_embeddings.parquet is the precomputed background embedding input passed to --background-embedding after renaming during install.

What you do not need to upload:

• background transform caches
• per-epitope sample embeddings
• per-run cluster tables
• HTML visualizations
• cluster_members_*.txt

Practical recommendation:

• if you run both TRA and TRB, include both required chain-specific AIRR/embedding pairs in the bundle
• ./scripts/run_redcea.sh uses the default local paths above directly

UMAP tuning note:

• the REDCEA plotting layout now accepts --umap-n-neighbors and --umap-min-dist
• scripts/run_redcea.sh keeps separate per-chain defaults via REDCEA_UMAP_N_NEIGHBORS_TRA / REDCEA_UMAP_MIN_DIST_TRA and REDCEA_UMAP_N_NEIGHBORS_TRB / REDCEA_UMAP_MIN_DIST_TRB
• if needed, a global REDCEA_UMAP_N_NEIGHBORS or REDCEA_UMAP_MIN_DIST environment override still takes precedence for ad hoc runs
• if sample clusters form detached islands far from the grey background cloud, try increasing these values, for example --umap-n-neighbors 50 --umap-min-dist 0.4
• changing either value invalidates the previous plotting transform on purpose, so the cached background transform is recomputed automatically

Notes

• TCRNET outputs now go to results/tcrnet/ and keep the standard names cluster_members.txt and motif_pwms.txt.
• TCRNET HTML visualizations are collected in results/tcrnet/viz/.
• TCRNET PDF figures go to the repository-level figures/ directory.
• TCRNET respects TCRNET_NPROC for parallel R steps.
• REDCEA writes both chains into results/redcea/.
• REDCEA HTML visualizations are collected in results/redcea/viz/.
• REDCEA cluster_members_TRA.txt and cluster_members_TRB.txt are written directly into results/redcea/.
• REDCEA uses vdjdb_release/vdjdb.slim.txt as the default VDJdb input table.
• REDCEA respects REDCEA_NPROC; when REDCEA_CHAIN=both, TRA and TRB are still launched in parallel as separate jobs.