Abstract: We propose Anomagic, a zero-shot anomaly generation method that produces semantically coherent anomalies without requiring any exemplar anomalies. By unifying both visual and textual cues through a crossmodal prompt encoding scheme, Anomagic leverages rich contextual information to steer an inpainting‐based generation pipeline. A subsequent contrastive refinement strategy enforces precise alignment between synthesized anomalies and their masks, thereby bolstering downstream anomaly detection accuracy. To facilitate training, we introduce AnomagicDataset, a collection of 12,987 anomaly–mask–caption triplets assembled from 13 publicly available datasets, where captions are automatically generated by multimodal large language models using structured visual prompts and template‐based textual hints. Extensive experiments demonstrate that Anomagic trained on AnomagicDataset can synthesize more realistic and varied anomalies than prior methods, yielding superior improvements in downstream anomaly detection. Furthermore, Anomagic can generate anomalies for any normal‐category image using user‐defined prompts, establishing a versatile foundation model for anomaly generation.

🤗 Interactive Demo

Try the live demo on Hugging Face Spaces:

Note: Click the badge or image to open the interactive interface on Hugging Face.

📚 Anomaly Generation Methods (Reference Library)

Provides a hierarchical taxonomy of key papers on anomaly generation methods, surpassing flat lists with fine-grained subcategories that delineate emerging hotspots.
GitHub Repository

🏭 Industrial Visual Anomaly Detection Datasets (AnomVerse)

Dataset	Hugging Face Link
BTech	Hugging Face
DAGM_anomaly_detection	Hugging Face
MTD	Hugging Face
MPDD	Hugging Face
MulSen_AD	Hugging Face
eyecandies_preprocessed	Hugging Face
KolektorSDD2	Hugging Face
mvtec_ad_2	Hugging Face
MANTA_TINY_256	Hugging Face
AITEX	Hugging Face
VisA_20220922	Hugging Face
mvtec	Hugging Face
mvtec3d	Hugging Face

💻 Requirements

• Python 3.8.20
• CUDA 11.6+

🛠️ Installation

conda env create -f env.yml

Activate the environment:

conda activate Anomagic

📥 Download Models

You can download the required pretrained models from here. To run the training script, you should also download the following models and place them in the specified directories:

• IP-Adapter weights (ip-adapter_sd15.bin): Place in the models/ folder.
• Image Encoder weights (model.safetensors and pytorch_model.bin): Place in the models/image_encoder/ folder.
• Checkpoints: (longclip-B.pt and longclip-L.pt): Place in the checkpoints/ folder.

🚀 Workflow

Follow these steps to prepare data, train models, and generate synthetic anomaly images. The pipeline is designed for VisA and MVTec datasets.

1. Dataset JSON Generation

Generate JSON files for labeling datasets in the label/ folder:

• label-single/: For individual datasets. Run the appropriate script to create structured JSON files with image paths, masks, and metadata.

Note: The label-all/ folder (for all datasets combined) should be generated after completing step 2, once the text prompts are available. Run the appropriate script at that point to create structured JSON files incorporating the prompts, along with image paths, masks, and metadata.

2. Generate Text Prompts Using Doubao

Use Doubao (a text generation tool) to create descriptive prompts for given defect data. Outputs are saved in the doubao/ folder:

• doubao.py: Basic version for raw text prompts.
• doubao-annotated.py: Annotates anomaly regions with red bounding boxes in images.
• doubao-cropper.py: Generates cropped images focusing on anomalies.

These prompts will guide the anomaly synthesis during generation.

3. Training with Anomagic_train.py

Train the Anomagic model using the prepared JSON data:

python Anomagic_train.py \
    --data_json_file /path/to/your/dataset.json \
    --output_dir /path/to/output/weights

• --data_json_file: Path to the JSON file from Step 1, specifying training images.
• --output_dir: Directory to save trained weights.

After training, process the weights using the Anomagic_weight.py files in the weight/ folder.

4. Training/Finetuning Inpainting Model with Anomagic_finetune_Inpaint.py

Coming soon...

If you want to train or finetune the inpainting model, use the Anomagic_finetune_Inpaint.py script instead. This script supports three finetune modes via the --finetune_mode parameter:

python Anomagic_finetune_Inpaint.py \
    --data_json_file /path/to/your/dataset.json \
    --output_dir /path/to/output/weights \
    --finetune_mode [lora_only/feature_extraction/full]

• --finetune_mode: Core parameter for finetuning strategy, with three optional values:
  • lora_only: Only finetune the LoRA (Low-Rank Adaptation) parameters. This mode has low resource consumption and is suitable for lightweight finetuning on limited GPU memory.
  • feature_extraction: Freeze LoRA parameters and only train the feature extraction part of the model. This mode fixes the adapter layer and focuses on optimizing the base feature extraction network.
  • full: Finetune all parameters of the model (both LoRA and base model). This mode requires the most computing resources but can achieve the best adaptation effect for the target dataset.

5. Similarity Judgment for References

Find similar images as references using similarity_judgment.py in the doubao/ folder:

python doubao/similarity_judgment.py \
    --merged_json /path/to/merged_ad_datasets.json \
    --dataset_file /path/to/dataset_info.json \
    --dataset_name <dataset_name>

This generates a JSON file (<dataset_name>_similarity_results.json) in the directory of the dataset file, mapping defects to similar reference objects across datasets.

6. Generate Masks for Defects

Create corresponding masks for defects:

• Apply mvtec.json or visa.json.
• Run creatMask.py:

python mask/creatMask.py \
    --json_path /path/to/visa.json \
    --output_root /path/to/masks

Masks are saved in the specified directory for use in generation.

7. Pretrained Model Checkpoints

Download the pretrained checkpoints for the Anomagic model. These files are hosted on Hugging Face and are used to initialize the IP-Adapter and attention modules, enabling zero-shot anomaly generation. Place them directly in the checkpoint/ directory for immediate use.

Filename	Size	Description	Download Link
attention_module.bin	13.4 MB	Attention module weights for cross-modal prompt alignment	Hugging Face
anomagic.bin	3.54 GB	Anomagic core weights for image-text fusion generation	Hugging Face

Usage:

• Download the files to the project's root checkpoint/ folder (create it if it doesn't exist).
• These pretrained weights can be used directly for inference without further training.
• These weights are compatible with the Diffusers library (v0.21+). For training, load them in Anomagic_train.py to fine-tune on custom datasets.

Repository: Anomagic Checkpoints

8. Anomaly Generation with Anomagic_test.py

Generate synthetic anomaly images using the trained weights and references:

python Anomagic_test.py \
    --ip_ckpt /path/to/trained/weights/anomagic.bin \
    --ip_ckpt_1 /path/to/trained/weights/attention_module.bin \
    --similarity_results /path/to/similarity_results.json \
    --dataset_base /path/to/normal_images \
    --mask_base /path/to/generated/masks \
    --output_base /path/to/generated/anomalies \
    --quality_model_path  /path/to/MetaUAS quality assessment model

• --ip_ckpt: Path to the trained Anomagic checkpoint.
• --similarity_results: JSON from Step 5.
• --dataset_base: Directory of normal images to inject anomalies into.
• --mask_base: Path to masks from Step 6.
• --output_base: Directory for generated anomaly images, masks, and overlays.
• --quality_model_path Path for MetaUAS quality assessment model. Checkpoints

The script applies quality control (e.g., MetaUAS scores) and saves results in a structured folder (e.g., abnormal_images/, mask_overlays/).

🖼️ Anomaly Inference

Our pipeline generates high-fidelity synthetic anomaly images across diverse categories, including industrial datasets (VisA, MVTecAD), medical imaging (BraTS [deVerdier et al. 2024], OCT [Kermany et al. 2018]), and web-crawled images for real-world objects. Below are examples of synthesized images from each category, demonstrating localized anomalies with quality masks.

Industrial Datasets (VisA & MVTecAD)

Synthesized anomalies on objects like PCBs and capsules, with precise defect localization.

Medical Imaging

Anomalies in brain tumors (BraTS) and retinal scans (OCT), preserving anatomical fidelity.

BraTS and OCT [deVerdier et al. 2024, Kermany et al. 2018]: Brain MRI and Retinal OCT with enhanced tumor region

Web-Crawled Real-World Objects

Diverse everyday items with realistic anomalies, sourced from web images.

💌 Acknowledgement

Our generation pipeline is based on the 🤗Diffusers library, incorporating implementations from IP-Adapter.

Big thanks to these amazing open-source works!

📝 Citation

If you find our work useful in your research, please consider citing our paper:

@article{jiang2025anomagic,
  title={Anomagic: Crossmodal Prompt-driven Zero-shot Anomaly Generation},
  author={Jiang, Yuxin and Luo, Wei and Zhang, Hui and Chen, Qiyu and Yao, Haiming and Shen, Weiming and Cao, Yunkang},
  journal={arXiv preprint arXiv:2511.10020},
  year={2025}
}