Awesome AI Security [](https://github.com/brandonhimpfen/awesome-lists)

     

A curated list of tools, frameworks, benchmarks, research, and resources focused on AI security — including adversarial attacks, model robustness, data poisoning, red teaming, model extraction, jailbreak defense, secure inference, and privacy-preserving AI.

Support ongoing maintenance and curation via GitHub Sponsors.

Contents

• Adversarial Attacks
• Defenses & Robustness
• Model Security
• Data Security & Poisoning
• Red Teaming & Testing
• Privacy-Preserving AI
• Evaluation & Benchmarks
• Learning Resources
• Related Awesome Lists

Adversarial Attacks

• CleverHans – Benchmark library for adversarial attacks and defenses.
• Foolbox – Python toolbox for creating adversarial examples.
• IBM Adversarial Robustness Toolbox (ART) – Comprehensive library for attacks and defenses across models.
• AdvTorch – Toolbox for adversarial robustness in PyTorch.
• TextAttack – Framework for NLP adversarial attacks.
• DeepSec – Benchmarking tool for adversarial attacks on neural networks.

Defenses & Robustness

• RobustBench – Leaderboard of adversarially robust models.
• Adversarial Training (Madry Lab) – Classic examples of adversarial training.
• ME-Net – Defense model against adversarial perturbations.
• Randomized Smoothing – Certified robustness technique for classification models.
• Feature Squeezing – Traditional defense against adversarial examples.

Model Security

• ModelGuard – Tools for securing ML supply chains and models.
• AI Model Watermarking – Tools and research on watermarking for model IP protection.
• Model Extraction Tooling – Scripts and examples for simulating model extraction attacks.
• ONNX Runtime Security Guidelines – Secure inference recommendations for ONNX-based deployments.

Data Security & Poisoning

• CleanLab – Identify noisy labels and corrupted datasets.
• pytorch-poisoning – Tools for dataset poisoning attacks.
• TensorFlow Privacy – Differential privacy training tools.
• PySyft – Privacy-preserving data science with encrypted computation.
• badNets Examples – Classic dataset poisoning case studies.

Red Teaming & Testing

• OpenAI Red Teaming Network – Global collaboration for model-level security testing.
• Microsoft Counterfit – Automation tool for adversarial ML security testing.
• Anthropic Red Teaming Resources – Tools and frameworks for systematic evaluations.
• LLM Jailbreak Datasets – Public sets of adversarial prompts for testing.
• PromptBench – Benchmark for prompt injection and jailbreak robustness.
• PromptTrace – Interactive AI security training platform with hands-on labs for prompt injection, RAG poisoning, and LLM red teaming.

Privacy-Preserving AI

• OpenDP SmartNoise – Differential privacy tools and synthetic data generators.
• Federated Learning (Flower) – Framework for federated and privacy-preserving learning.
• FATE – Industrial-grade federated learning platform.
• PyGrid / PySyft – Secure, distributed data science tools.
• Homomorphic Encryption Libraries – Tools for encrypted inference.

Evaluation & Benchmarks

• HarmBench – Safety and harm classification benchmark for AI systems.
• HELM – Holistic evaluation of model safety and robustness.
• ISC-Bench – Benchmark for evaluating LLM safety and alignment failures, including task-completion vs safety tradeoffs.
• MLSec Resources – Community-driven lists of security-focused ML tools.
• OpenAI Evals – Evaluation framework adaptable for adversarial and security testing.
• SafetyBench – Suite of tests for model safety.
• RobustBench – Benchmark suite for robust classification models.

Learning Resources

• MIT ML Security Course – Seminal course on adversarial ML.
• AI Security Papers (CAIS) – Curated papers on robustness and adversarial threats.
• Microsoft ML Security Research – Applied research on securing ML systems.
• OpenAI Security Research – Findings on model vulnerabilities and defenses.
• NIST AI Security Framework – Guidelines for secure and trustworthy AI.

Related Awesome Lists

• Awesome AI Safety & Alignment
• Awesome AI
• Awesome Machine Learning
• Awesome Open Governance
• Awesome AI Research Tools

Contribute

Contributions are welcome. Please ensure your submission fully follows the requirements outlined in CONTRIBUTING.md, including formatting, scope alignment, and category placement.

Pull requests that do not adhere to the contribution guidelines may be closed.

License