Awesome MLSecOps: Machine Learning and AI Security Resources 🛡️🤖

September 3, 2026 · View on GitHub

What is MLSecOps?

MLSecOps (Machine Learning Security Operations) is the practice of integrating security throughout the machine learning lifecycle—from data collection and model development to deployment, monitoring, and incident response. It applies security testing, threat modeling, supply-chain protection, access controls, and continuous monitoring to machine learning models, MLOps pipelines, LLM applications, and AI agents.

This curated catalog helps security engineers, ML practitioners, developers, and AI red teams discover open-source MLSecOps tools, adversarial machine learning research, AI security frameworks, threat-modeling resources, and practical learning materials.

⭐ If this catalog is useful, star the repository or read the contribution guidelines to suggest a resource.

MLSecOps Banner

What is MLSecOps?
Open Source Security Tools
Commercial MLSecOps and AI Security Tools
Data Privacy and Anonymization Tools
ML Code Security
AI Security and MLSecOps Beginner Resources
Threat Modeling
Attack Vectors
Blogs and Publications
MLOps Infrastructure Vulnerabilities
MLSecOps Pipeline
Adversarial Machine Learning Research Repositories
Community Resources
Books
Infographics
Contributions
Contributors
Repository Stats
Activity
Support Us
License

Open Source Security Tools

Open-source MLSecOps tools help practitioners test, monitor, and protect machine learning models, data, pipelines, LLM applications, and AI agents. The resources below include active projects and proofs of concept; evaluate maintenance status, licensing, threat coverage, and deployment suitability before production use.

Model and Artifact Security

Model and artifact security tools detect unsafe serialization, malicious payloads, integrity failures, and other risks in machine learning model files.

Tool	Description
ModelScan	Protection Against ML Model Serialization Attacks
Safetensors	Tensor storage format designed to avoid executable deserialization; it does not establish model provenance or behavioral safety

Adversarial Machine Learning Testing

Adversarial machine learning testing tools evaluate model robustness against evasion, poisoning, extraction, inversion, and other adversarial techniques.

Tool	Description
Adversarial Robustness Toolbox	Library of defense methods for ML models against adversarial attacks
Foolbox	Python toolbox for creating and evaluating adversarial attacks and defenses

LLM Security and Red Teaming

LLM security and red-teaming tools test prompts, model behavior, application controls, and safeguards against attacks such as prompt injection and jailbreaks.

Tool	Description
Garak	LLM vulnerability scanner
Promptfoo Scanner	An open-source LLM red teaming tool

AI Agent Security

AI agent security tools assess memory, tools, permissions, workflows, and runtime boundaries used by autonomous and tool-enabled AI systems.

Tool	Description
Agent Memory Guard	OWASP project providing a runtime defense layer that screens AI agent memory reads and writes against prompt injection, secret leakage, and memory poisoning
Agent-Wiz	Python CLI by Repello AI for extracting agentic workflows from LangChain/LangGraph/CrewAI/AutoGen and running automated threat modeling
HOL Guard	Open-source local-first runtime security layer for AI agents that evaluates supported tool actions and local artifacts for prompt injection, secret exposure, unsafe commands, package risks, and MCP threats
Prismor	Self-hosted runtime control plane that screens AI coding-agent tool calls against a signed policy before execution, blocking or routing to human approval secret exfiltration, prompt-injection-driven actions, and destructive commands across Claude Code, Codex, and MCP servers

Privacy-Preserving Machine Learning

Privacy-preserving machine learning tools help limit sensitive-data exposure through differential privacy, encrypted computation, anonymization, and privacy testing.

Tool	Description
TensorFlow Privacy	Library of privacy-preserving machine learning algorithms and tools
OpenDP	The core library of differential privacy algorithms powering the OpenDP Project

AI Supply-Chain Security

AI supply-chain security tools protect model artifacts, provenance, dependencies, signatures, bills of materials, and deployment pipelines.

Tool	Description
Model Transparency	Generate model signing metadata for provenance verification
BomLens	Local-first SBOM generator that builds CycloneDX ML-BOMs for Hugging Face models, with license and known-vulnerability reports

Model Testing, Monitoring, and Evaluation

These projects support model validation, monitoring, robustness, safety, and comparative evaluation, including both general-purpose and security-focused assessment workflows.

Tool	Description
TensorFlow Model Analysis	A library for analyzing, validating, and monitoring machine learning models in production
CircleGuardBench	A full-fledged benchmark for evaluating protection capabilities of AI models

Additional Open-Source Tools

The following tools and research prototypes address additional MLSecOps testing, monitoring, defense, and assurance use cases pending more granular classification.

Tool	Description
MLSploit	Cloud framework for interactive experimentation with adversarial machine learning research
Advertorch	Python toolbox for adversarial robustness research
MITRE ATLAS	Knowledge base of adversary tactics and techniques against AI systems; successor to the deprecated Adversarial ML Threat Matrix
CleverHans	A library of adversarial examples and defenses for machine learning models
AdvBox	Advbox is a toolbox to generate adversarial examples that fool neural networks in PaddlePaddle、PyTorch、Caffe2、MxNet、Keras、TensorFlow
Audit AI	Bias Testing for Generalized Machine Learning Applications
Deep Pwning	Deep-pwning is a lightweight framework for experimenting with machine learning models with the goal of evaluating their robustness against a motivated adversary
Privacy Meter	An open-source library to audit data privacy in statistical and machine learning algorithms
PromptInject	A framework that assembles adversarial prompts
TextAttack	TextAttack is a Python framework for adversarial attacks, data augmentation, and model training in NLP
OpenAttack	An Open-Source Package for Textual Adversarial Attack
TextFooler	A Model for Natural Language Attack on Text Classification and Inference
Flawed Machine Learning Security	Practical examples of "Flawed Machine Learning Security" together with ML Security best practice across the end to end stages of the machine learning model lifecycle from training, to packaging, to deployment
Adversarial Machine Learning CTF	CTF challenge demonstrating adversarial-image attacks against neural-network image classifiers
Damn Vulnerable LLM Project	A Large Language Model designed for getting hacked
Gandalf Lakera	Prompt Injection CTF playground
Vigil	LLM prompt injection and security scanner
PALLMs (Payloads for Attacking Large Language Models)	list of various payloads for attacking LLMs collected in one place
AI-exploits	Exploits for MLOps systems, extending beyond inputs provided to LLMs such as ChatGPT
Offensive ML Playbook	Offensive ML Playbook. Notes on machine learning attacks and pentesting
AnonLLM	Anonymize Personally Identifiable Information (PII) for Large Language Model APIs
AI Goat	vulnerable LLM CTF challenges
Pyrit	The Python Risk Identification Tool for generative AI
Raze to the Ground: Query-Efficient Adversarial HTML Attacks on Machine-Learning Phishing Webpage Detectors	Source code of the paper "Raze to the Ground: Query-Efficient Adversarial HTML Attacks on Machine-Learning Phishing Webpage Detectors" accepted at AISec '23
Giskard	Open-source testing tool for LLM applications
Model-Inversion-Attack-ToolBox	A framework for implementing Model Inversion attacks
NeMo Guardrails	NeMo Guardrails allow developers building LLM-based applications to add programmable guardrails between the application code and the LLM
AugLy	Multimodal data-augmentation library for evaluating and improving model robustness; it is not an adversarial-attack generator
Knockoffnets	Proof of concept for training a functional surrogate from black-box model queries
VGER	Jupyter Attack framework
AIShield Watchtower	An open-source tool from AIShield for studying AI models and scanning for vulnerabilities
PS-fuzz	tool for scanning LLM vulnerabilities
Mindgard CLI	Evaluate the security of AI systems through a CLI
Purple Llama	Meta's umbrella suite of LLM safety and security tooling, including Llama Guard, Code Shield, and CyberSecEval
ARTkit	Automated prompt-based testing and evaluation of Gen AI applications
LangBiTe	A Bias Tester framework for LLMs
TF Encrypted	Framework for encrypted deep learning in TensorFlow using secure multi-party computation
Agentic Security	Agentic LLM Vulnerability Scanner / AI red teaming kit
skill-audit-mcp	Static security scanner for MCP servers, AI agent skills, and plugins, with SARIF output and CI integration
AIsbom	Disassembles Pickle bytecode and parses SafeTensors/GGUF binary headers to detect malware and license risks in ML model files before load. Generates CycloneDX/SPDX SBOMs.
AI-Scan-Interceptor	Self-hostable DLP gateway for enterprise prompts to ChatGPT/Claude/Gemini (Squid + Go ICAP + mTLS, AGPL-3.0)
NB Defense	Jupyter notebook security scanner from Protect AI (now part of Palo Alto Networks)
TrustGate	An open-source Generative Application Firewall (GAF)
Whistleblower	Open-source offensive tool by Repello AI for testing LLM apps against system prompt leakage
IronClaw	Self-hosted runtime that sandboxes autonomous AI agents with isolated execution, restricted networking, and human-approval gating

Commercial MLSecOps and AI Security Tools

Commercial MLSecOps and AI security tools support model protection, application testing, monitoring, governance, and incident response. Evaluate technical documentation, deployment options, integrations, and independent evidence before adoption.

Tool	Description
Citadel Lens	Commercial model-testing platform; verify supported standards and security checks in current product documentation
Cisco AI Defense (formerly Robust Intelligence)	Commercial platform for AI model validation and runtime protection; Robust Intelligence was acquired by Cisco
HiddenLayer AI Detection and Response	Commercial AI detection and response platform from HiddenLayer
Guardian	Model protection in CI/CD
NeuralTrust	Tools to protect, secure and test GenAI Applications
SourceryKit	Verifies an AI agent's outbound requests and MCP handoffs against a source of truth using zero-knowledge proofs, so a call only goes out if its claims check out; hooks the HTTP libraries, logs each outbound call, and blocks endpoints not on the trusted allow-list
Trent AI	Agentic AI security platform that continuously assesses AI agents, MCP servers, LLM and AI-native applications, and code shipped with AI coding tools, traces attack chains, and verifies proposed fixes landed

Data Privacy and Anonymization Tools

Data privacy and anonymization tools help reduce exposure of personal or sensitive information in machine learning datasets, images, videos, and model interactions.

Tool	Description
ARX - Data Anonymization Tool	Tool for anonymizing datasets
Data-Veil	Data masking and anonymization tool
Tool for IMG anonymization	Image anonymization
Tool for DATA anonymization	Data anonymization
BMW Anonymization API	This repository allows you to anonymize sensitive information in images/videos. The solution is fully compatible with the DL-based training/inference solutions that we already published/will publish for Object Detection and Semantic Segmentation
DeepPrivacy2	A Toolbox for Realistic Image Anonymization

ML Code Security

ML code security resources help practitioners identify vulnerable dependencies, unsafe model formats, insecure library behavior, and model-extraction risks in machine learning software.

lintML - Security linter for ML training code, maintained by a member of the NVIDIA AI red team
HiddenLayer: Model as Code - Research article on executable model formats and unsafe deserialization paths in ML libraries
Copycat CNN - Proof of concept for training a surrogate convolutional neural network from black-box queries
differential-privacy-library - Library designed for differential privacy and machine learning

AI Security and MLSecOps Beginner Resources

These foundational resources explain how adversaries attack AI systems and how defenders secure models, prompts, data, and MLOps workflows.

AI Security Study Map

Full size map in this repository

Threat Modeling

AI threat modeling identifies assets, trust boundaries, attack paths, and security controls across models, data, prompts, pipelines, infrastructure, and agent capabilities.

More in Adversarial AI Attacks, Mitigations, and Defense Strategies: A cybersecurity professional's guide to AI attacks, threat modeling, and securing AI with MLSecOps.

Attack Vectors

Machine learning systems can be attacked through poisoned data, adversarial inputs, model extraction, privacy attacks, compromised artifacts, excessive agent permissions, and vulnerable MLOps infrastructure. The resources below explain individual attack classes and their defensive implications.

Blogs and Publications

These publications cover MLSecOps practices, adversarial machine learning research, LLM security, AI red teaming, privacy, and security measurement.

MLOps Infrastructure Vulnerabilities

MLOps infrastructure introduces security risks across notebooks, training pipelines, model registries, artifact formats, cloud services, Kubernetes workloads, and inference endpoints. These resources document vulnerabilities, exploits, and defensive controls affecting the operational ML lifecycle.

SILENT SABOTAGE - Study on bot compromise for converting Pickle to SafeTensors
NOT SO CLEAR: HOW MLOPS SOLUTIONS CAN MUDDY THE WATERS OF YOUR SUPPLY CHAIN - Study on vulnerabilities for the ClearML platform
Uncovering Azure's Silent Threats: A Journey into Cloud Vulnerabilities - Study on security issues of Azure MLAAS
The MLOps Security Landscape
Confused Learning: Supply Chain Attacks through Machine Learning Models

MLSecOps Pipeline

Adversarial Machine Learning Research Repositories

These research repositories provide implementations, experiments, benchmarks, attacks, and defenses for adversarial machine learning, model privacy, extraction, inversion, robustness, and LLM security.

Repository	Category	Security focus
AgentPoison	LLM agent backdoor poisoning	Red-teaming LLM agents via memory or knowledge-base poisoning attacks.
DeepPayload	Neural trojan research	Explores embedding malicious payload behaviors into deep neural networks.
backdoor	Backdoor attacks	Investigates hidden-trigger backdoor attacks in deep learning models.
Stealing_DL_Models	Model stealing	Demonstrates model-stealing techniques against deep learning systems.
datafree-model-extraction	Model extraction	Studies data-free extraction of target model behavior.
LLMmap	LLM attack-surface mapping	Maps and analyzes LLM behavior to support security-oriented assessment.
GoogleCloud-Federated-ML-Pipeline	Federated learning pipelines	Implements a federated machine-learning pipeline using Google Cloud infrastructure.
Class_Activation_Mapping_Ensemble_Attack	Adversarial evasion attacks	Uses CAM-based ensemble methods to craft adversarial examples.
COLD-Attack	Adversarial optimization attacks	Implements constrained optimization attacks against deep models.
pal	Adaptive adversarial attacks	Research code for adaptive attack strategies against defended models.
ZeroShotKnowledgeTransfer	Model extraction	Zero-shot knowledge-transfer methods relevant to model-stealing risk.
GMI-Attack	Model inversion attacks	Implements generative model inversion attacks against trained models.
Knowledge-Enriched-DMI	Model inversion attacks	Extends deep model inversion with auxiliary knowledge priors.
vmi	Model inversion attacks	Implements variational model inversion attacks to recover private training information.
Plug-and-Play-Attacks	Black-box attack methods	Provides plug-and-play attack pipelines for black-box ML models.
snap-sp23	Privacy extraction attacks	Implements SNAP poisoning attacks for private-property extraction from models.
privacy-vs-robustness	Privacy vs. robustness research	Evaluates trade-offs between differential privacy and adversarial robustness.
ML-Leaks	Membership inference attacks	Implements membership-inference attacks to test privacy leakage.
BlindMI	Membership inference attacks	Studies label-only and blind membership-inference techniques.
python-DP-DL	Privacy defenses	Differential privacy utilities for protecting training data in deep learning.
MMD-mixup-Defense	Adversarial defenses	Defense approach designed to improve robustness against adversarial attacks.
MemGuard	Membership inference defenses	Defends model outputs against membership-inference attacks.
unsplit	Split-learning attacks	Demonstrates model inversion, model stealing, and label inference against split learning.
face_attribute_attack	Biometric adversarial attacks	Explores attacks on face-attribute recognition systems.
FVB	Biometric adversarial attacks	Attack methods targeting face-verification models.
Malware-GAN	Malware evasion research	Uses GAN-generated variants to study ML-based malware-detection evasion.
Generative_Adversarial_Perturbations	Adversarial example generation	Generates adversarial perturbations with generative models.
Adversarial-Attacks-with-Relativistic-AdvGAN	Adversarial example generation	Implements AdvGAN-based adversarial attacks for deep models.
llm-attacks	LLM jailbreak research	Research code for adversarial prompting and jailbreak attacks on LLMs.
LLMs-Finetuning-Safety	LLM safety defenses	Studies safer LLM fine-tuning under harmful-behavior and leakage risks.
DecodingTrust	Trust and safety benchmarking	Benchmark suite for evaluating trust, safety, and robustness dimensions in LLMs.
promptbench	Prompt robustness benchmarking	Benchmarking framework for evaluating prompt robustness and reliability.
rome	Model editing and reliability	Implements factual model-editing methods used in reliability and safety research.
llmprivacy	LLM privacy attacks	Research on privacy leakage and protection in large language models.

_{@RiccardoBiosas}	_{@wearetyomsmnv}	_@Copilot _{GitHub Copilot coding agent}	_@badarahmed	_@haksungjang
_@deadbits	_@alespignaNT	_{@alexcombessie}	_{@aryaman-titan}	_@typpo
_@Igralino	_@eltociear	_@mik0w	_@omerzamir	_{@robvanderveer}
_{@clubanderson}	_@lab700xdev	_{@mshirakawa-ssp}	_@anmorgan24