EAGLE: Expert-Augmented Attention Guidance for Tuning-Free Industrial Anomaly Detection in Multimodal Large Language Models

February 24, 2026 · View on GitHub

arXiv

📖Catalogue

👇Framework overview:

framework We propose EAGLE, a tuning-free framework for anomaly detection. The framework comprises two core components: (1) a PatchCore-based expert model that performs preliminary anomaly detection and transforms the detection results into visual and textual prompt, and (2) a Multimodal Large Language Model (MLLM) that receives the visual and textual prompts generated by the expert model for reasoning.

To avoid fine-tuning overhead, we introduce two key mechanisms. The Distribution-Based Thresholding (DBT) mechanism transforms expert model outputs into prompts suitable for MLLMs input by estimating a decision threshold from anomaly score distributions on normal samples. This threshold classifies test samples and determines the corresponding visual prompts and textual prompts (linguistic priors) for MLLMs input. The CAAS mechanism identifies low-confidence predictions from the expert model and selectively amplifies visual token attention in MLLMs intermediate layers. By strengthening visual evidence, CAAS prevents the MLLMs from over-relying on erroneous linguistic priors, enabling more accurate correction of expert model misclassifications.

🧮 DBT mechanism

DBT Analysis

A key observation motivating DBT is that only a small fraction of patch features are retained in the memory bank, while the majority are discarded during coreset sampling. These discarded features, although not stored in the memory bank, are still extracted from normal training images and thus encode rich information about the normal data distribution.

We analyze the proportion of sampled and unsampled patches per training image during memory bank construction on representative classes. The results show that, on average, only about 10% of patches are selected into the memory bank, while the unsampled patches account for approximately 90% of all extracted features. Under this, we make the following observation: For a normal training sample, the image-level anomaly score can be determined solely by the unsampled patches Pi(un)P_i^{(un)}.

The green histogram corresponds to the anomaly scores of normal training images, while the blue histogram represents those of normal test images. We observe that the distribution estimated from the training set covers most normal test samples, owing to the large number of normal training images. In contrast, images containing defects produce substantially higher anomaly scores, as defect regions yield large patch-level distances that exceed the normal score distribution.

🛳️ Setup

Environment

  • Python 3.10
  • CUDA 11.4
  • PyTorch 2.2.0 + cu118

Create a virtual environment:

conda create --name Eagle python=3.10.16
conda activate Eagle

Install the required packages:

pip install torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 \
  --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt

👇 Dataset Preparation

Our dataset follows the same setting as the MMAD project:

Download the dataset:

cd dataset/MMAD
wget -O ALL_DATA.zip https://huggingface.co/datasets/jiang-cc/MMAD/resolve/refs%2Fpr%2F1/ALL_DATA.zip?download=true
unzip ALL_DATA.zip

🦿 MLLM Backbone

Download Qwen2.5-VL

pip install -U huggingface_hub
huggingface-cli download Qwen/Qwen2.5-VL-7B \
  --local-dir /MLLMs/Qwen2.5-VL-7B

🏃‍♀️ Running EAGLE

Step 1: Configure the Model

Before running, please check and modify the default settings in qwen.yaml, especially the dataset and model paths:

data_path_image: "Your_dataset_path/MVTec-AD"
backbone_names: [wideresnet50]
imagesize: 224
resize: 256
batch_size: 2
num_workers: 4
faiss_on_gpu: true
faiss_num_workers: 8
pretrain_embed_dimension: 1024
target_embed_dimension: 1024
model_path: "Your_MLLM_path"
model_name: qwen
dataset_name: DS-MVTec
domain_knowledge: true
threshold_method: k-sigma
status: true

Step 2: Check Import Paths

Please ensure that the import paths in the following files point to your local project directory:

  • framework.py
  • expert_base.py
  • connector_sort.py
# Make sure that the required modules can be imported
sys.path.insert(0, "Your_project_path/Eagle/Eagle/src")
  • mllm_base.py:
# Make sure that the required modules can be imported
sys.path.insert(0, "Your_project_path/Eagle")

Step 3: Run Evaluation

python qwen_eval.py --config qwen.yaml

🔍 CAAS mechanism

coming soon...

🙇‍♀️ Thanks 🙌

  • Our implementation of the Distribution-Based Thresholding (DBT) mechanism is heavily built upon PatchCore, and the evaluation pipeline is adapted from MMAD . We sincerely thank the authors for open-sourcing their work!!
  • This project is developed under CUDA 11.4, which is intentionally selected for compatibility with legacy industrial environments. If you encounter any issues during reproduction, please report them in the Issues section.

Citation

If you find this repository and our paper useful, please consider citing our work 😊

@article{peng2026eagle,
  title={EAGLE: Expert-Augmented Attention Guidance for Tuning-Free Industrial Anomaly Detection in Multimodal Large Language Models},
  author={Peng, Xiaomeng and Huang, Xilang and Choi, Seon Han},
  journal={arXiv preprint arXiv:2602.17419},
  year={2026}
}