Web Artifact Attacks Disrupt Vision Language Models

Official implementation of Web Artifact Attacks Disrupt Vision Language Models.

Artifact Dataset

Download directly from URL(s)

We obtain the attack Artifacts by filtering the CC12M dataset. We release the subset images' url(s) under

cc12m_artifacts_dataset/logo_filename_to_url.pkl

To download the subset, run:

python artifact_dataset_scripts/download_artifact_subset.py

Note that some images' urls might die with time.

Filter CC12M dataset

Alternatively, we also provide the filtering pipeline code to filter CC12M.

First, downlaod CC12M using the img2dataset and then set the path to your dataset in

artifact_dataset_scripts/cc12m_dataset_path.txt

Then:

1- Score CC12M images "logoness" by running:

python artifact_dataset_scripts/dataset_score_artifacts.py

2- Sort and save the top scoring images by running:

python artifact_dataset_scripts/dataset_get_artifacts.py

Generate Text Artifacts

In addition to the graphical artifacts, we extract the text in the artifacts and use it as another vector of attacks. To obtain the text, run the following after obtaining the artifact subset dataset as instructed above.

python artifact_dataset_scripts/dataset_get_artifacts_text.py

Mining Attack Artifacts

1- Score the two types of artifacts in cc12m_artifacts_dataset, namely

• logos
• texts

(Note: In Step 2, logos is split into graphics with and without text.)

To score artifacts, run:

python attack_score_artifacts.py --dataset [fairface_age/fairface_gender/...] --pretrained [laion400m_e32/...] --model_name [ViT-B-32/...] --num_subjects [32/16/...] --batch_size [32/...] --logos_type [logos/texts]

2- Retrieve the most effective artifacts for each model/pretrained/dataset combination:

python attack_get_artifacts.py 

This will automatically search in output/scores/ and categorize artifacts into:

• logos_graphics: Graphics wo/ Text
• logos_text: Graphics w/ Text
• texts: Just text

3- For typographic attacks:

python attack_get_artifacts_typo.py 

4- To caption graphical artifacts for Artifact-Aware Prompting:

python attack_caption_artifacts.py 

Below, we provide all the CLIP model/pretrained combos that we use in this study:

[
  ("ViT-L-14-336", "openai"),
  ("ViT-L-16-SigLIP-256", "webli"),
  ("ViT-B-16", "laion400m_e32"),
  ("ViT-B-32", "laion400m_e32"),
  ("ViT-B-32", "datacomp_xl_s13b_b90k"),
  ("ViT-B-16", "datacomp_xl_s13b_b90k"),
  ("ViT-B-32", "laion2b_s34b_b79k"),
  ("ViT-B-16", "laion2b_s34b_b88k"),
  ("ViT-L-14", "laion2b_s32b_b82k")
]

Note that for LLaVA, we use its visual encoder to mine for artifacts, namely: ("ViT-L-14-336", "openai")

Evaluating Attack Artifacts

To evaluate the mined artifacts from the previous step, please run:

python eval_dataset.py --dataset [fairface_age/fairface_gender/...] --pretrained [laion400m_e32/...] --model_name [ViT-B-32/...] --num_subjects [32/16/...] --batch_size [32/...] --logos_mode [None/Generic/Concept] --logos_type [logos_graphics/logos_text/texts/typo]

--logos_mode specifies the type of artifact:

• None: No artifact
• Blank: Blank black artifact
• Concept: One of the following from --logos_type:
  • logos_graphics: Graphics wo/ Text
  • logos_text: Graphics w/ Text
  • texts: Just text

Optional settings:

• Size: --factor_shrink (e.g. 10 for 10th of the image size)
• Transparency: --transparency (e.g. 1.0 for fully visible)

for evaluating LLaVA:

python eval_dataset_lvlm.py --dataset [fairface_age/fairface_gender/...] --num_subjects [32/16/...] --batch_size [32/...] --logos_mode [None/Generic/Concept] --logos_type [logos_graphics/logos_text/texts/typo]

As stated before, we use the artifacts mined for the underlying visual encoder: ("ViT-L-14-336", "openai")

To evaluate with Artifact Aware Prompting, set:

• --add_caption True

Finally, to reproduce the results figure, refer to

plotting_scripts/plot.ipynb 

Citation

If you find this repository useful please give it a star and cite as follows! :) :

    @article{qraitem2025web,
    title={Web Artifact Attacks Disrupt Vision Language Models},
    author={Qraitem, Maan and Teterwak, Piotr and Saenko, Kate and Plummer, Bryan A},
    journal={arXiv preprint arXiv:2503.13652},
    year={2025}
    }