Data Poisoning and Adversarial ML
Definition
Attacks that target the model rather than the serving infrastructure: manipulating training data so the model learns wrong behavior (poisoning, backdoors), or crafting inputs that make a trained model fail predictably (adversarial examples).
The property that makes poisoning dangerous is leverage — a tiny fraction of corrupted data can compromise a whole model.
Core Ideas
Nightshade and Glaze — poisoning as artist defense
The best-known consumer-facing case, aimed at text-to-image models like Stable Diffusion. Notable because the same technique is both an attack and a protective measure for artists whose work is scraped for training.
- Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models — the paper behind Nightshade.
- Poison ratios are shockingly low: 0.1%, even 0.04% of images is enough to ruin a concept in the model.
- Glaze — the free app the same group shipped.
- Counter-tooling exists: NightshadeAntidote detects poisoned images — so this is an arms race, not a solved defense.
Backdoors in instruction-tuned LLMs
Virtual Prompt Injection — backdooring an instruction-tuned model so that it behaves as if a hidden instruction were present in every prompt, without that instruction existing in the input at all. The backdoor lives in the weights, so prompt-level inspection cannot find it.
PoisonGPT (Mithril Security) — a demonstration of hiding a surgically edited, fake-news-spreading model on Hugging Face. This is the supply-chain form of the attack: the poisoned artifact is the distribution channel.
Adversarial example tooling
- Foolbox — Python library for running adversarial attacks against neural networks (built on EagerPy).
- CleverHans — library for constructing attacks, building defenses, and benchmarking both.
- ordered-topk-attack — white-box targeted attacks on ImageNet-1k that force a DNN to predict any specified K targets in any given order as its top-K classes. Worth noting how much finer-grained the control is than “make it wrong”.
- Awesome Data Poisoning and Backdoor Attacks — the survey list.
Why this is a defender’s problem, not a researcher’s
- Provenance is the only real control: you cannot inspect a poisoned dataset into safety at 0.04% contamination.
- Fine-tuning inherits every problem of the base model plus the tuning set’s.
- Public model hubs are the distribution surface for both poisoned datasets and poisoned weights.
Relationships
- OWASP Top 10 for LLM Applications — this is LLM03 (training data poisoning) and LLM05 (supply chain) in depth
- Agentic Cyber Attack — the July 2026 incident began with a malicious dataset, making poisoning an infrastructure entry vector, not only a model-quality issue
- Machine Learning — training-data provenance as a first-class concern
- Data Quality — the benign version of the same dependency: models inherit whatever the data contains
- AI & Machine Learning
References
- Data Poison for LLM