Adversarial Machine Learning
Two CIFAR-10 model instances—one clean and one trained on poisoned data—used to demonstrate evasion, a universal perturbation, a backdoor, and an illustrative Neural Cleanse-style analysis with a clean probe set.
- Year
- 2025
- Role
- Self-directed research
- Status
- Complete
The question
Can a model be 92% accurate and still be completely compromised?
I wanted to feel the answer, not just read it. So I built two CIFAR-10 classifiers, one clean and one I poisoned myself, then spent the project attacking both. The clean model I hit from the outside. Small, bounded pixel perturbations. Then a single reusable pattern that turns almost any image into a cat. The poisoned one I attacked from the inside, hiding a backdoor that stays invisible until a 3×3 red square shows up in the corner.
Then I switched sides and tried to catch my own backdoor with a Neural Cleanse-style probe. The first attempt confidently blamed the wrong class. That mistake taught me more than any of the wins, and it is why I now trust a defense only as far as I trust the data I feed it.
My role: Self-directed research.
One image, two attack methods
The clean input is classified as airplane. Both saved attacks change its prediction to ship.
Method
The decisions
Fold normalization into the model and attack in pixel space
The mean/std normalization lives inside the network, not the data pipeline, so every attack budget is measured in real 0-to-255 pixels. An epsilon of 8/255 means exactly that, and every adversarial image stays a valid image rather than some rescaled abstraction.
Train a separate poisoned model from scratch
The notebook trains a second instance of the architecture from random initialization on a dataset where 10% of samples receive the 3×3 trigger and the dog label. This is not fine-tuning the clean model. The resulting model keeps 91.7% clean accuracy while the trigger succeeds 99.94% of the time.
Use a Neural Cleanse-style clean-probe analysis
For each target class, the notebook optimizes a small mask and pattern against 256 randomly sampled clean test images. The planted dog class needs an unusually small effective mask and reaches anomaly index 4.62. The method therefore uses model gradients plus a clean probe set; it does not recover the trigger from weights alone.
Record a seeded reference run and its limits
The notebook fixes random seeds and records every output, but GPU kernels, library versions, and hardware can still move the exact values. The embedded numbers are one Colab reference run, not guaranteed constants.
Evidence
Where it broke
The defense flagged the wrong class
SymptomNeural Cleanse confidently flagged airplane as the backdoored class. The real planted trigger was dog, which sat mid-pack and went undetected.
CauseThe probe set was 256 images taken from the front of the test set. CIFAR-10 loads in class order and the loader was not shuffled, so those 256 images were all airplanes. Every class was being reverse-engineered using nothing but airplane photos, and making an airplane look like an airplane needs a near-zero patch, so airplane won by construction.
FixSample the probe images randomly across all ten classes. Once the defender saw a representative batch, dog dropped to the smallest mask and the false positive disappeared. Added a check that the recovered trigger actually forces the class, so a degenerate tiny mask cannot win on a fluke.
LessonThe bug was not in the algorithm, it was in the data I fed it. A defense is only as honest as its probe set, and a class-ordered dataset is a quiet trap.
Right answer, borderline verdict
SymptomWith the probe fixed, dog was correctly ranked as the smallest trigger, but its anomaly index came out at 1.53, just under the standard detection threshold of 2. Correct ranking, unconvincing verdict.
CauseThe trigger was a 5x5 square. Its recovered mask was the smallest, but not a dramatic enough outlier against a pack clustered between 30 and 49, so the anomaly score stayed soft.
FixThe final illustrative run uses a 3×3 trigger, which raises the dog anomaly index to 4.62. Because trigger size and detector settings were adjusted after observing earlier misses, this demonstrates the mechanism but is not an unbiased estimate of detection performance.
LessonDetectability depends on the trigger, probe distribution, optimization, and threshold. A clean result after iterative tuning should be labelled as an illustration, not evidence that the defense will reliably catch unseen backdoors.
Results and limits
Outcomes
- Clean model: 92.5% full-test accuracy; FGSM 93.9% and PGD 100% fool rates on a labelled 2,048-image diagnostic prefix
- Universal pattern: 98.0% target-prediction rate on a labelled 5,120-image diagnostic prefix, including images already predicted as the target class
- Separately trained poisoned model: 91.7% clean accuracy and 99.94% trigger attack success
- Neural Cleanse-style analysis using 256 clean probes flags the planted class at anomaly index 4.62 in an iteratively tuned illustrative setup
- A self-contained Colab reference run with reproducibility limits and a documented probe-set false positive
Guardrails
- Evasion needs no access to training. A few pixels of change, invisible to a human, flips a confident prediction
- A universal perturbation is one reusable pattern, so an attacker solves once and applies everywhere
- A backdoor survives normal evaluation: 91.7% clean accuracy hides a 99.94% trigger, so accuracy alone is not a safety metric
- Defenses have blind spots. Neural Cleanse can be fooled by a biased probe set, and can miss a backdoor whose trigger is not enough of an outlier