How Well Does Your
AI Defense Actually Work?
We built an adversarial benchmark to find out. 18,232 trials across six frontier model families, three geographic regions, and 24 test conditions.
What We Found
Three high-level findings from the benchmark. Detailed methodology, raw data, and per-condition breakdowns are available on request.
The most widely deployed defense performed worse than no defense.
Token-level redaction — the default pre-processing layer in most enterprise AI deployments — scored below the undefended baseline across every condition tested.
Entity-level data can be fully protected. Semantic signals cannot.
Under layered defense, extraction of names, amounts, and identifiers dropped to zero. But contextual and categorical signals persisted regardless of defense configuration — suggesting a boundary that may be architectural rather than implementation-specific.
One widely-assumed defense class fails completely. A different defense class passes.
Across all 24 test conditions, the most commonly proposed structural defense was reconstructed in every trial — 0% defense success. A separate defense class, validated in the same benchmark, exceeded baseline. The passing method is proprietary; details available under NDA for technical evaluation.
Reproducibility
All experiments use seeded randomization, per-run checkpointing, and deterministic similarity scoring. Full methodology, configuration parameters, and raw datasets are available on request for research collaboration and technical due diligence.
All findings are backed by DOI-registered publications. Papers and datasets are shared directly with researchers, investors, and technical evaluators upon request.
Want to see the full data?
Methodology, raw results, and per-condition breakdowns available on request.
or email yc@oia-lab.com