Cisco
2026-05-27
Cisco Study: All Frontier LLMs Vulnerable to Multi-Turn Attacks, Security Perimeter Must Shift
Summary
Key Takeaways
Cisco published a study evaluating 15 closed frontier LLMs under multi-turn attacks. Models tested include GPT-5.2, GPT-5.4, Claude Opus 4.5/4.6, Sonnet 4.5/4.6, Haiku 4.5, Gemini 3 Pro, Nova Lite/Micro/2 Lite, and Grok 4.1 Fast (reasoning and non-reasoning).
Using 30,090 single-turn prompts and 6,986 multi-turn attacks across 1,456 conversations, results show multi-turn ASR ranges from 7.89% to 88.30%, versus single-turn ASR of 2.19% to 64.91%. Cross-regime deltas are dramatic: Gemini 3 Pro jumps from 18.10% to 73.35% (4x), GPT-5.4 from 2.74% to 24.68% (9x). Grok 4.1 Fast non-reasoning hits 88.30% vs 43.47% with reasoning enabled.
Cisco proposes three rituals: publish ASR by strategy family, gate deployments on top-3 procedures/content types with a 3pp regression threshold, and flag models with >15pp absolute cross-regime gap. The study concludes no base model is iteratively safe, forcing security perimeter outside the model with runtime guardrails like Cisco AI Defense.
Why It Matters
Cisco's study is a strategic move to defend against emerging AI security vendors (Palo Alto Networks, Zscaler, open-source Guardrails) by defining multi-turn evaluation as the new standard. This shifts the control point to Cisco's proprietary Cisco Integrated AI Security and Safety Framework, locking enterprises into its assessment and protection toolchain.
Cisco downplays the limitations of its own AI Defense product: the evaluation framework and prompt bank are proprietary, forcing vendor lock-in. No independent comparison of AI Defense against multi-turn attacks is provided. The proposed 3pp regression threshold lacks theoretical grounding, risking false positives or misses. Multi-turn evaluation is computationally expensive and hard to scale in production, creating high ongoing costs and asset depreciation with model updates.
PRO Decision
【Vendors】 Competitors like Palo Alto Networks and Zscaler should: 1) Publish independent multi-turn evaluation reports comparing their products against Cisco AI Defense, highlighting latency and resource overhead. 2) Promote open-source evaluation frameworks (e.g., Garak, PyRIT) to break Cisco's standard monopoly. 3) Point out Cisco's sample bias (only 15 models, missing latest open-weight LLMs).
【Enterprises】 CIOs and architects should: 1) Demand independent benchmarks of Cisco AI Defense under multi-turn attacks, compared with at least two alternatives. 2) Build internal multi-turn evaluation capability using open-source tools to avoid lock-in. 3) Contractually require suppliers to provide multi-turn ASR data, but calibrate thresholds based on own use cases, not Cisco's 15pp rule.
【Investors】 See through Cisco's marketing: the study is a tool to boost AI Defense awareness. The real trend is multi-turn evaluation, but Cisco's closed framework faces open-source alternatives. Favor vendors with open, integrable tools (Protect AI, HiddenLayer). Cisco's vendor concentration risk rises; without independent validation, AI Defense may face enterprise trust erosion.
Get 3-5 key AI infrastructure signals weekly →
💬 Comments (0)