Research by 1Password, led by Keith Hoodlet, demonstrates that frontier LLMs such as ChatGPT-5.5 and Claude Opus 4.8 frequently generate "Fix-Like Artifacts with Embedded Defects" (FLAWED) when addressing complex vulnerabilities. These models often produce fragile patches that block specific Proof-of-Concept (PoC) inputs rather than remediating the underlying architectural root cause. This failure mode resulted in a 53.9% failure rate during testing, with 49.3% of patches leaving exploitable attack paths open. The research highlights critical risks in automated remediation workflows, where AI-generated fixes may pass syntactic checks while remaining vulnerable to alternative exploitation vectors, potentially creating a false sense of security for CISOs and engineering teams.

• Research Overview & Methodology

  • Conducted large-scale evaluation of 6,080 patches across multiple frontier LLMs.
  • Analyzed performance against complex vulnerabilities including CVE-2026-34197 (ActiveMQ RCE) and CVE-2026-45185 (EXIM RCE).
  • Evaluated model-specific outputs from ChatGPT-5.5 and Claude Opus 4.8.

• Technical Failure Modes: The 'FLAWED' Phenomenon

  • Classification of FLAWED artifacts: Fixes that are syntactically correct but architecturally insufficient.
  • Fragile patching: Models prioritize blocking specific PoC input characters over addressing the root vulnerability.
  • Regressive security risk: 2.3% of patches introduced entirely new vulnerabilities during the remediation attempt.
  • Functional disruption: 20.1% of successful vulnerability resolutions resulted in unintended changes to application behavior.

• Quantitative Impact Data

  • 53.9% overall failure rate for patches targeting complex, non-trivial vulnerabilities.
  • 49.3% failure to remove at least one viable, exploitable attack path.
  • ~26% success rate for patches achieving full remediation without altering application functionality.
  • Cost-benefit misalignment: Low per-patch generation costs ($2.11–$2.81) are offset by the high cost of mandatory expert human validation.

• Defensive Implications & Mitigation Strategies

  • Avoidance of "automation bias" in CI/CD pipelines where AI code is accepted without deep security inspection.
  • Requirement for "execution-grounded" verification to test patches against diverse, non-PoC exploitation vectors.
  • Implementation of human-in-the-loop controls as a mandatory final security gate for all AI-generated remediation.
  • Shift focus from input-filtering (syntax-based) to structural remediation (architectural-based).