Current research from IBM, the NDSS Symposium, and Boston University's PEAC Lab indicates that Large Language Models (LLMs) are fundamentally insufficient for autonomous vulnerability discovery and risk-based prioritization. While LLMs demonstrate pattern recognition capabilities, they suffer from high false-positive rates and a systemic lack of architectural context, preventing them from understanding how vulnerabilities interact with specific deployment environments. This creates an "automation paradox," where the volume of unverified LLM-generated findings increases the manual verification workload for Application Security (AppSec) professionals. Furthermore, models demonstrate a critical failure in reasoning about actual exploitability, making them unreliable for determining the real-world risk of identified security flaws.

• Research Overview: The LLM Capability Gap
  • Disconnect between pattern recognition and nuanced security reasoning.
  • Inability to incorporate environmental or architectural context into findings.
  • Significant failure in predicting actual exploitability and risk likelihood.
• Technical Methodology: Evaluation Frameworks
  • IBM Research: Development of comprehensive benchmarks for security reasoning.
  • NDSS Symposium (2026): Technical analysis of LLM-based detection reliability.
  • BU PEAC Lab: Rigorous testing of LLM reasoning within security contexts.
• The Automation Paradox: AppSec Operational Impact
  • High false-positive rates driving increased manual verification toil.
  • LLM outputs acting as a noise multiplier rather than a force multiplier.
  • Deterioration of triage efficiency due to unverified AI-generated alerts.
• Key Findings: Reasoning & Exploitability Failures
  • Models struggle to distinguish between theoretical flaws and exploitable vectors.
  • Lack of semantic depth required to assess real-world runtime impact.
  • Failure to align with established exploitability prediction models.
• Industry Implications: Defensive Response
  • Necessity for robust "Human-in-the-loop" (HITL) verification protocols.
  • Requirement for context-aware models integrated with deployment metadata.
  • Shift in strategy from autonomous discovery to AI-augmented triage.