FlagThis

Abstract

Large Language Models (LLMs) are increasingly adopted for automated vulnerability detection, but the optimal way to represent program structure and semantics for LLM-based vulnerability reasoning remains unclear. Most existing prompting-based approaches expose the model directly to raw source code, implicitly assuming that more source-level context provides better evidence. This paper challenges that assumption through a systematic empirical benchmark of raw source code and various static-analysis-based code representations for LLM vulnerability analysis. We present RepBench, a unified benchmark pipeline that converts real-world C/C++ vulnerability testcases into multiple program representations—raw source code, Abstract Syntax Trees (ASTs), Control-Flow Graphs (CFGs), Program Dependence Graphs (PDGs), and representation combinations thereof—along with an auxiliary track of enriched Program Dependence Graphs (ePDGs). Using a curated PrimeVul-derived corpus of 107 standard Joern-based testcases across five CWE categories, we evaluate ten representation variants under a fixed Chain-of-Thought (CoT) prompting and structured-output protocol, with an additional 19 auxiliary ePDG cases generated through an external VulChecker/Hector pipeline. Our results show that representation choice has a substantial effect on LLM vulnerability reasoning. The strongest variant, AST+PDG, achieves 83.2% curated accuracy compared with 53.5% for the raw-source baseline—a 29.7 percentage point improvement. At the prompt-family level, graph-only prompts outperform both source-only and source-plus-graph prompts while requiring substantially lower prompt overhead. This reveals a counter-intuitive context dilution effect: adding raw source code to compact structural graph evidence often degrades reasoning performance rather than improving it. Our findings suggest that carefully selected structural representations offer a stronger accuracy-prompt-overhead tradeoff than simply exposing LLMs to more raw input, and that static analysis can serve as an effective prompt-construction layer for security-focused LLM reasoning.