FlagThis

Abstract

Security misconfigurations in Terraform Infrastructure-as-Code represent a documented and growing attack surface in cloud deployments, and large language models are increasingly applied as automated repair agents. Existing evaluations remain inadequate: most studies declare a repair successful when the targeted static-analysis finding disappears, ignoring planning validity, behavioral comparison, and security-intent alignment. This paper presents TerraProbe, a five-layer oracle evaluation framework applied to 288 first-pass LLM-generated repairs produced by three models (gemini-2.5-flash-lite, GPT-4o, and Claude 3.5 Sonnet) across two tracks (68 real-world TerraDS modules and 28 controlled injected-defect modules). Statistical comparison between tracks using chi-square tests and Fisher exact tests reveals that plan-comparison reachability differs significantly (chi-sq=31.64, p < 0.001, Cohens h=1.36) and that deceptive-fix rates in TerraDS adjudicated cases are statistically indistinguishable across all three models (57.1%-71.4%, Fisher exact p > 0.10 for all pairwise comparisons), confirming that, under the conditions studied (first-pass repairs, minimal security-intent-free prompts, and three frontier instruction-following models), the deceptive-fix pattern is systemic rather than model-specific. Three candidate mechanisms (training distribution bias, the check specification gap, and prompt under-specification) are analyzed to explain why deceptive fixes arise regardless of model capability. The paper introduces a formally defined taxonomy of deceptive fixes organized along four dimensions (Mechanism, Intent Alignment, Security Impact, and Detection Difficulty), with inter-rater reliability Kappa=0.78 (Krippendorff alpha=0.76). IAM permission-level analysis confirms that wildcard Resource grants are preserved post-repair in all nine CKV2 AWS 1 1 deceptive-fix cases, representing unmitigated privilege escalation risk. A full replication package, including prompts, corpus, evaluation scripts, and a reproducibility Docker image, is made available. The paper also presents the Multi-Layer Oracle Evaluation (MLOE) framework, abstracting TerraProbe into a domain-general evaluation design for any IaC technology. TerraProbe therefore contributes an evaluation methodology rather than another repair model, distinguishing intent-aligned security repairs from scanner-passing false successes.