FlagThis

Abstract

Large Language Models (LLMs) are increasingly deployed as general-purpose interfaces to information, code, and real-world services, making prompt-level security failures a practical concern. Jailbreak attacks, defensive wrappers, model safety policies, datasets, and automated judgers have all advanced rapidly, yet the field remains difficult to assess: prior work often uses incompatible threat models, access assumptions, cost budgets, datasets, and success criteria. This fragmentation makes reported attack success rates or defense gains hard to compare and can obscure whether progress comes from a stronger method, a weaker target, a different judging rule, or a larger query budget. This SoK systematizes LLM prompt security across concepts, data, tooling, and measurement. First, we propose three linked taxonomies for jailbreak attacks, defenses, and model vulnerabilities that separate technical mechanisms from attacker or defender capabilities. Second, we formalize threat, access, and cost assumptions as explicit evaluation metadata. Third, we release JAILBREAKDB, a curated corpus containing 445,752 jailbreak systemuser prompt pairs from 48 sources and 1,094,122 benign prompts from 14 sources, together with PROMPTSECURITY-EVAL, which contains 166,200 harmfulquery output records and 9,200 benign-utility output records. Fourth, we introduce PROMPTSECURITY, a modular evaluation platform that represents each experiment as a tuple of model, attack, defense, dataset, and judger, enabling accesscompatible and auditable comparisons. Finally, we use the platform to evaluate 11 target models, 21 attack settings, 10 defense states, and three judgers under matched settings, with compatible main-protocol measurements available through an interactive result browser. Our results show that model access regime, native harmful-query behavior, attack cost, defense backfire, taxonomy subcategory, and judger choice all materially affect security conclusions. Together, these artifacts and findings provide a foundation for reproducible, cost-aware, and taxonomy-grounded evaluation of LLM prompt security.