FlagThis

Abstract

Enterprises are moving toward _autonomous cyber defense_ : agentic AI that builds situational awareness of an organizations security state and reasons from it to assessments, decisions, and actions. This rests on a _holistic view of the enterprises security state_ , the continuous, cross-vendor picture of identities, cloud and infrastructure, data, applications, and their configurations that security posture management assembles. As agents take on this work, what matters is not whether an agent _can_ produce an answer but whether it _should be trusted_ to. The field cannot yet answer this question. Real enterprise environments are private, crossvendor, and deeply correlated, and none is exposed publicly as a shared, queryable target for evaluating such agents end to end. We call this the _environment data gap_ . We present **Open Security Benchmark (OSB)** , a framework that benchmarks agentic AI on this work. OSB surfaces a curated enterprise environment-a frozen, holistic view of the security state-and evaluates posture investigation across two modalities: _text-to-SQL_ over a relational snapshot and each vendors _native API_ over a served instance of the same environment. Freezing the environment pins the target state as an immutable snapshot and anchors answers to a closed-form ground truth. OSB is built from five components: a data layer, a task and evaluation-set layer, a multi-dimensional scoring layer, a minimal auditable harness, and a bring-your-own path that serves public comparison and private tenant evaluation from one substrate. We instantiate the framework with two identity-security packs and a family of synthetic-organization environment datasets spanning multiple scales, and chart its extension to further posture subdomains, investigation modalities, and defense stages from assessment toward remediation.