FlagThis

Abstract

Membership inference (MIA) on language models is usually summarised by an aggregate ROC-AUC, but recent work shows these evaluations are confounded: model-free _blind_ baselines separate members from non-members from surface text alone (Das et al., 2025). We study black-box, sampling-based training-data leakage through a probabilistic lens. Treating _N_ independent samples from _p_ ( _ | x_ ) as an empirical estimate of a models output distribution, we cast overlap-based leakage signals as functionals of that distribution (mean overlap, extreme value, and self-concentration). Under this view we confirm and extend the blind-baseline critique into the sampling regime: on WikiMIA a blind bag-of-words classifier reaches AUC 0 _._ 97 (TPR@5%FPR 0 _._ 90) while sampling statistics add nothing, and on an IID Pile split (MIMIR) neither self-concentration nor gold-continuation recovery significantly exceeds a blind baseline in aggregate (incremental AUC 95% CI includes zero). Aggregate metrics, however, hide the real harm. The same probabilistic sampling _verbatim-extracts_ training data for a tail of documents that no blind attack can reach. On Pythia-6.9B, 16 _._ 6% of 500 Pile documents bearing a real identifier (83 documents; 21 _._ 3% of those bearing an email address) have that exact identifier reproduced _and_ not reproduced under a mismatched-prefix control, so each leak is attributable to that specific document rather than to a globally common string. This is a per-document disclosure that aggregate AUC cannot see. The risk is uneven, and we report the decomposition rather than only the average: identifier leakage is __ 3 __ stronger in code than in prose, though prose stays clearly positive and grows with capacity (4 _._ 0% to 12 _._ 1% from 410M to 6 _._ 9B), while recovery of _arbitrary_ held-out continuations is essentially confined to code (+0 _._ 44 member gap on GitHub vs __ +0 _._ 014 on prose). We characterise the extraction regime: temperature and nucleus sampling have minor effect, a 16-token prefix already suffices, and the sample budget needed scales inversely with prefix length. We detect no reduction from training on a deduplicated corpus. Our results argue that language-model privacy audits should report per-document extraction, not aggregate membership, and motivate differential privacy as the mitigation. We release leakit, a black-box extraction-audit tool.