FlagThis

Abstract

Open-weight medical language models are increasingly used as the base of patient-facing and clinician-support applications. Their model cards prohibit specific behaviors recommending exact drug dosages, issuing definitive diagnoses, prescribing treatments, adjudicating drugdrug interactions, and advising that emergency care can be skippedyet a model card describes _intended_ behavior, not _robust_ behavior. We quantify that gap for MedGemma-4B-it under attacks that require no technical sophistication. We build a fully factorial benchmark of 5 guardedbehavior concepts __ 50 deterministically templated questions __ 6 lay-accessible attack manners __ 3 repetitions (4 _, 500 generations), serve the model locally through Ollama under default sampling, and code every response refuse/hedge/comply with three independent judges (an LLM judge, a transparent regex judge, and an NLI-entailment judge). Under the primary LLM judge the overall Attack Success Rate (ASR, the fraction coded comply) is 38 _.0_ %. The two framings that reinterpret the request as legitimate dominate: recasting a question as a medical board exam item raises ASR from a 29 _.0_ % baseline to 53 _.1_ % (= +24 _.0_ points), and an appeal to an alleged doctors authority raises it to 43 _.7_% (= +14 _.7_ ); crude instruction-override prefixes do not move the needle significantly. Robustness is dominated by _topic_: the drug-interaction guardrail is nearly absent (83 _.2_% ASR) while the emergency-deferral guardrail is strong (4 _.7_ %) and the authority framing is the _only_ attack that breaches it. We report Wilson confidence intervals, cluster-bootstrap effect sizes, a cluster-robust logistic regression, Cochrans _Q_, per-manner McNemar tests, and inter-judge reliability (Fleiss __ = 0 _.26_), and we discuss why absolute ASR is judge-dependent while the ordering of attacks and topics is not. Our findings motivate deployment-time guardrailspolicy refusals, harm classifiers, and retrieval with citationfor open medical models.