FlagThis

Abstract

Ransomware encrypts victim data in seconds; however, current machine learning detectors use Ransomware detection symmetric loss functions that equally penalize missed detections, known as false negatives (FN), and Deep reinforcement learning benign false alarms, known as false positives (FP). This assumption is misaligned with operational Asymmetric reward design reality, where an FN causes irreversible data loss and high recovery costs, while an FP is reversible. False-negative minimization Detection is further complicated by ransomware variants with diverse behaviors, reducing the Behavioral ransomware analysis effectiveness of fixed supervised weighting strategies. To address these challenges, this study proposes DDQN a Security-Aware Deep Reinforcement Learning (SA-DRL) framework that embeds FNFP cost asymmetry into the reinforcement learning reward signal, optimizing detection policies to minimize missed detections. The framework also introduces a Security-Optimal Model Selection (SOMS) criterion and an adaptive sample-weighting mechanism through episode-level random permutation. Four DRL agents, DQN, DDQN, PPO, and A2C were trained using a symmetric baseline reward ( __ 1) and a security-aware asymmetric reward ( __ 2) that penalizes FNs more heavily. Each configuration was evaluated with four discount factors, five-fold cross-validation, and three random seeds, resulting in 480 training runs on a balanced dataset. The SOMS criterion prioritizes minimizing the false-negative rate (FNR), maximizing the F1-score, and minimizing training time. Results show that asymmetric reward shaping improves detection performance. The SOMS-selected configuration, DDQN with __ 2 and __ = 0 _._ 1, achieved an FNR of 0.0080, an F1-score of 0.9915, and an AUC of 0.998, reducing missed detections by 67.6% compared with the best baseline model, MLP with FNR = 0.0247. __ 2 reduced the mean FNR by 43% relative to __ 1 across all configurations. These findings highlight the importance of reward-function design in security-sensitive detection systems. SA-DRL is the first framework to combine runtime features, asymmetric reward design, and a security-first model-selection criterion with statistical validation for ransomware detection.