FlagThis

Abstract

Recent deep learning approaches for network intrusion detection increasingly incorporate temporal architectures such as recurrent networks and Transformers, often reporting near-perfect performance on CIC-IDS2017. However, many existing studies neither supply their temporal modules with genuine sequence inputs nor evaluate under realistic, leakage-free conditions, making it unclear whether reported gains arise from true sequence-modelling capability. In this work, we reformulate CICIDS2017 as a temporal intrusion-detection task by constructing ordered flow sequences from network conversations and benchmarking nine classical and deep learning architectures under a random split, two leakage-free splits, and a padding-scheme ablation. The central finding is that padding convention, not architecture, determines the Transformers performance: on genuinely sequential (non-padded) windows the Transformer achieves the highest macro-F1 of any model in the experiment (0.89); under zero-pad+mask evaluation it drops markedly (0.24 macro-F1), while LSTM, GRU, and 1D-CNN remain stable. Under leakagefree group evaluation the Random Forest is the most robust model (0.779), while the Transformers false-alarm rate grows from 0.04% to 2.7% a 67x increase invisible under conventional protocols. These findings demonstrate that evaluation methodology specifically padding convention and split protocol has a larger effect on reported performance than architectural choice, and that widely used random splits with repeat-last padding can overestimate model robustness by up to 0.24 macro-F1. We advocate for leakage-free splits, explicit padding disclosure, and sequence-aware benchmarking as standard practice in future IDS research.