Early-Stage Detection of Encrypted Malware Traffic via Multi-flow Temporal Graph Learning

Malware widely adopts network traffic encryption techniques to conceal malicious activities. Recent research has demonstrated the effectiveness of machine learning (ML)-, deep learning (DL)-, and pre-training-based malware traffic detection methods. However, a vast majority of these methods rely on the collected complete traffic during the malware attack. While certain methods can operate on partial traffic, their detection accuracy often significantly decreases when the available data is restricted to the extreme early stage, where information is most sparse. In this paper, we propose DawnGuard, an effective early-stage encrypted malware traffic detection framework through multi-flow temporal graph learning. Specifically, based on the temporal packet density distribution analysis, DawnGuard innovatively proposes a self-adjusting data augmentation strategy for early-stage malware traffic, which can force the model to focus on the early-stage interaction phase with more distinguishable properties. Meanwhile, considering that temporal-topological correlations among multiple flows can provide more distinguishable properties in a malware attack, we further develop a temporal graph learning framework to extract features, which can form Multi-Flow Graph Features (MGF). By utilizing MGF, Dawn-Guard implements a Vision Transformer-based detection mechanism, enabling accurate and precise encrypted malware traffic detection with early-stage traffic by capturing both local and global contextual relationships. Extensive experiments with two real-world datasets demonstrate that DawnGuard outperforms the state-of-the-art (SOTA) methods in three typical scenarios: varying early-stage time windows, imbalanced data, and unseen malware detection. Particularly, DawnGuard achieves an average F1 of 95.11%, 8.7% higher than the SOTA method, by only utilizing the first 20% loading ratio of complete traffic.