Adaptive detection of encrypted malware traffic via fully convolutional masked autoencoders

Network traffic encryption techniques are widely adopted to protect data confidentiality and prevent privacy leakage during data transmission. However, malware often leverages these traffic encryption techniques to conceal malicious activities. Recent research has demonstrated the effectiveness of machine and deep learning-based malware traffic detection methods. However, these methods rely on a sufficient amount of labeled data readily available for model training, limiting the capability of transferring to new malware detection. In this paper, we propose Malcom, an adaptive encrypted malware traffic detection method based on fully convolutional masked autoencoders to detect malware traffic hidden in the encrypted traffic. We first propose a novel traffic representation named Header-Payload Matrix (HPM) to extract discriminative features that can differentiate from malware and benign traffic. Subsequently, we develop a hierarchical ConvNeXt traffic encoder and a lightweight ConvNeXt traffic decoder to learn high-level features from a large amount of unlabeled data. The masked autoencoder framework enables our model to be adaptive to new malware detection by fine-tuning with only a few labeled data. We conduct extensive experiments with real-world datasets to evaluate Malcom. The results demonstrate that Malcom outperforms the state-of-the-art (SOTA) methods in two typical scenarios. Particularly, in the scenario of few-shot learning, Malcom achieves an average F1 score of 97.35%, with an improvement of 8.24% over the SOTA method, by fine-tuning with only 10 samples per malware type.