ReTrial: Robust Encrypted Malicious Traffic Detection via Discriminative Relation Incorporation and Misleading Relation Correction

Encryption techniques greatly ensure the confidentiality and integrity of network communications. However, they also allow attackers to conceal malicious activities within encrypted traffic, posing severe cybersecurity challenges. Current detection methods primarily rely on statistics and correlation analysis. However, both statistical features and inter-entity relations can be easily obfuscated. Moreover, issues with low-quality data and fixed feature sets limit the generalizability and adaptability to defend against various evasion techniques. Robustifying encrypted malicious traffic detection in adverse conditions is still an open problem. In this paper, we propose ReTrial, a robust encrypted malicious traffic detection system via discriminative relation incorporation and misleading relation correction. The key motivations behind ReTrial are to accurately leverage the rich relations among flows for contextual analysis, and correct misleading ones for robust threat detection. Specifically, we construct a relational multigraph and develop a tailored Graph Attention Network (GAT) to selectively incorporate contextual information. Then we retrieve multi-order neighborhood similarity graphs as observations for adaptive relation correction. Following an iterative scheme, both detector performance and graph topology mutually optimize. To validate the robustness of ReTrial, we simulate various adverse conditions by randomly dropping packets and greedily injecting perturbation edges. The experimental results show that ReTrial is competitive in ideal condition. Under adverse conditions, though the performances of other state-of-the-art methods degrade significantly, ReTrial consistently exhibits superior performance with a maximum reduction of only 5.88% in F1, highlighting its robustness in threat detection.