CBSeq: A Channel-Level Behavior Sequence for Encrypted Malware Traffic Detection

Susu Cui, Cong Dong, Meng Shen, Yuling Liu, Bo Jiang*, Zhigang Lu

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

4 Citations (Scopus)

Abstract

Machine learning and neural networks have become increasingly popular solutions for encrypted malware traffic detection. They mine and learn complex traffic patterns, enabling detection by fitting boundaries between malware traffic and benign traffic. Compared with signature-based methods, they have higher scalability and flexibility. However, affected by the frequent variants and updates of malware, current methods suffer from a high false positive rate and do not work well for unknown malware traffic detection. It remains a critical task to achieve effective malware traffic detection. In this paper, we introduce CBSeq to address the above problems. CBSeq is a method that constructs a stable traffic representation, behavior sequence, to characterize attacking intent and achieve malware traffic detection. We novelly propose the channels with similar behavior as the detection object and extract side-channel content to construct behavior sequence. Unlike benign activities, the behavior sequences of malware and its variant's traffic exhibit solid internal correlations. Moreover, we design the MSFormer, a powerful Transformer-based multi-sequence fusion classifier. It captures the internal similarity of behavior sequence, thereby distinguishing malware traffic from benign traffic. Our evaluations demonstrate that CBSeq performs effectively in various known malware traffic detection and exhibits superior performance in unknown malware traffic detection, outperforming state-of-the-art methods.

Original languageEnglish
Pages (from-to)5011-5025
Number of pages15
JournalIEEE Transactions on Information Forensics and Security
Volume18
DOIs
Publication statusPublished - 2023

Keywords

  • Malware traffic
  • behavior sequence
  • encrypted traffic
  • transformer
  • unknown detection

Fingerprint

Dive into the research topics of 'CBSeq: A Channel-Level Behavior Sequence for Encrypted Malware Traffic Detection'. Together they form a unique fingerprint.

Cite this