RBFUZZ: Network Protocol Fuzzing Guided by Rare Branch

As network protocols grow increasingly complex, traditional greybox protocol fuzzing faces several challenges, particularly in state and seed selection strategies, which do not take into account branches with low execution frequency that may contain key methods of the protocol. These branches, referred to as rare branches, may reduce the effectiveness of fuzzing. To address these challenges, we propose RBFUZZ, a rare branch guided protocol fuzzing approach that enhances state selection and seed selection. To improve state selection, RBFUZZ adopts a strategy that incorporates the rare branch score as a new criterion and uses the TOPSIS decision-making method to evaluate protocol states by comprehensively considering this criterion with AFLNET’s original criteria. To improve the seed selection, we propose a rare branch guided strategy that prioritizes seeds capable of executing the least-executed branches associated with a given protocol state. We further evaluate the performance of RBFUZZ by comparing with AFLNET, AFLNWE and StateAFL, on 13 typical protocol implementations from ProFuzzBench. The experimental results show that RBFUZZ discovers 15.36%, 41.63% and 30.60% more paths, 49.26%, 187.43% and 57.19% more crashes than AFLNET, AFLNWE, and StateAFL on average, respectively. Besides, RBFUZZ discovers 50.0% more states and 21.59% state transitions than AFLNET on average. That highlights RBFuzz could improve the effectiveness of fuzzing.