Optimized Mutation of Grey-box Fuzzing: A Deep RL-based Approach

As a vulnerability discovery technique, fuzzing has been widely used in the field of software test in the past years. Traditional fuzzing has several drawbacks, including poor efficiency, low code coverage, and a high dependence on expert experience. By introducing the deep reinforcement learning technique, one can train the mutator of the fuzzer to move in a desired direction, such as maximizing code coverage or finding more code paths. This paper proposes a reinforcement learning-based fuzzing method to enhance the code coverage and explore potential code vulnerabilities. First, the concept of the input field is introduced to the seed file, reducing invalid operations by marking whether each byte of the seed file is a valid byte. Then, we optimize mutation by modeling the grey-box fuzzing as a reinforcement learning problem and training mutator's behavior on test cases. By observing the rewards caused by mutating with a specific set of actions performed on an initial program input, the fuzzing agent learns a policy that can next generate new higher-reward inputs. Finally, experimental results show that the proposed deep reinforcement learning-based fuzzing method outperforms the baseline random fuzzing algorithms.