Quantifying the Limitations of Learning-Assisted Grammar-Based Fuzzing

Abstract

Grammar-based fuzzing is effective at finding the vulnerabilities of various input-parsing programs which take as inputs complex data conforming to a certain grammar. Traditional grammar-based fuzzing techniques require a manually-generated grammar for valid test input generation. However, writing an input grammar by hand has two major drawbacks: (1) it is costly and error-prone and (2) it has no capability to generate interesting inputs which induce high test-coverage (for finding many vulnerabilities). To address these problems, a state-of-the-art technique, Learn&Fuzz, automatically generates an input grammar via deep neural network-based statistical learning. Even Learn&Fuzz, however, has significant limitations; especially, it cannot successfully generate a (long) sequence of instructions (consisting of opcode plus zero or more operands), which contribute to high test-coverage of instruction-interpreting code. In this paper, we focus on and quantify the limitations of the current learning-assisted grammar-based fuzzing, i.e, how ineffective it is at generating instruction sequences triggering high test coverage. Through our experiments using a re-implementation of Learn&Fuzz and real instruction-interpreting code, we measure the test-coverage of the target code when tested by Learn&Fuzz. Our experimental results show the coverage is surprisingly low, and the analysis of the results open up new research directions to enhance learning-assisted grammar-based fuzzing.