Detecting Communication Protocol Security Flaws by Formal Fuzz Testing and Machine Learning

  • Guoqiang Shu
  • Yating Hsu
  • David Lee
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5048)


Network-based fuzz testing has become an effective mechanism to ensure the security and reliability of communication protocol systems. However, fuzz testing is still conducted in an ad-hoc manner with considerable manual effort, which is mainly due to the unavailability of protocol model. In this paper we present our on-going work of developing an automated and measurable protocol fuzz testing approach that uses a formally synthesized approximate formal protocol specification to guide the testing process. We adopt the Finite State Machine protocol model and study two formal methods for protocol synthesis: an active black-box checking algorithm that has provable optimality and a passive trace minimization algorithm that is less accurate but much more efficient. We also present our preliminary results of using this method to implementations of the MSN instant messaging protocol: MSN clients Gaim (pidgin) and aMSN. Our testing reveals some serious reliability and security flaws by automatically crashing both of them.


Fuzz testing Security Testing Protocol Synthesis 


  1. 1.
    Angulin, D.: Learning regular sets from queries and counterexamples. Information and Computation 75, 87–106 (1987)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Cui, W., Kannan, J., Wang, H.: Discoverer: Automatic Protocol Reverse Engineering from Network Traces. In: The 16th USENIX Security Symposium (2007)Google Scholar
  3. 3.
    Dolev, D., Yao, A.: On the security of public-key protocols. IEEE Transaction on Information Theory 29, 198–208 (1983)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Godefroid, P., Klarlund, N., Sen, K.: DART: Directed Automated Random Testing. In: Proceedings of PLDI 2005 (ACM SIGPLAN 2005 Conference on Programming Language Design and Implementation), pp. 213–223 (2005)Google Scholar
  5. 5.
    Godefroid, P., Levin, M.Y., Molnar, D.: Automated Whitebox Fuzz Testing. Technical Report MS-TR-2007-58, Microsoft (May 2007)Google Scholar
  6. 6.
    Gören, S., Ferguson, F.J.: On state reduction of incompletely specified finite state machines. Computers and Electrical Engineering 33(1), 58–69 (2007)CrossRefzbMATHGoogle Scholar
  7. 7.
    Howard, M.: Inside the Windows Security Push. IEEE Security & Privacy, 57–61 (2003)Google Scholar
  8. 8.
    Lee, D., Yannakakis, M.: Principles and methods of testing finite state machines - A survey. In: Proceedings of the IEEE, 1090–1123 (1996)Google Scholar
  9. 9.
    Oehlert, P.: Violating Assumptions with Fuzzing. IEEE Security & Privacy, pp. 58-62 (2005)Google Scholar
  10. 10.
    Peled, D., Vardi, M.Y., Yannakakis, M.: Black-box checking. In: Proceedings of IFIP FORTE/PSTV (1999)Google Scholar
  11. 11.
    Shu, G., Lee, D.: Testing Security Properties of protocol implementations – a machine learning based approach. In: Proceedings of IEEE ICDCS (2007)Google Scholar
  12. 12.
    Wang, L., Ellis, C., Yin, W., Luong, D.D.: Hercules: An Environment for Large-Scale Enterprise Infrastructure Testing. In: Proceedings of the Workshop on Advances and Innovations in Systems Testing (2007)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2008

Authors and Affiliations

  • Guoqiang Shu
    • 1
  • Yating Hsu
    • 1
  • David Lee
    • 1
  1. 1.Department of Computer Science and Engineeringthe Ohio State UniversityColumbusUSA

Personalised recommendations