ReFormat: Automatic Reverse Engineering of Encrypted Messages

  • Zhi Wang
  • Xuxian Jiang
  • Weidong Cui
  • Xinyuan Wang
  • Mike Grace
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5789)


Automatic protocol reverse engineering has recently received significant attention due to its importance to many security applications. However, previous methods are all limited in analyzing only plain-text communications wherein the exchanged messages are not encrypted. In this paper, we propose ReFormat, a system that aims at deriving the message format even when the message is encrypted. Our approach is based on the observation that an encrypted input message will typically go through two phases: message decryption and normal protocol processing. These two phases can be differentiated because the corresponding instructions are significantly different. Further, with the help of data lifetime analysis of run-time buffers, we can pinpoint the memory locations that contain the decrypted message generated from the first phase and are later accessed in the second phase. We have developed a prototype and evaluated it with several real-world protocols. Our experiments show that ReFormat can accurately identify decrypted message buffers and then reveal the associated message structure.


Security Reverse Engineering Network Protocols Data Lifetime Analysis Encryption 


  1. 1.
    Paxson, V.: Bro: A System for Detecting Network Intruders in Real-Time. Computer Networks 31(23-24), 2345–2463 (1999)CrossRefGoogle Scholar
  2. 2.
    Wang, H.J., Guo, C., Simon, D.R., Zugenmaier, A.: Shield: Vulnerability-Driven Network Filters for Preventing Known Vulnerability Exploits. In: Proceedings of ACM SIGCOMM 2004, pp. 193–204 (2004)Google Scholar
  3. 3.
    Cui, W., Peinado, M., Wang, H.J., Locasto, M.: Shieldgen: Automatic Data Patch Generation for Unknown Vulnerabilities with Informed Probing. In: Proceedings of 2007 IEEE Symposium on Security and Privacy, Oakland, CA (May 2007)Google Scholar
  4. 4.
    The Protocol Informatics Project,
  5. 5.
    Caballero, J., Song, D.: Polyglot: Automatic Extraction of Protocol Format using Dynamic Binary Analysis. In: Proceedings of the 14th ACM Conference on Computer and and Communications Security, CCS 2007 (2007)Google Scholar
  6. 6.
    Cui, W., Kannan, J., Wang, H.J.: Discoverer: Automatic Protocol Reverse Engineering from Network Traces. In: Proceedings of the 16th USENIX Security Symposium (Security 2007), Boston, MA (August 2007)Google Scholar
  7. 7.
    Cui, W., Peinado, M., Chen, K., Wang, H.J., Irun-Briz, L.: Tupni: Automatic Reverse Engineering of Input Formats. In: Proceedings of the 15th ACM Conferences on Computer and Communication Security, CCS 2008 (October 2008)Google Scholar
  8. 8.
    Lin, Z., Jiang, X., Xu, D., Zhang, X.: Automatic Protocol Format Reverse Engineering Through Context-Aware Monitored Execution. In: Proceedings of the 15th Annual Network and Distributed System Security Symposium (NDSS 2008) (February 2008)Google Scholar
  9. 9.
    Wondracek, G., Comparetti, P.M., Kruegel, C., Kirda, E.: Automatic Network Protocol Analysis. In: Proceedings of the 15th Annual Network and Distributed System Security Symposium (NDSS 2008) (February 2008)Google Scholar
  10. 10.
    Comparetti, P.M., Wondracek, G., Kruegel, C., Kirda, E.: Prospex: Protocol Specification Extraction. In: Proceedings of 2009 IEEE Symposium on Security and Privacy, Oakland, CA (May 2009)Google Scholar
  11. 11.
    Chow, J., Pfaff, B., Christopher, K., Rosenblum, M.: Understanding Data Lifetime via Whole-System Simulation. In: Proceedings of the 13th USENIX Security Symposium, San Diego, CA (2004)Google Scholar
  12. 12.
    Newsome, J., Song, D.: Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software. In: Proceedings of the 14th Annual Network and Distributed System Security Symposium (NDSS 2005), San Diego, CA (February 2005)Google Scholar
  13. 13.
    SHTTP: An Embeddable Web Server,
  14. 14.
    Know your Enemy: Tracking Botnets - Bot-Commands,
  15. 15.
    Wang, Z., Jiang, X., Cui, W., Wang, X.: Reformat: Automatic Reverse Engineering of Encrypted Messages (Department of Computer Science Technical Report, North Carolina State University, TR-2008-26) (2008)Google Scholar
  16. 16.
    Ircd-hybrid – High Performance Internet Relay Chat,
  17. 17.
    Cui, W., Paxson, V., Weaver, N., Katz, R.H.: Protocol-Independent Adaptive Replay of Application Dialog. In: Proceedings of the 13th Annual Network and Distributed System Security Symposium (NDSS 2006), San Diego, CA (February 2006)Google Scholar
  18. 18.
    Leita, C., Mermoud, K., Dacier, M.: ScriptGen: An Automated Script Generation Tool for Honeyd. In: Srikanthan, T., Xue, J., Chang, C.-H. (eds.) ACSAC 2005. LNCS, vol. 3740, pp. 203–214. Springer, Heidelberg (2005)Google Scholar
  19. 19.
    Chow, J., Pfaff, B., Garfinkel, T., Rosenblum, M.: Shredding Your Garbage: Reducing Data Lifetime through Secure Deallocation. In: Proceedings of the 14th USENIX Security Symposium, Baltimore, Maryland (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Zhi Wang
    • 1
  • Xuxian Jiang
    • 1
  • Weidong Cui
    • 2
  • Xinyuan Wang
    • 3
  • Mike Grace
    • 1
  1. 1.North Carolina State UniversityUSA
  2. 2.Microsoft ResearchUSA
  3. 3.George Mason UniversityUSA

Personalised recommendations