Polymorphic Code Detection with GA Optimized Markov Models

  • Udo Payer
  • Stefan Kraxberger
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3677)


This paper presents our progression in the search for reliable anomaly-based intrusion detection mechanisms. We investigated different options of stochastic techniques. We started our investigations with Markov chains to detect abnormal traffic. The main aspect in our prior work was the optimization of transition matrices to obtain better detection accuracy. First, we tried to automatically train the transition matrix with normal traffic. Then, this transition matrix was used to calculate the probabilities of a dedicated Markov sequence. This transition matrix was used to find differences between the trained normal traffic and characteristic parts of a polymorphic shellcode. To improve the efficiency of this automatically trained transition matrix, we modified some entries in a way that byte-sequences of typical shellcodes substantially differs from normal network behavior. But this approach did not meet our requirements concerning generalization. Therefore we searched for automatic methods to improve the matrix. Genetic algorithms are adequate tools if just little knowledge about the search space is available and the complexity of the problem is very hard (NP-complete).


intrusion detection polymorphic shellcode detection markov models genetic algorithms optimization 


  1. [Cj04]
    Chang, J.: Stochastic Processes (Accessed 2004/11/17),
  2. [CLET03]
    CLET Team: Polymorphic shellcode engine. Phrack Magazine 49(14)Google Scholar
  3. [ADM03]
    ADMmutate: ADMmutate shellcode engine (Accessed 2004/11/24),
  4. [Mm03]
    Mahoney, M.: Network traffic anomaly detection based on packet bytes. In: Proc. ACM-SAC (2003)Google Scholar
  5. [KL04]
    Kolesnikov, O., Lee, W.: Advanced Polymorphic Worms: Evading IDS by blending in with normal traffic (Accessed 2004/11/16),
  6. [Oe02]
    Oswald, E.: Enhancing Simple Power-Analysis Attacks on Elliptic Curve Cryptosystems. In: CHES 2002 4th International Workshop on Cryptographic Hardware and Embedded Systems, Redwood Shores, CA, USA (2002)Google Scholar
  7. [DHS00]
    Duda, O.R., Hart, E.P., Stork, G.D: Pattern Classification. Wiley Intersience. New York (2000)Google Scholar
  8. [RN95]
    Russell, A., Norvig, P.: Artificial Intelligence: A Modern Approach. Prentice-Hall, Englewood Cliffs (1995)Google Scholar
  9. [PJ84]
    Pearl, J.: Heuristics: Intelligent search strategies for computer problem solving. Addison-Wesley, Reading (1984)Google Scholar
  10. [HJK95]
    Houck, C., Joines, C., Kay, M.: A Genetic Algorithm for Function Optimization - A Matlab Implementation. NCSU-IE TR 95-09Google Scholar
  11. [Yn01]
    Ye, N., et al.: Probabilistic Techniques for Intrusion Detection Based on Computer Audit Data. IEEE Transactions on Systems, man and cybernetics - Part A: Systems and Humans 31(4) (July 2001)Google Scholar
  12. [YEZ02]
    Ye, N., Ehiabor, T., Zhang, Y.: First-order versus high-order stochastic models for computer intrusion detection. Quality and realiability engineering international (2002)Google Scholar
  13. [Yn00]
    Ye, N.: A Markov chain model of temporal behavior for anomaly detection. In: Proc. of the, IEEE Systems, Man, and Cybernetics Information Assurance and Security Workshop, New York (2000)Google Scholar
  14. [JTM01]
    Jha, S., Tan, K., Maxion, R.A.: Markov Chains, Classifiers, and Intrusion Detection. In: CSFW 2001 14th IEEE Computer Security Foundations Workshop, Cape Breton, Novia Scotia, Canada (2001)Google Scholar
  15. [JV99]
    Ju, W.H., Vardi, Y.: A hybrid high-order Markov chain model for computer intrusion detection. Technical Report. TR92. National Institute of Statistical Sciences (1999)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2005

Authors and Affiliations

  • Udo Payer
    • 1
  • Stefan Kraxberger
    • 2
  1. 1.Institute for Applied Information Processing and Communications (IAIK)University of Technology GrazAustria
  2. 2.Stiftung – Secure Information and Communication Technologies (SIC)GrazAustria

Personalised recommendations