Diversification of System Calls in Linux Binaries

  • Sampsa RautiEmail author
  • Samuel Laurén
  • Shohreh Hosseinzadeh
  • Jari-Matti Mäkelä
  • Sami Hyrynsalmi
  • Ville Leppänen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9473)


This paper studies the idea of using large-scale diversification to protect operating systems and make malware ineffective. The idea is to first diversify the system call interface on a specific computer so that it becomes very challenging for a piece of malware to access resources, and to combine this with the recursive diversification of system library routines indirectly invoking system calls. Because of this unique diversification (i.e. a unique mapping of system call numbers), a large group of computers would have the same functionality but differently diversified software layers and user applications. A malicious program now becomes incompatible with its environment. The basic flaw of operating system monoculture – the vulnerability of all software to the same attacks – would be fixed this way.

Specifically, we analyze the presence of system calls in the ELF binaries. We study the locations of system calls in the software layers of Linux and examine how many binaries in the whole system use system calls. Additionally, we discuss the different ways system calls are coded in ELF binaries and the challenges this causes for the diversification process. Also, we present a diversification tool and suggest several solutions to overcome the difficulties faced in system call diversification. The amount of problematic system calls is small, and our diversification tool manages to diversify the clear majority of system calls present in standard-like Linux configurations. For diversifying all the remaining system calls, we consider several possible approaches.


  1. 1.
    Apvrille, A., Strazzere, T.: Reducing the window of opportunity for android malware gotta catch ’em all. Int. J. Ambient Comput. Intell. 8(1–2), 61–71 (2012)Google Scholar
  2. 2.
    Bruschi, D., Cavallaro, L., Lanzi, A.: An efficient technique for preventing mimicry and impossible paths execution attacks. In: Performance, Computing, and Communications Conference, 2007, IPCCC 2007. IEEE Internationa, pp. 418–425, April 2007Google Scholar
  3. 3.
    Chew, M., Song, D.: Mitigating buffer overflows by operating system randomization (2002)Google Scholar
  4. 4.
    Cohen, F.B.: Operating system protection through program evolution. Comput. Secur. 12(6), 565–584 (1993)CrossRefGoogle Scholar
  5. 5.
    Collberg, C., Thomborson, C., Low, D.: A taxonomy of obfuscation tranformations. Technical report 148, The University of Auckland (1997)Google Scholar
  6. 6.
    TIS Committee: Tool Interface Standard. Executable and Linking Format (ELF) Specification. Version 1.2. Submitted to Journal of Information Security and Applications (Elsevier), under evaluation (1995)Google Scholar
  7. 7.
    Cooper, K.D., Harvey, T.J., Waterman, T.: Building a control-flow graph from scheduled assembly code. Technical report 02–399, Rice University (2002)Google Scholar
  8. 8.
    Falcarin, P., Carlo, S.D., Cabutto, A., Garazzino, N., Barberis, D.: Exploiting code mobility for dynamic binary obfuscation. In 2011 World Congress on Internet Security (WorldCIS), pp. 114–120, February 2011Google Scholar
  9. 9.
    Jang, M.H., Jang, M.: Security Strategies in Linux Platforms and Applications. Jones & Bartlett Publishers, Burlington (2010)Google Scholar
  10. 10.
    Jiang, X., Wang, H.J., Xu, D., Wang, Y.-M.: Randsys: thwarting code injection attacks with system service interface randomization. In: IEEE International Symposium on Reliable Distributed Systems, SRDS 2007, pp. 209–218 (2007)Google Scholar
  11. 11.
    Kerrisk, M.: The Linux Programming Interface. No Starch Press, San Francisco (2010)Google Scholar
  12. 12.
    Kinder, J., Zuleger, F., Veith, H.: An abstract interpretation-based framework for control flow reconstruction from binaries. In: Jones, N.D., Müller-Olm, M. (eds.) VMCAI 2009. LNCS, vol. 5403, pp. 214–228. Springer, Heidelberg (2009) CrossRefGoogle Scholar
  13. 13.
    Liang, Z., Liang, B., Li, L.: A system call randomization based method for countering code injection attacks. In: International Conference on Networks Security, Wireless Communications and Trusted Computing, NSWCTC 2009, pp. 584–587 (2009)Google Scholar
  14. 14.
    Linn, C., Debray, S.: Obfuscation of executable code to improve resistance to static disassembly. In: Proceedings of the 10th ACM Conference on Computer and Communications Security, CCS 2003, pp. 290–299. ACM, New York, USA (2003)Google Scholar
  15. 15.
    Madou, M., Anckaert, B., De Bus, B., De Bosschere, K., Cappaert, J., Preneel, B.: On the effectiveness of source code transformations for binary obfuscation. In: Proceedings of the International Conference on Software Engineering Research and Practice (SERP06), pp. 527–533. CSREA Press (2006)Google Scholar
  16. 16.
    Popov, I.V., Debray, S.K., Andrews, G.R.: Binary obfuscation using signals. In: USENIX Security (2007)Google Scholar
  17. 17.
    S. Rauti, J. Holvitie, and V. Leppänen. Towards a Diversification Framework for Operating System Protection. In: Proceedings of International Conference on Computer Systems and Technologies, CompSysTech 2014 (2014)Google Scholar
  18. 18.
    Rauti, S., Leppänen, V.: Browser extension-based man-in-the-browser attacks against Ajax applications with countermeasures. In: Proceedings of International Conference on Computer Systems and Technologies, CompSysTech 2012, pp. 251–258. ACM Press (2012)Google Scholar
  19. 19.
    Rauti, S., Leppänen, V.: A proxy-like obfuscator for web application protection. Int. J. Inf. Technol. Secur. 5(1) (2014)Google Scholar
  20. 20.
    Lee, J.W., Lee, Y.J., Kim, H.K., Hwang, B., Ryu, K.H.: Discovering temporal relation rules mining from interval data. In: Shafazand, H., Tjoa, A.M. (eds.) EurAsia-ICT 2002. LNCS, vol. 2510, pp. 57–66. Springer, Heidelberg (2002) CrossRefGoogle Scholar
  21. 21.
    Rauti, S., Leppänen, V.: Resilient code protection by JavaScript and HTML obfuscation for Ajax applications against man-in-the-browser attacks. Submitted to Journal of Information Security and Applications (Elsevier), under evaluation (2014)Google Scholar
  22. 22.
    Schwarz, B., Debray, S., Andrews, G.: Disassembly of executable code revisited. In: Proceedings of Ninth Working Conference on Reverse Engineering, pp. 45–54 (2002)Google Scholar
  23. 23.
    Sobell, M.G.: A Practical Guide to Linux. Addison-Wesley, Boston (1999)Google Scholar
  24. 24.
    Srivastava, A., Lanzi, A., Giffin, J., Balzarotti, D.: Operating system interface obfuscation and the revealing of hidden operations. In: Holz, T., Bos, H. (eds.) DIMVA 2011. LNCS, vol. 6739, pp. 214–233. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  25. 25.
    Tanenbaum, A.S.: Modern Operating Systems, 3rd edn. Prentice Hall Press, Upper Saddle River (2007) Google Scholar
  26. 26.
    Theiling, H.: Extracting safe and precise control flow from binaries. In: Proceedings of Seventh International Conference on Real-Time Computing Systems and Applications, pp. 23–30. IEEE (2000)Google Scholar
  27. 27.
    Wang, S.P.: Mastering Linux. CRC Press, Boca Raton (2011) Google Scholar
  28. 28.
    Wu, Z., Gianvecchio, S., Xie, M., Wang, H.: Mimimorphism: a new approach to binary code obfuscation. In: Proceedings of the 17th ACM Conference on Computer and Communications Security, CCS 2010, pp. 536–546. ACM, New York, USA (2010)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Sampsa Rauti
    • 1
    Email author
  • Samuel Laurén
    • 1
  • Shohreh Hosseinzadeh
    • 1
  • Jari-Matti Mäkelä
    • 1
  • Sami Hyrynsalmi
    • 1
  • Ville Leppänen
    • 1
  1. 1.University of TurkuTurkuFinland

Personalised recommendations