Measuring Application Security

Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 782)


We report on a qualitative study of application security (AppSec) program management. We sought to establish the boundaries used to define program scope, the goals of AppSec practitioners, and the metrics and tools used to measure performance. We find that the overarching goal of AppSec groups is to ensure the security of software systems; this is a process of risk management. AppSec boundaries varied, but almost always excluded infrastructure-level system components. Seven top-level questions guide practitioner efforts; those receiving the most attention are Where are the application vulnerabilities in my software?, Where are my blind spots?, How do I communicate & demonstrate AppSec’s value to my management?, and Are we getting better at building in security over time?. Many metrics are used to successfully answer these questions, but one challenge stood out: there is no good way to measure AppSec risk. No one metric system dominated observed usage.


Application security Security management Security metrics Program management Risk management 



This work was made possible by Secure Decisions, the Department of Homeland Security, and AppSec practitioners, many introduced to us by Code Dx, Inc. We would sincerely like to thank these practitioners for their time and candidness during the interviews; this work would have not been possible without their participation.

This material is based on research sponsored by the Department of Homeland Security (DHS) Science and Technology Directorate, Cyber Security Division (DHS S&T/CSD) via contract number HHSP233201600058C. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Department of Homeland Security.


  1. 1.
    Davis, N.: Developing Secure Software. New York’s Software & Systems Process Improvement Network. New York, NY (2004)Google Scholar
  2. 2.
    2016 Data Breach Investigations Report. Verizon EnterpriseGoogle Scholar
  3. 3.
    CISO AppSec Guide: Application Security Program – OWASP.
  4. 4.
    About the Building Security In Maturity Model.
  5. 5.
  6. 6.
    Framework for Improving Critical Infrastructure Cybersecurity (2014).
  7. 7.
    McGraw, G., Migues, S., West, J.: BSIMM8 (2017).
  8. 8.
  9. 9.
    Sanders, B.: Security metrics: state of the Art and challenges. Inf. Trust Inst. Univ. Ill. (2009)Google Scholar
  10. 10.
    Miles, J., Gilbert, P.: A Handbook of Research Methods for Clinical and Health Psychology. Oxford University Press, Oxford (2005)CrossRefGoogle Scholar
  11. 11.
  12. 12.
    Steiner, G.A.: Strategic Planning. Simon and Schuster, New York (2010)Google Scholar
  13. 13.
  14. 14.
    Douglas, M.: Strategy and tactics are treated like champagne and two-buck-chuck (2015).
  15. 15.
    Marrinan, J.: What’s the difference between a goal, objective, strategy, and tactic? (2014).
  16. 16.
    Basili, V.R., Caldiera, G., Rombach, H.D.: The goal question metric approach. Encycl. Softw. Eng. 2, 528–532 (1994)Google Scholar
  17. 17.
    ISO/IEC/IEEE 24765:2010(E) Systems and software engineering — Vocabulary (2010).
  18. 18.
  19. 19.
    Young, B.: Measuring software security: defining security metrics (2015)Google Scholar
  20. 20.
    Crouhy, M., Galai, D., Mark, R.: The Essentials of Risk Management. McGraw-Hill, New York (2005)Google Scholar
  21. 21.
    Krebs, B.: What’s your security maturity level? (2015).
  22. 22.
    Richardson, J., Bartol, N., Moss, M.: ISO/IEC 21827 Systems Security Engineering Capability Maturity Model (SSE-CMM) a process driven framework for assuranceGoogle Scholar
  23. 23.
    Acohido, B., Sager, T.: Improving detection, prevention and response with security maturity modeling (2015).
  24. 24.
    Microsoft Security Development Lifecycle.
  25. 25.
  26. 26.
    SP 800-53 Rev. 5 (DRAFT), Security and Privacy Controls for Information Systems and Organizations.
  27. 27.
    Wichers, D.: Getting started with OWASP: the top 10, ASVS, and the guides. In: 13th Semi-Annual Software Assurance Forum, Gaithersburg, MD (2010)Google Scholar
  28. 28.
  29. 29.
    Jansen, W.: Directions in security metrics research (2009).
  30. 30.
    Savola, R.: On the feasibility of utilizing security metrics in software-intensive systems. Int. J. Comput. Sci. Netw. Secur. 10, 230–239 (2010)Google Scholar
  31. 31.
    Hallgren, K.A.: Computing inter-rater reliability for observational data: an overview and tutorial. Tutor. Quant. Methods Psychol. 8, 23–34 (2012)CrossRefGoogle Scholar
  32. 32.
  33. 33.
    Gaillard, J.C.: Cyber security: board of directors need to ask the real questions (2015).
  34. 34.
  35. 35.
    Gordon, L.A., Loeb, M.P.: The economics of information security investment. ACM Trans. Inf. Syst. Secur. TISSEC. 5, 438–457 (2002)CrossRefGoogle Scholar
  36. 36.
  37. 37.
    Hoo, K.J.S.: How much is enough? A risk management approach to computer security. Stanford University Stanford, California (2000)Google Scholar
  38. 38.
    Schryen, G.: A fuzzy model for IT security investments (2010)Google Scholar
  39. 39.
    Böhme, R.: Security metrics and security investment models. In: IWSEC, pp. 10–24. Springer (2010)CrossRefGoogle Scholar
  40. 40.
    Held, G.: Measuring end-to-end security. AppSecUSA 2017, Orlando, FL (2017)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Secure DecisionsClifton ParkUSA
  2. 2.Code DxNorthportUSA

Personalised recommendations