A Structured Comparison of the Corporate Information Security Maturity Level

Abstract

Generally, measuring the information security maturity is the first step to build a knowledge information security management system in an organization. Unfortunately, it is not possible to measure information security directly. Thus, in order to get an estimate, one has to find reliable measurements. One way to assess information security is by applying a maturity model and assess the level of controls. This does not need to be equivalent to the level of security. Nevertheless, evaluating the level of information security maturity in companies has been a major challenge for years. Although many studies have been conducted to address these challenges, there is still a lack of research to properly analyze these assessments. The primary objective of this study is to show how to use the analytic hierarchy process (AHP) to compare the information security controls’ level of maturity within an industry in order to rank different companies. To validate the approach of this study, we used real information security data from a large international media and technology company.

1 Introduction

Information security can only be measured indirectly [6]; unfortunately there is still no gold standard. One way to indirectly measure it is to use metrics and KPIs [1] which aim to approximate the real status of information security. This approach is not always reliable [22]. Some information to build those metrics are obtained from technical systems (e.g. firewalls, intrusion detection/prevention systems, security appliances). However, most of these metrics and KPIs have to be quantified by humans and are therefore prone to errors.

This can lead to possible inaccuracies, measurement errors, misinterpretations, etc. [4]. If these metrics are then compared across the board, the information security managers face a major challenge. As a consequence, this could lead to bad decisions based on wrong conclusions. Moreover, by just comparing the metrics, without any weighting the specifics of the respective industry are not considered. Thus, a prioritisation within the comparison is not possible [3]. This problem is reinforced when the comparison of information security metrics between different companies or departments would take place [9], which is exactly on of the current challenges enterprises face today: How to compare their (sub-)companies of a specific industry (e.g. eCommerce) in terms of information security.

The main goal of this paper is to compare the effect of multiple factors in the information security assessment process. Aiming at achieving this goal, the analytic hierarchy process (AHP) is applied. The Analytical Hierarchy Process (AHP) is one of the most commonly used Multiple Criteria Decision Methods (MCDM), combining subjective and personal preferences in the information security assessment process [20]. It allows a structured comparison of the information security maturity level of companies with respect to an industry [25] and to obtain a ranking [13]. This allows us to define a separate weighting of information security metrics for each industry with to respect their specifics while using a standardized approach based on the maturity levels of the ISO 27001:2013 controls [12]. ISO 27001 was in particular selected, because this standard is shown to be mature, widespread and globally recognized. This minimizes the additional effort for collecting the required metrics. In this study, the maturity level is based on a hierarchical, multi-level model to analyze the information security gap for the ISO 27001:2013 security standard [20]. As a prerequisite for the comparison, we assume companies have implemented an information security management system (ISMS) in accordance with ISO 27001 [26].

To validate the approach of this study, we used real information security data (i.e. security controls’ maturity level) from Hubert Burda Media (HBM) a large international media and technology company consisting of over 200 individual companies. This provides sufficient data with a high degree of detail in the area of information security. The result from our AHP-based approach is then compared with the perceived status of information security by experts.

The remainder of this work is structured as follows: In Sect. 2 we give a brief overview of related work. Section 3 describes our methodology when we developed our approach shown in Sect. 4. Our results are shown in Sect. 5 followed by a discussion and our conclusion in Sect. 6, respectively Sect. 7.

2 Background and Related Work

In addition to the differences in the assessment of information security, all assessment procedures have in common that the ratings of the maturity level and the weighting of weights remain separate judgements and are not allocated to a common overall value in the sense of an ‘information security score’. It is therefore up to the evaluator to carry out the respective evaluation, as he or she is forced to choose between these two quantitative aspects of the evaluation, i.e. the ratings on the one hand and the weighting on the other [15]. In contrast to this, the works of Boehme [6] and Anderson [3] deal more with the economic impact of investments in information security. The focus of this work is to compare the degree of maturity within an industry. This could later lead to a monetary assessment of information security or maturity.

A solution which involves to merging ratings and weights and thus integrates different assessment measures at the same time offers multi-attribute decision-making procedures [8]. These are methods that offer support in complex decision-making situations, i.e. when a decision has to be made in favour of one of several options against the background of several decision criteria (so-called attributes).

The prerequisite for using the multi attribute decision procedure is, as described above, the determination of weights. A popular method of doing this is the Analytic Hierarchy Process (AHP) method developed by Saaty [23]. Nasser [2] describes how to measure the degree of maturity using AHP. In contrast to our paper which deals with the comparison of the maturity level within an industry, Nasser [20] focuses on the determination of inaccurate expert comparison judgement in the application of AHP.

Some recent works deals with this problem setting using the AHP but there exist further restrictions. Watkins [27] uses for his approach not the control maturity level and is only valid in the cyber security environment. Bodins’ [5] approach is based on the comparison of the CIA-Triangle and not on ISO 27001-controls. Peters [21] has already shown the application of AHP in the domain of project management but did not use real data to validate the approach.

2.1 Multiple Criteria Decision Methods

Multi criteria decision problems which could be solved with a multiple-criteria decision analysis method (MCDM) are a class of procedures for the analysis of decision or action possibilities characterized by the fact that they do not use a single superordinate criterion, but a multitude of different criteria. Problems in evaluating multiple criteria consist of a limited number of alternatives that are explicitly known at the beginning of the solution process. For multiple criteria, design problems (multiple objective mathematical programming problems), the alternatives are not explicitly known. An alternative (solution) can be found by solving a mathematical model. However, both types of problems are considered as a kind of subclass of multi-criteria decision problems [17]. MCDM helps to determine the best solution from multiple alternatives, which may be in conflict with each other. There are several methodologies for MCDM such as: Analytical hierarchical process (AHP), Grey relational analysis (GRA), Technique for order preference by similarity to ideal solution (TOPSIS), Superiority and inferiority ranking (SIR), Simple additive weighting (SAW), and Operational competitiveness rating (OCRA) [7].

2.2 The Analytical Hierarchy Process

The AHP, is a method developed by the mathematician Saaty [24] to support decision-making processes. Because of its ability to comprehensively analyse a problem constellation in all its dependencies, the AHP is called ‘analytical’. It is called a ‘process’ because it specifies how decisions are structured and analysed. In principle, this procedure is always the same, which makes the AHP an easy-to-use decision tool that can be used more than once and is similar to a routine treatment [16]. The goal of the Analytic Hierarchy Process method is to structure and simplify complex decision problems by means of a hierarchical analysis process in order to make a rational decision. The AHP breaks down a complex evaluation problem into manageable sub-problems.

3 Research Methodology

Many companies use the maturity level measurement of the controls from ISO standard 27001 to obtain a valid and reliable metric. The ISO standard is well established and the maturity assessment of the standard’s controls is an adequate possibility to create a picture of the information security processes of a company. While this might be sufficient for a continuous improvement within the same company, a problem arises if one wants to compare the information security processes of different companies or departments. Depending on the field of industry, some of the processes might be more important than others.

The general aim of this approach is to determine which company within an industry is better or worse in a (sub)area of information security, in order to create transparency among the companies within an industry concerning information security. Positive effects of this approach would be the improvement or deterioration of the information security in a sector within an industry recognizable up to the question where the management should invest money economically for information security in order to improve a sector.

We define the requirements in the next subsection, then determine the proper algorithm and finally describe the data collection for our approach.

3.1 Requirements

The most important requirement is that the metrics we rely on should be easy to gather. Assuming that the investigated company is running an information security management system (ISMS), a natural approach is to rely on the controls of the ISO/IEC 27001 standard and their maturity level. Existing data (e.g. information security maturity level) should be used wherever possible. Furthermore, the approach should consider the environment of the industry in which the company is located. Additionally, the information gathering should be repeatable and stable. Comparing and evaluating over a long period should be possible as well as an overall as an comparison of security levels of business units or companies in a similar area. Finally, the approach should allow it to visualize and explain the results of the comparison and allow to derive the areas where companies could improve.

3.2 Algorithm Selection

Taking all requirements into account, our problem is a multi-dimensional decision problem, and thus can be addressed by a multiple-criteria decision analysis method (MCDM). Our comparison criteria (dimensions) are the ISO/IEC 27001 controls and we compare the different companies based on their corresponding maturity levels for each control. Thus, the MCDM needs discrete, quantitative input and a criteria weighting method. Since the underlying controls are hierarchically and therefore very structured, the chosen method/model should reflect that also.

This leads us to the analytical hierarchy process (AHP) as a best fit method in the above described context. The AHP is a mature structured technique for organizing and analyzing complex decisions, combining subjective and personal preferences. The AHP has been the most widely used technique of multi-criteria decision making during the last twenty five years [19]. The advantage of this method over the utility value analysis, for example, is that it goes beyond the evaluation of ideas and generates a clear selection recommendation. Its hierarchical structuring of decision making fits well to the ISO/IEC 27001 controls’ hierarchy and the qualitative evaluation part of the AHP is very much in line with the maturity level for information security. Since the AHP compares the maturity level for each control company-wise, it naturally allows to understand where each company’s security level is ranking related to each control. Additionally, the weight of each criteria (control) can be easily derived. In the concrete application case it is possible to compare the importance of individual controls of ISO 27001 very granularly with each other (pairwise). This is in particular necessary in order to be able to establish an industry reference. Furthermore, the AHP enables precise calculations of weights, in this case the information security maturity ratings of companies in a specific sector.

Thus, we used a paired comparison questionnaire based on the AHP to compare controls and their maturity level for an industry.

3.3 Data Collection

To test the above approach it is necessary to set up the model and verify it with real data. We need a maturity assessment of the ISO/IEC controls and to weight them according to the considered industry. We focused on the eCommerce industry for the following reasons:

• Available data from a large range of companies

• Excellent data quality and validity

• High actuality of the existing data

• Very good know-how available in the expert assessment of the industry.

Maturity Assessment of ISO/IEC 27001 Controls. We collected data from Hubert Burda Media (HBM), an international media and technology company (over 10,000 employees, more than 2 billion annual sales, represented in over 20 countries). This group is divided into several business units that serve various business areas (including print magazines, online portals, e-commerce, etc.). The business units consists of over 200 individual companies with about 30 of them being in the eCommerce industry. Each subsidiary operates independently of the parent corporation. There is a profit center structure, so the group acts as a company for entrepreneurs and the managing directors have the freedom to invest money into information security or choose the appropriated level of security.

We will briefly describe how this data is collected before going into more detail on the data used for the comparison. Each individual company in the group operates its own Information Security Management System (ISMS) in accordance with ISO/IEC 27001:2013, which is managed by an Information Security Officer (ISO) on site and managed by a central unit in the holding company. As part of the evaluation of the ISMS, the maturity level for the respective ISO 27001 controls is ascertained - very granularly at the asset level. The maturity level is collected/updated regular once a year as part of a follow-up.

First, the information values of the respective company (e. g. source code, customer data, payment data, etc.) are determined according to the protection goals of confidentiality, integrity and availability and assigned to a technical system (e. g. application, client, server, etc.).

Second, these technical systems undergo a threat analysis¹ of the assets in relation to the respective asset type as part of information security risk management. The threat analysis is classically evaluated with regard impact² and the probability of occurrence. This results in an aggregated risk value (1–5) for each asset after a pre-defined settlement. This risk value is later transferred to the control valuation as the target maturity level. In this way, a comparison is made between the protection requirements of the information values and the protection level of the respective (IT) system.

Third, the control evaluation is then carried out using the Cobit maturity level. The controls are dynamically selected³ according to the previously evaluated threats. The Cobit maturity level is a 6-step evaluation scale (0–5) with which a continuous improvement can be measured and a potential improvement can be identified. This allows it to evaluate the actual maturity level per control and asset. The assessment of the current status of the controls is carried out by the information security officer of the respective company. The collected data is therefore not technical data but subjectively quantified data with a possible bias. Although, the evaluated data is reviewed by further experts, a complete review cannot be carried out due to resource limits. The target maturity level is already determined by the risk value/protection level of the system. This provides a clear picture of the ISMS status at a very granular asset level.

Fourth, the picture is completed by the Cobit maturity analysis of the IT-/ISM processes⁴. For each of these processes, the controls (e. g. A.16 for incident management) are evaluated with an actual maturity level [10]. In the later evaluation (typically by means of a spider graphic) the complete ISO 27001 standard is evaluated with the aid of the Cobit degree of maturity [14].

The available data is very granular on asset level (application, client, server, etc.). However, although the companies are from the same industry, they do not necessarily have the same kind of assets. Thus, we decided to abstract from the assets and to aggregate the data at company level. To do this automatically, we used the mean value of all evaluated assets per control. For the following proof of concept, we only show data from 5 companies.

4 The Approach - the AHP-Implementation

In this section, we discuss how the AHP is applied to our comparison. The first step of the AHP, to model the problem as a decision hierarchy, we have already done by deciding that our decision-criteria will be the ISO/IEC 27001 controls. The goal is clearly defined: to find the subsidiary within the company with the best information security/level of maturity within an industry. Appendix A of ISO 27001 helps us to select criteria and sub criteria, which is divided into 14 Control Categories, 35 Control Objectives and 114 Controls (see Fig. 1).

The next step is the prioritization of all criteria and sub criteria (Sect. 4.1). This represents the domain specific part of the AHP calculations and it only needs to be done once per domain. It is followed by the evaluation of the alternatives (Sect. 4.2). The alternatives represent the agile part of the calculation. We describe in the corresponding section, how the evaluation can be directly derived from the maturity level of a company’s control. Based on the individual evaluations and prioritizations of controls, the AHP uses a mathematical model to determine a precise weighting of all alternatives in relation to the respective criteria and assembles them in a percentage order (Sect. 4.3).

In the next subsections we describe in detail how the AHP was used and show how the applied AHP model was implemented in a statistical software (in this case in R).

Fig. 1.

Exemplary ISO 27001 Appendix A structure

4.1 Pairwise Comparison of the Control Categories and Controls

The characteristics of an industry have a significant influence on the pairwise comparison when comparing the individual controls. If the information security of companies is to be compared with each other, e.g. in the e-commerce sector, it will differ significantly from that of companies in other sectors, e.g. publishing or the manufacturing industry. On the one hand this is due to the different business models within the industries, because the IT strategy and the information security strategy are derived from the business strategy. On the other hand this is due to the different focus in information security. For example, the eCommerce industry is very focused on application development and (confidentiality) protection of customer data, whereas the highest commodity to be protected in the manufacturing industry is the availability of systems.

The decision-maker must compare each criterion with its pair and denotes which of the two criteria appears more important to him/her. This method of pairwise comparisons allows the decision-maker to elicit a very precise evaluation from the multitude of competing criteria. The comparisons must be carried out specifically for one industry (e. g. eCommerce). In the case of our hierarchy based on the ISO/IEC 27001 controls, 91 pairwise comparisons have to be made for the control categories and 208 for the controls, respectively. This leads to a ranking order in which the criteria are ranked according to their importance.

Table 1. AHP scores and their application

The comparison is done as follows: Each result of a pairwise comparison of two criteria entered in the evaluation matrix shows how much more significant a criteria is in relation to the criteria of the level above. To do this, refer to the scale in Table 1a. In order to make a comparison for one criteria, i.e. the control categories, we compare the individual control categories with each other. The authors made this comparison in a straight forward Excel spreadsheet. The assessment of the relative importance of the criteria at the criterion level can be found in Table 1b. These pairwise comparisons are always carried out by an expert with the background knowledge and with reference to the industry (here eCommerce). The comparison for the sub criteria, the controls, follows the same guidelines.

4.2 Pairwise Evaluation of the Controls’ Maturity Levels

The alternatives in our example are the information security maturity of 5 eCommerce companies of HBM. For each control and each company there is a corresponding maturity level based on the Cobit Maturity Model. 0 represents the worst and 5 the best result, always in relation to the evaluation of a control. As already discussed in Sect. 3.3, the maturity levels for each company were based on assets and we aggregated the maturity levels by calculating the average maturity level for each control over all evaluated assets of the respective company.

For the pairwise comparison, the gap between the comparative maturity levels of two companies’ controls is considered to decide which company is doing better at a specific control. For that purpose, we need to map the 6-stage scale of the Cobit maturity grade gaps (see Table 2) to the 9-stage AHP score. The result is a table where each GAP Cobit interval represents an AHP score, which is verbally described. An exemplary calculation can be found in Table 2c). Alternative A (Company 1) is compared with the alternatives B (Company 2 to 5). A Cobit GAP -2 (i.g. 1–3) means hat Company 2 is 2 control maturity better than Company 1, the AHP score is, corresponding to the Cobit GAP interval, 4, respectively 1/4. This can be used to calculate which of the 5 companies performs best in Control A.5.1.1.

Table 2. Combined GAP of Cobit Maturity Model and AHP score

The step of comparing the companies’ maturity levels for each control represents the business unit specific part of the analysis. Note that, due to our mapping of the GAP Cobit interval and the AHP score, this can be done fully automatic if the corresponding maturity levels are provided. The pairwise comparison, the calculation of the difference and the ‘translation’ to the GAP intervals is done in the statistics software R.

4.3 Calculation of the Comparison

As mentioned above, the actual calculation of the AHP is done with R. The implementation in R worked with the help of a YAML (Ain’t Markup Language) script executed in R. The YAML script is a simplified markup language for data serialization. The YAML script contains all results of the pairwise comparison of criteria and sub criteria, as well as the maturity levels of the 114 controls of the 5 eCommerce companies. The decision hierarchy built up in the YAML script corresponds to the ISO standard. The decision hierarchy is then enriched with alternatives. The paired comparison of the alternatives is executed by a function of the R-package ‘ahp’ (version 0.2.12 from Christoph Glur) at script runtime for a simple data processing flow. The runtime of the script (with data from 5 companies) on an iMac (3.2 GHz Intel Core i5) was less than 10 s, indicating that it is efficient enough to handle large amounts of data easily.

5 Results of the Comparison

The AHP was used to compare the maturity level in order to find the company with the best information security within an industry (here eCommerce).

Prioritization of Controls. Here we show which priority the control categories (criteria) and controls (sub criteria) have in relation to the complete Appendix A of ISO 27001 over all. The pairwise comparison for the eCommerce industry shows that the controls of the control category ‘A.14’ have the highest priority (17.6%), followed by ‘A.17’ (14.7%) and ‘A.12’ (10.1%). Within control category ‘A.14’, controls ‘A.14.2.8’ (22.6%), ‘A.14.2.7’ (15.2%) and ‘A.14.2.6’ (11.8%) are the most important as shown in Fig. 2.

Fig. 2.

Top3 control categories prioritized and companies ranked

Comparision of the Companies. The Control Category ‘A.14’ was used to exemplarily show the evaluation. Figure 2 also shows how the individual eCommerce companies weighting compare with each other in the control category ‘A.14’ in detail. Overall (cf. Fig. 3), Company3 (21.0%), Company5 (20.9%) and Company1 (20.5%) came out best in a direct comparison. The differences are marginal and only on closer inspection are there more pronounced differences observed at the control level. In relation to a control category e.g. of ‘A.14’, the maturity of Company1 (4.4%) and Company4 (4.0%) is better in detail, but considering the control category ‘A.17’, Company3 (5.2%) is clearly ahead of Company4 (1.7%).

Fig. 3.

Control category A.14 weight contribution and ranked companies

6 Discussion

Based on these results, we discuss the main findings as follows. The results show that with the pairwise comparison it is possible to obtain a priority for each individual control, and thus very granular, in the overall context of ISO/IEC 27001 for the eCommerce industry. The priorities of the larger control categories are also very helpful, as a quick comparison of priorities is possible here. The approach with the pairwise comparison by AHP meets all requirements of the methodology part. Similarly, it is shown that the weighting of the pairwise comparisons of the maturity level of eCommerce companies can be mapped very granularly to the controls of the ISO/IEC 27001 standard. It was also possible to derive the AHP score from the maturity levels automatically. This makes it easy to compare the rankings of the companies. The only effort which needs to be invested (for each industry) is the prioritization of the controls.

The results suggest that the approach works in conjunction with real data (the maturity levels of HBM’s eCommerce companies) at least for the chosen area. The results of the comparison also withstand the reality that one of the authors observes in his daily professional life. The results also showed that the ranking results reflect the reality of at least the HBM eCommerce companies. However, it can be strongly assumed that the method is directly applicable to other companies with the same or similar results.

6.1 Limitations

For reasons of simplification and clarity, we have demonstrated the approach only with a small number of companies. But is easily possible to run the approach with the full set of HBM’s companies and to extend it to other business units by readjusting the ISO/IEC 27001 controls’ priorities.

The application of the AHP methodology is not undisputed in technical literature. At this point the authors consider some points of this criticism. On the one hand, these are points concerning the mathematical part of the AHP and on the other hand, the criticism is based on the procedure. In the model calculated above, the pairwise comparison of the criteria and sub criteria has been carried out by one person (with expert knowledge), which can be regarded as a very subjective survey of all pair comparisons. This assumes that there are high demands on the respondent due to the many pair comparisons, which is why there are often problems with validity [18]. This could lead to a limitation of the size of the decision model and is seen as a critical and possible optimization point of the AHP methodology in literature and practice [11].

If you take a closer look at the origin of the maturity level, you immediately notice that it is determined by the information security officer’s self-disclosure. As with all quantification, the human factor, a lack ob objectivity or bias, cannot be excluded here. However, it can be largely validated by a team of experts. Another point concerns the type of data collection, the resulting prevailing data quality and possible imponderables in data evaluation. These issues could only be reduced but not completely eliminated by several iterations of quality assurance.

In the next chapter, some of the limitations will be discussed and further improvements of the methodology/model will be proposed.

7 Conclusion and Future Work

The results of the pairwise comparison suggest that AHP is very well suited to compare the information security maturity of different companies and to find the company with the best information security within an industry.

It has been proven that a comparison within the eCommerce industry is possible using this model and thus ranking the prioritization of control categories and, above all, the individual controls can follow. The AHP provides in this case a robust and comprehensive treatment for decision makers in both qualitative and quantitative ways as found in this study and it can be assumed that this will also work for other companies in the same environment. The real insight is to adapt the AHP or the data so that it works together. The AHP-model has shown how AHP might be used to assist decision maker evaluate information security in one branch. Very interesting, and also for validation, would be the pairwise comparison for other industries such as publishing houses, manufacturing industry. Companies with very different degrees of maturity could also be interesting here.

Some of the limitations mentioned above regarding the AHP methodology deal with the comparison of pairs. A possible improvement of the model would be to compare it with the help of a team of experts from the eCommerce industry. This would have the advantage that the pair comparison is subject to validation.

In future work, the focus will be on the details of implementing this model across a variety of different examples, as well as working on more expanded decision hierarchy with an additional level of sub criteria (control objectives). In addition, it would be interesting to calculate the approach with different aggregated data (min, max, median) in addition to the mean value and to observe the effects. Furthermore, it would be interesting to apply the AHP methodology to other industries (e. g. publishing, manufacturing industry etc.). Ultimately, this would provide the prerequisites for comparing information security across industries, comparing apples and pears, so to speak.