Keywords

1 Introduction

The civil aviation industry plays a major role in global economic activity and development. As many other high-risk industries, safety is the most important prerequisite and foundation for the development of civil aviation. Despite several high-profile accidents, the year 2018 is one of the safest years according to the Aviation Safety Network data. There is a total of 18 fatal airliner (14+ passengers) hull-loss accidents excluding corporate jet and military transport accidents/hijackings, resulting in 561 fatalities in 2018. The statistics of number of fatal accidents and fatalities from 1990−2018 is showed in Fig. 1 below [1]. Though there is a decrease trend in the overall number of fatal accidents as well as fatalities, dealing with safety in civil aviation is a continuous and never-ending challenge because civil aviation industry is open, complex, and evolving.

Fig. 1.
figure 1

Number of fatal accidents and fatalities from 1990–2018 (Data from the Aviation Safety Network)

Full size image

Safety is typically managed through continuous risk management, including hazard identification, risk evaluation, risk mitigation and risk monitoring [2]. Control measures in different dimensions have been used to control technical risk, personnel risk, organizational risk, and so on. Industry has found that the traditional emphasis on personnel safety will not prevent the low probability and high consequence accidents [3].

The International Civil Aviation Organization (ICAO) revised Annexes 1, 6, 8, 11, 13, and 14 to refine the framework requirements for implementing Safety Management System (SMS) in 2008. With the wide promotion of safety management system (SMS) in civil aviation, more and more attention has been paid to safety performance management. Safety performance management has been recognized as playing a key role in achieving an acceptable level of safety in civil aviation organizations.

According to the ICAO Doc 9859 Safety Management Manual [4], safety performance is ‘A State or a service provider’s safety achievement as defined by its safety performance targets and safety performance indicators’. Safety performance indicator is ‘A data-based parameter used for monitoring and assessing safety performance’. ICAO describes indicators as two levels, the State level, which monitor its safety indicators, and the individual service provider that monitors safety performance indicators as part of its SMS.

Safety performance indicators play an important role in providing information on a civil aviation organization’s current safety status, motivating people to work on safety and increasing the efficacy of an organization’s safety improvement efforts. The establishment of a safety performance indicator system allows a civil aviation organization to assess whether it is implementing appropriate safety programs and policies, whether these programs and policies are achieving their desired objectives, and to assess the effectiveness of its SMS. On the other hand, the development of a safety performance indicator system is an important management responsibility that will prevent civil aviation accidents by identifying, measuring, analyzing, and adjusting key activities or indicators [5].

In recent years, there is increasing interest for researchers and industries to paid more and more attention to safety performance management in different domains. Enoma et al. [6] collected data from a series of interviews, workshops, the internet and other media to develop and test a set of key safety and security performance indicators for airport facility management. Lu et al. [7] evaluated airport safety risk based on modification of quantitative safety management model. In the proposed model, airport safety risk evaluation indicator system was established. Reiman et al. [8] presented a theoretical framework for utilizing safety performance indicators in safety-critical organizations. Kim et al. [9] built a structure of safety performance indicators to measure the safety performances of airline. In the safety performance assessment proposed by the Safety Management International Collaboration Group (SMICG) [10], safety performance indicators are divided into three tiers, where tier 1 measure the outcomes of the whole civil aviation system, tier 2 indicators depict safety management performance of operators and tier 3 address the activities of the regulator. McDonald et al. [11] introduced the overall Proactive Safety Performance for Operations project concept and high level system requirements as they emerged from industry needs. Gander et al. [12] provided the first comprehensive description of how sleep/wake history and circadian phase influence measures of pilot fatigue pre-flight and at the TOD on long trans-meridian flights by combining data sets from four airline studies using the same measures. Chang et al. [13] used a two-stage process to evaluate the performance of the SMS operations at Taiwan’s three international airports by using the Analytic Network Process and the fuzzy Technique of Ordering Preference by Similarity to Ideal Solution. Chen et al. [14] proposed a specific method based on the analytic hierarchy process and Delphi method to measure the safety management of civil aviation unit with a series of safety performance indicators. Panagopoulos et al. [15] introduced an integrated, conceptual framework for measuring and improving aviation system’s safety performance by adapting and combing classical Quality Management tools, a leading indicators program and Lean-six Sigma methodology. Thekdi et al. [16] presented and discussed an enhanced framework for unification of performance management and risk management principles based on analysis of combinations of different risk management and performance management practices/polices. Rong et al. [17] established an airport operational risk warning model based on determining ten key modules affecting the airport operational safety and designing 165 airport operational risk monitoring indicators. MacLean et al. [18] developed a benchmark from the aggregation of failure data on a set of comparable airports to compare airports’ performance with the benchmark using the defined safety measures. Sun et al. [19] established a safety performance index system based on Reason’s model and a safety performance evaluation model to compare fleet safety performance from five dimensions quantitatively. Gerede et al. [20] adopted the Delphi Method to produce a list of safety performance indicators used to measure safety in flight training organizations in Turkey. Sun et al. [21] established a comprehensive evaluation model based on the subjective and objective combination weight method for quantitatively assess the safety state of the maintenance department. Chen et al. [22, 23] proposed comprehensive risk evaluation models to evaluate civil airport’s operation risk.

The remaining of this paper is organized as follows. The framework to assess safety performance, safety performance indicator life cycle, weight assignment for different types of safety performance indicators, and the comprehensive risk evaluation model are presented in Sect. 2. Then, Sect. 3 introduces the safety performance assessment tool. In Sect. 4, a case study is given to verify the proposed model. Finally, conclusions are provided in Sect. 5.

2 A Framework for Assessing Safety Performance

A systematic framework is presented to quantitatively assess the safety performance and operation risk for civil aviation organization as shown in Fig. 2. In this framework, four main assessment steps are given. In the first step, safety performance indicators and corresponding metrics of different departments/posts of a civil aviation organization are defined. Second, the severity of the event, deviation, management represented by each safety performance indicator is different. In this step, weights for different types of safety performance indicator will be assigned. Third, a comprehensive assessment model will be established based on safety performance indicators and corresponding weights. The last step is verifying the proposed model by case study. Safety performance indicators, metrics, weights, and the assessment model would be adjusted if the verification results are inconsistent with the actual operation of civil aviation organization.

Fig. 2.
figure 2

Steps for the safety performance assessment framework

Full size image

2.1 Safety Performance Indicator System

The Organization for Economic Cooperation and Development (OECD) proposed a step-by-step approach to create and implement the safety performance indicators programs [24, 25]. There are seven steps in the proposed approach from establishing a safety performance indicator team to evaluating and refining safety performance indicators. In this paper, the safety performance indicator system will be established according to a six steps process illustrated in Fig. 3.

Fig. 3.
figure 3

Six steps to define safety performance indicators

Full size image

The first step is to establish a safety performance management team. In the establishment process of safety performance indicator system, the involvement of management is critical to ensure resources are committed to the program. Appropriate team members should be involved in the safety performance management team, including an individual leader or leader group, technical experts and front-line employees. Additionally, employee participation is an excellent learning and motivating experience.

Once the safety performance management team are in place, the next step is to define safety objectives. Safety objectives are brief, high-level statements of safety achievements or desired outcomes to be accomplished. The safety performance management team should focus on what to monitor rather than how to monitor and avoid the pitfall of measuring what they can measure instead of what they should measure.

In the next step, the safety performance management team should define safety performance indicators and related metrics. Safety performance indicators should be related to the safety objective based on available data and reliable measurement. A metric, which defines the unit of measurement, should be assigned to each safety performance indicator. When establishing safety performance indicators, the safety performance management team should consider the following factors: measuring the right thing, availability of data, reliability of the data, and common industry safety performance indicators.

Once the safety performance indicators and corresponding metrics have been defined, the next step is to collect and analyze safety performance-related data. The data should be collected frequently enough to identify critical changes in a timely manner so actions can be taken to ensure safety.

Results from collection and analyzation safety performance data must be monitored to identify actions required. This information is important to ensuring quick follow-up action on adverse findings to fix problems in the associated policies, procedures, and operation processes.

The safety performance indicators and corresponding metrics should be periodically reviewed and assessed. These reviews help to ensure the safety performance indicators and corresponding metrics are well-defined and respond to safety objectives. It will identify specific safety performance indicators which are no longer needed and adjust safety performance indicators.

For key safety performance indicators, more does not mean better. The right number for organizations should depend on how many is manageable by organizations’ safety performance management team.

Signs of good safety performance indicators are as follows:

  • It is aligned directly with safety objectives;

  • It is quantifiable;

  • Data sources used to monitor and track are reliable and measurable; and

  • Few or no other organizations will have this exact same statistic.

Safety performance indicators can be categorized into four types: unsafe event type, foundation type, management type, and operation type. Different methods can be used to define these four types of safety performance indicators mentioned above. The methods are list in Table 1.

Table 1. Different methods for defining safety performance indicators
Full size table

It can be found that safety performance indicators can be defined through forward analysis process and backward analysis process from Table 1. The forward analysis process is a logical technique that starts from hazard identification and lays a path to unsafe activities or status, then lead to unsafe events. Safety performance indicators and corresponding metrics are defined based on the event chain. Many methods can be used in the forward analysis process, includes: system and job analysis, What-if analysis, event tree analysis (ETA), hazard and operability analysis (HAZOP), etc. The backward analysis process starts from unsafe events and lays a path to direct or indirect causes. Then, safety performance indicators and corresponding metrics are defined. Methods used in the backward analysis process include: Reason model, SHELL model, fault tree analysis (FTA), etc.

2.2 Weights for Different Types of Safety Performance Indicators

In order to assign weights for different types of safety performance indicators, Heinrich’s Law is used as baseline while taking into account ‘Civil aircraft incident standard’ and ‘Event sample’ published by Civil Aviation Administration of China (CAAC). According to Heinrich’s Law, there are about 29 minor injuries and 300 near misses and 1000 potential hazards behind every serious accident. As mentioned above, safety performance indicators can be categorized as four types. For the unsafe event type, it can be classified as accident, serious incident, general incident, and serious error. Based on Heinrich’s Law and in conjunction with experts’ evaluation, different weights for different types of safety performance indicators are listed in Table 2.

Table 2. Weights for different types of safety performance indicators
Full size table

2.3 Comprehensive Safety Performance Assessment Model

For a typical civil aviation organization, there are a number of departments with several posts. According to the hierarchy of a civil aviation organization, a multi-level safety performance assessment model is proposed. At the post level, safety performance of different posts can be calculated based on safety performance indicator system. At the department level, safety performance of different departments can be calculated based on posts’ safety performance. And for the organization level, safety performance can be assessed based on departments’ safety performance.

At the post level, safety performance indicators mainly include unsafe event type and operation type. For each safety performance indicator related to a certain post, its statistical value can be obtained every month. According to the evaluation standard of safety performance indicator, every safety performance indicator of a post will be evaluated as one of these five grades, namely ‘very poor’, ‘poor’, ‘average’, ‘good’ and ‘very good’. Correspondingly, values for these five grades can be assigned as 100,80,60,40, and 0 in the hundred-mark system.

Then, the safety performance of a certain post can be calculated as follow.

$$ R_{p} = R_{pc} + R_{pp} = \sum\nolimits_{i = 1}^{l} {w_{pci} \times S_{pci} + \sum\nolimits_{i = 1}^{r} {w_{pnci} \times S_{ppi} } } $$
(1)

In Eq. (1), \( R_{pc} \) is the assessment value of unsafe event type of safety performance indicators, \( R_{pp} \) is the assessment value of non-unsafe event types, \( w_{pci} \) is the weight of unsafe event type, \( w_{ppi} \) is the weight of non-unsafe event types, \( l \) and \( r \) are the number of unsafe event type and operation type of safety performance indicators, respectively.

At the department level, safety performance indicators mainly include unsafe event type, operation type, management type, and foundation type. Then, the safety performance of a certain department can be calculated as follow.

$$ R_{d} = R_{dc} + R_{dnc} = R_{dc} + R_{db} + R_{dm} + R_{dp} $$
$$ = \frac{1}{m}\sum\nolimits_{j = 1}^{m} {R_{pcj} + \sum\nolimits_{i = 1}^{s} {w_{dbi} } } \times S_{dbi} + \sum\nolimits_{i = 1}^{t} {w_{dmi} } \times S_{dmi} + \sum\nolimits_{j = 1}^{m} {w_{dj} \times R_{ppj} } $$
(2)

\( R_{dc} \) is the assessment value of unsafe event type of safety performance indicators, \( R_{dnc} \) is the assessment value of non-unsafe event types, \( R_{db} \) is the assessment value of foundation type, \( R_{dm} \) is the assessment value of management type, \( R_{dp} \) is the assessment value of operation type, \( w_{dj} \) is the weight of different post, \( R_{pcj} \) is the assessment value of unsafe event type for post \( j \), \( w_{dbi} \) is the weight of foundation type, \( w_{dmi} \) is the weight of management type, and \( R_{ppj} \) is the assessment value of operation type for post \( j \).

At the civil aviation organization level, safety performance indicators mainly include unsafe event type, operation type, management type, and foundation type. Then, the safety performance of the civil aviation organization can be calculated as Eq. (3).

$$ R_{o} = R_{oc} + R_{onc} = \frac{1}{k}\sum\nolimits_{i = 1}^{k} {R_{dci} } + \sum\nolimits_{i = 1}^{k} {w_{oi} \times R_{dnci} } $$
(3)

\( R_{oc} \) is the assessment value of unsafe event type of safety performance indicators, \( R_{onc} \) is the assessment value of non-unsafe event types, \( w_{oi} \) is the weight of different department, \( R_{dci} \) is the assessment value of unsafe event type for department \( i \), and \( R_{dnci} \) is the assessment value of non-unsafe event type for post \( i \).

3 Safety Performance Assessment Tool

A safety performance assessment tool is developed to realize the function of safety performance indicator definition, data filling of safety performance indicators, and data statistics and presentation. The structure of this safety performance tool is illustrated as Fig. 4.

Fig. 4.
figure 4

Structure of the safety performance assessment tool

Full size image

4 Case Study

Civil airports are important infrastructures in the civil aviation industry. In order to verify the proposed comprehensive safety performance assessment model, a case of a civil airport in Central and Southern Regional of China is studied. There are six departments and twenty key posts involving in the safety performance assessment. For each key post, there are several unsafe event type and operation type of safety performance indicators. The numbers of safety performance indicators for these twenty key posts of different departments are listed in Table 3 below.

Table 3. Numbers of safety performance indicators for different departments and posts
Full size table

The total number of safety performance indicators for different posts is 189. The numbers of management type and foundation type of safety performance indicators for each department are 8 and 4, respectively.

According to the safety performance assessment model described above, the airport’s operation data is used to assess safety performance of the airport, six departments, and 20 key posts. The assessment results are presented in Figs. 5, 6 and 7.

Fig. 5.
figure 5

Safety performance assessment result of follow me car post

Full size image
Fig. 6.
figure 6

Safety performance assessment result of aircraft maintenance department

Full size image
Fig. 7.
figure 7

Safety performance assessment result of the airport

Full size image

Figure 5 presents the safety performance assessment result of follow me car post, other 19 posts’ assessment results can be shown as Fig. 5. Figure 6 presents the safety performance assessment result of aircraft maintenance department; other 5 departments’ assessment results can also be shown as Fig. 6.

5 Conclusion

Safety performance management plays an important role in revealing a civil aviation organization’s current safety status and safety level. In order to assess safety performance of civil aviation organizations, the research progress of safety performance management is summarized. A framework used to assess safety performance is proposed. The life cycle of safety performance indicator is presented to defining/refining safety performance indicators for civil aviation organizations. Safety performance indicators have been classified into four types: unsafe event type, operation type, management type, and foundation type. Weights for different types of safety performance indicators have been assigned based on Heinrich’s Law and in conjunction with experts’ evaluation. A novel safety performance assessment model is proposed to assess the safety performance of a civil aviation organization, departments and key posts. The developed safety performance assessment tool can improve efficiency of safety performance assessment process and realize data visualization. A civil airport in Central and Southern Regional of China is used to illustrate the efficiency and flexibility of the proposed model. 189 safety performance indicators have been defined for 20 key posts. The numbers of management type and foundation type of safety performance indicators for each department are 8 and 4, respectively. The assessment results for the airport, departments, and posts are presented. These assessment results can be used to further analyze the safety performance level and causes, and make the improvement plan.