Accreditation in Europe: benchmarking the operations of European accreditation bodies using an innovative management tool

Abstract

Accreditation is one of the pillars of a national Quality Infrastructure, as the competence of conformity assessment bodies is assured through accreditation performed by accreditation bodies. To compare the operation of accreditation bodies in Europe and to identify best practices, a management tool, the Process Maturity Benchmarking Tool, was developed and validated by applying it to European accreditation bodies. The benchmarking project comprised two major phases: in the first phase, the processes of accreditation bodies were systematically analyzed. A process map was developed, and processes of special relevance were identified and underpinned by indicators. In the second phase, the practical applicability of the theoretical model was demonstrated by analyzing the processes of eight European accreditation bodies. The results of this comparative assessment were subsequently discussed in a workshop with experts from those accreditation bodies, giving the opportunity to identify best practices. This article has a twofold objective. First, to present a method to benchmark European accreditation bodies, based on the European Foundation for Quality Management excellence model. The successful application of the Process Maturity Benchmarking Tool gives evidence that it is a suitable and capable management tool to assess the processes of the European accreditation bodies and to benchmark them. Second, the article presents the results of the first adaption of the Process Maturity Benchmarking Tool. A general trend of process maturity was identified: while processes based on stakeholder involvement tend to have an overall lower maturity on average, internal processes are more mature.

Introduction

In autumn 2014, the German accreditation body (AB) Deutsche Akkreditierungsstelle GmbH (DAkkS) and the German federal institute Bundesanstalt für Materialforschung und –prüfung (BAM) have initiated a project to compare the operations of European ABs. The major goal of the benchmarking project was to discuss and identify good and best practices. The comparison of different operational practices using a structured, systematic procedure is a powerful tool to identify potentials for improvement and to evaluate the current operational performance of an organization. Such a benchmarking is particularly useful for organizations that are not exposed to competition, as corrective incentives from the market are missing and the exchange of data and experiences between peers is acceptable.

This article describes the concept, methodology, development, and first adaption of the Process Maturity Benchmarking Tool (PMBT) and presents the major results of its first application. The article is structured as follows. In the background section, the current situation of ABs in Europe and the theoretical background of the methods used for the development of the PMBT are described [1,2,3,4]. The development of the PMBT and its application procedure are introduced in the methodology section. Subsequently the sample represented by the participating ABs is described, basic data are analyzed. The accreditation process is presented in detail as an example for the application of the PMBT. Results are discussed and conclusions including limitations and prospects are drawn in the appropriately named sections.

Background

Accreditation in Europe

Accreditation, defined as third-party attestation of competence, impartiality, and consistent operation in the performance of specific conformity assessment activities related to a conformity assessment body (CAB) performed by an authoritative body [5], is of major importance to ensure the reliability of conformity assessments. Conformity assessment activities covered by accreditation include testing, calibration, inspection, certification of management systems, persons, products, processes and services, provision of proficiency testing (PT), production of reference materials, validation and verification [6]. In Europe, the requirements for performing accreditation as well as the requirements for bodies acting as an AB are regulated [1]. As required by this Regulation, there is only one national AB in each member state of the European Union. This national AB is responsible for the accreditation of the CABs established in its country, regardless of whether accreditations are mandatory or voluntary. While the requirements for accreditation are regulated in the European Union, the legal status of national ABs is not determined. In addition to establishing a public authority, the member states are also free to entrust a private organization with the implementation of accreditation as a public authority activity and to grant it formal recognition [1]. This flexibility and the different sizes of the members lead to a situation in the European Union where national ABs with different legal status and strongly varying numbers of accreditations issued operate according to the same rules.

The European ABs have to operate on a non-profit basis and shall not compete with other national ABs in their activities [1]. The national ABs of the member states of the European Union may only operate across national borders, on territory of another member state, under defined circumstances [1]. Nevertheless, as these rules do not apply to the accreditation of CABs established outside the European Union, some European ABs compete with each other outside the European market. However, as only a limited number of the ABs which participated in the benchmarking are active outside Europe and only to a limited extent, the ABs can be assumed to be non-competitive. This situation leads to a lack of market-induced incentives to improve and optimize work. Moreover, since there is only one AB in each member state, peer learning at the national level is precluded. Against this background, a project has been designed with the aim of developing a management tool making it possible to compare the processes of ABs in Europe despite their different size, legal status, and historical development.

Management tools

Literature review revealed only scattered publications comparing the performance of the actors in the quality infrastructure system and none comparing the bodies that perform accreditation as defined in ISO/IEC 17000 [5]. A model for the comparison of accredited laboratories based on indicators and a summarizing quality index was developed by Catini et al. [7], it uses indicators evaluated by the participating laboratories through self-assessment but does not refer to process maturities and their description. Karthiyayini et al. [8] identify critical factors and performance indicators related to laboratory accreditation using a benchmarking approach but refer to ISO/IEC 17025 [9] rather than to a holistic management model. The review of suitable management tools for comparing business processes of European ABs leads to the identification of two long-approved approaches: Benchmarking as a tool for comparison and Total Quality Management (TQM) as a model for holistic management. The first book on benchmarking was published in 1989 [3], and since then the tool has been widely used in industry, but also in some cases concerning organizations of national quality infrastructures such as CABs [8, 10], standardization bodies [11] and accreditation and certification infrastructures in specific economic sectors [12]. Benchmarking is widely accepted by economic operators, as it was ranked third of the ten most important management tools in 2017, according to a regular survey on management tools and trends [2], a survey in which benchmarking was already selected as a top ten management tool in 1993. The main added value of benchmarking compared to other improvement tools is that participants can learn “HOW to improve from others” [13]. Due to the non-competitive nature of the European accreditation market, the benchmarking approach is particularly suitable, as there is no commercial reason not to share best practices. Current research on TQM with its recent application Business Excellence gives evidence that despite the type of organization, business excellence models provide a holistic approach to continuous improvement and that these models have been proven their viability [14], including in specific applications related to quality infrastructure institutions [15, 16]. Business excellence models deliver a framework enabling organizations to benchmark their performance and identify best practices [14]. In the periodic survey cited above, TQM is also identified as a top ten management tool in 2017 and receives the highest overall satisfaction rates for the systematic approach to quality improvement. The European Foundation for Quality Management (EFQM) has developed a business excellence model [4], which consists of three components: Besides eight key management principles (Fundamental Concepts of Excellence) and an evaluation scheme (RADAR logic), the third component of the model defines nine criteria and associated sub-criteria. The criteria are subdivided into five Enabler criteria (Leadership; People; Strategy; Partnerships and Resources; Processes, Products and Services) and four Results criteria (People; Customer; Society; Business). The Results criteria are separated into Perceptions and Indicators: Perceptions describe how stakeholders evaluate the results of a process, while Indicators are based on parameters used by an organization to measure its operative performance. Since the EFQM model is applicable to all types of organizations, criteria and sub-criteria are described in a generic way.

Methodology

The unique situation of the European ABs led the BAM-DAkkS project team to the conclusion that a valid comparison can best be carried out by developing a tailor-made management tool. As the EFQM business excellence model [4] is internationally accepted, has been used for decades and is adapted to the European economic landscape, it was identified as an appropriate management tool for comparing European ABs. Therefor the PMBT refers to the EFQM excellence model as framework for the evaluation of the maturity of selected processes. The PMBT determines measurable indicators based on a) data from data records of the ABs and b) self-evaluations of process maturities. These two sets of information are used to investigate and compare the way the ABs perform a defined set of processes, regardless of different operational procedures. The customization and application of the PMBT consist of a series of four consecutive steps and is described below for its first application:

Step 1: Identification of processes, criteria and indicators to be used for the benchmarking.

The PMBT is based on the analysis of selected processes with high relevance for the operation of ABs. The processes are derived from a generic process map tailored to ABs. To elaborate on this overview, all processes necessary for running an AB are identified, listed, analyzed, and then reduced to a simplified, universally applicable process map. The specific processes to be used for the first application of the PMBT were identified by experts from DAkkS and BAM. This resulted in the nine relevant processes from the three major process groups (management, business and support processes), which are highlighted in Fig. 1. For the first type of information used—data from data records of the Abs—a set of 210 variables was identified. These cover a wide spectrum from economic data (e.g. turnover, revenue), personnel-related data (e.g. working days, overtime), customer-related data (e.g. type, size, number of complaints, number of suggestions) to process specific data (e.g. duration of specific process steps). These data were used to compare the participants and their operations based on measurable indicators. Evaluation criteria regarding the maturity of the nine selected processes—the second type of information used—were determined by combining the generic Enabler and Results criteria of the EFQM model [4] with the processes selected for comparison, thereby identifying the EFQM-criteria and sub-criteria relevant for the specific processes. Thirty-three evaluation criteria were identified, and the 16 criteria used for examining the maturity of the business processes are shown in Fig. 2. The criteria were then customized to the specific procedures of an AB. Measures related to the Enabler criteria are underpinned by short descriptions of typical steps describing a growing process maturity from minimum requirements to best practices, resulting in four to eight steps for each criterion (IT integration is presented as example below). For the results achieved, up to 11 ideal state descriptions of specific results related to each of the nine selected processes were elaborated. The degree of achievement reached for each step of a measure and each specific result is evaluated based on a scale between 1 and 5.

Fig. 1

Generic process map of an AB. Processes which have been selected for comparing the operations of ABs are marked with a black frame, those not used for the comparison are shaded grey

Fig. 2

Evaluation criteria regarding conducted measures and achieved results used to examine the three major business processes of an AB

Step 2: Elaboration and calibration of the Data Collection Guideline.

A description of the required data from data records and the maturity descriptions were merged in a Data Collection Guideline, designed in a modular way, making it possible to draw up individualized versions, reduced to the processes to be evaluated by a certain person. In summer 2016, the preliminary Data Collection Guideline was validated in cooperation with experts from DAkkS. In interviews with the DAkkS employees responsible for the respective processes, the process maturity descriptions and the data supposed to be collected from data records were scrutinized. Based on these interviews, the guideline was optimized and supplemented by aspects not considered before. Finally, the revised Data Collection Guideline was approved by the project team from DAkkS and BAM by the end of 2016.

Step 3: Sample selection, data collection and analysis.

The members of the umbrella organization of European ABs, the European co-operation for Accreditation (EA), were informed about the intention to conduct a benchmarking in September 2016. Besides DAkkS, seven European ABs decided to participate in the project, all being full members of the EA and signatories of the EA Multilateral Agreements (MLAs) for testing, medical testing, calibration, inspection, management systems, products certification, and certification of persons. As the EA itself is a peer evaluated Recognized Regional Accreditation Group, the EA MLA is recognized at international level by the International Accreditation Forum (IAF) and International Laboratory Accreditation Cooperation (ILAC), thereby being equivalent to the Multilateral Recognition Arrangement of IAF (IAF MLA) and the ILAC Mutual Recognition Arrangement (ILAC MRA) respectively.

The data from the participating ABs were collected between January and June 2017. As kick-off of the project, an information meeting was conducted at the premises of all seven ABs not familiar with the PMBT, during which the relevant process owners, executives, and managers of the respective AB were introduced to the concept, approach and upcoming steps of the benchmarking. A few weeks later, data acquisition was conducted in the course of interviews with the persons responsible for the processes concerned. This practice turned out to be very useful because the opportunity to clarify the details and the handling of the PMBT approach ensured a common understanding of the benchmarking criteria. In cases where a process was handled by different organizational units of an AB, each responsible person of each unit concerned received a personalized Data Collection Guideline. All responses for a specific process were then weighted equally, regardless of the size of the units or the number of accreditations managed by each unit. In order to promote that the ratings were not distorted by the intention of glossing over their own organization to the public, it was decided to keep the participating ABs anonymous in all publications. Finally, the responses from each AB were merged in a single file, which was approved by the respective AB prior to usage. All further work on the raw data, the statistical analysis, the calculation of the indicators, and the computation of the graphics were performed with the statistical software “STATA” (Version 12.1).

Step 4: Conduction of the Results-Workshop.

The results of the data analysis were presented and discussed during a two-day Results Workshop in fall 2017 attended by 28 experts from the eight ABs. The workshop was organized in three work sessions: First, all participants were given an overview of remarkable results from the analyzed processes with an emphasis on the measurable indicators represented by the data from data records. In the second session, the participants were divided into two groups, one group representing top management personnel and the other group consisting of experts with executive tasks. Both groups discussed the processes relevant to their area of responsibility, focusing on the maturity of the respective processes. In the third session, again two groups were formed, the first one took a close look at the management of stakeholder relationships, the second worked on the human relations management. Here, too, the focus of discussion was the maturity of the respective process. The workshop was concluded with a joint debate on the participants’ personal impressions of the benchmarking process and their wishes and proposals for future activities.

The PMBT approach of comparing process maturity self-evaluations is described below for a certain aspect of the accreditation processes, since the initial accreditation of a competence is, together with the surveillance of accredited competencies, the most important value-adding business process of an AB. The largest part of the total revenue is generated by their implementation. Moreover, since new customers have their first business contact with an AB in the context of the initial accreditation of a competence, this process is of outstanding importance in terms of customer satisfaction. The measures related to the accreditation processes were evaluated in terms of customer orientation and process management (Fig. 2). The maturity of the process management was considered under the aspects process design and key indicators, IT integration, and process improvement as shown in Fig. 2. As the handling of large amounts of data, the reliable safeguarding of workflows and deadlines as well as the cooperation with numerous external experts make a comprehensive and user-oriented IT integration of the accreditation process a prerequisite for controlled, robust and promptly monitored services, the maturity descriptions and the related self-evaluations of the aspect IT integration are presented in detail the next section as an example of the application of the PMBT approach.

Results

The participating European ABs cover a broad spectrum in terms of size, finances, personnel, and clients. A comparison of the type and number of accredited CABs reveals a wide range in the absolute number of accreditations issued, as three of the ABs had accredited less than 500 CABs by December 2016, while the AB with the most customers had issued more than 4000 accreditations by that date (Fig. 3, right graph). The number of accredited CABs refers to the different types of CABs, not to locations or legal entities. A single legal entity can be accredited for different types of conformity assessment, resulting in more issued accreditations than organizations performing accredited conformity assessment tasks. The participating ABs issued a total of 11,418 accreditations by the end of 2016, thereby representing 33 % of all accreditations granted by EA members that year [17].

Fig. 3

Number of CABs (CAB₂₀₁₆), subdivided by their type, holding valid accreditations of the respective AB at 2016-12-31, and number of initially accredited CABs in 2016 being new customers of the respective AB (CAB_nc₂₀₁₆)

The number of accredited CABs handled by the participating ABs correlates with relevant macroeconomic indicators, in particular the population and the size of the economies of the respective countries given by the gross domestic product (GDP) (Fig. 4). A strong positive correlation with the GDP was also found in previous studies for the number of accredited laboratories in North and South America [18], in Croatia [19] and in Europe [20]. Nevertheless, between the participants the number of accredited inspection bodies and medical laboratories varies independently from the macroeconomic indicators. The exclusion of these two types of CABs from the analysis leads to a regression fit that explains the variation of accredited CABs as a function of the macroeconomic indicators better: The coefficients of determination R² increase from 0.89 to 0.96 for the GDP and from 0.86 to 0.93 for the population. This result is in line with the relationship between the total number of accredited laboratories and the GDP in North and South America, which also shows a strong positive correlation, indicated by a high Pearson correlation coefficient of r = 0.97 [18]. The corresponding value of the Pearson correlation coefficient between the accredited CABs excluding inspection bodies and medical laboratories and the GDP of the eight European ABs is r = 0.98. Finding that the number of accredited inspection bodies and medical laboratories, in contrast to the other types of CABs, does not correlate with the size of the respective national economy, points to varying national approaches related to the accreditation of these types of CABs. As the decision to become accredited is either a business decision or the result of a regulatory requirement [19], the absence of a correlation with the size of the national economy indicates that the variations regarding inspection bodies and medical laboratories are based on differences in national regulation. Grochau et al. [20] find a deviation of the number of accredited laboratories from the expected pattern shown by GDP or population in some European countries. The hypothesis the authors mention to explain this also points to regulatory issues, leading to a high number of laboratories in areas such as environment, human and animal health in the concerning countries.

Fig. 4

Number of accredited CABs excluding inspection bodies and medical laboratories (CAB_{2016_im}) as function of the GDP and the population. The fitted values are calculated by linear regression; R² is the coefficient of determination. Data for GDP and population taken from the EUROSTAT database. All data are for 2016, the number of accreditations issued CAB_{2016_im} applies to 2016–12-31

The initial accreditations of new customers show significant differences between the ABs regarding the type of CAB accredited in 2016 (Fig. 3, left graph): In three countries, laboratories that perform medical examinations are the dominant type of new customer; in one country inspection bodies dominate. There is no correlation between the number of new customers accredited as inspection bodies or medical laboratories (Fig. 3, left graph) and the size of the AB measured by the absolute number of accreditations issued (Fig. 3, right graph), thereby confirming the trend described above for all customers of the ABs. In contrast, for new fields of accreditation such as PT¹ we observe a different picture: initial accreditations of PT providers only occur in countries with an annual GDP > 1,000,000 million € in 2016, indicating that PT providers initially seek accreditation in the larger economies. The assumption that the CABs accredited in 2016 were first movers is confirmed as the trend of a growing number of accredited PT providers continued, with a total of 179 accreditations granted by EA members by the end of 2018 [23].

The way the work is performed by the ABs differs: the distribution of work between internal staff and external assessors/technical experts indicates that the ABs with the smallest and largest workloads rely more on external expertise than the medium-sized ABs, which perform their tasks without allocating more than 25 % of the work to external experts and assessors (Fig. 5). The overtime worked by internal staff also differs: the two ABs which rely most on external personnel (W_ex/tot > 25 %) as well as the two with the lowest percentage of external work (W_ex/tot < 15 %) need a significant amount of overtime to fulfill their duties (Fig. 5, lower graph). In contrast, the ABs allocating 20–25 % of their total work to external assessors and technical experts did not need comparable amounts of overtime in 2016. When taking into account only the larger ABs with more than 10,000 working days (WD_tot) in 2016, the proportion of external work enlarges as the absolute workload increases, thereby enabling the ABs to handle more customers and to cover larger technical scopes without increasing the internal staff accordingly.

Fig. 5

Work days provided by the ABs WD_tot in 2016 and amount of overtime WD_{ot_in} in 2016. One participating AB did not deliver data regarding time worked

As example for the analysis of process maturities based on self-evaluated achievement levels the measures of the IT integration aspect of the process initial accreditation of a competence are shown in Fig. 6. The answers of the ABs are summarized as box plots right to each maturity step described on the left. The medians of the achievement levels rapidly decline up step 4. Maturity step 4, which is associated with the lowest median of all steps, refers to an online application procedure with a linkage to the database of the AB. Although such an online application tool was rated as very useful by the participants of the Results Workshop as described below, its realization seems to be a large burden for the participating ABs. The very wide spread of responses for the steps above step 2 points to large differences in the (evaluated) capabilities of the IT systems used by the ABs to manage their accreditation processes (step 3, 6 and 7) and in the (evaluated) ability of the ABs to adapt their IT solution promptly (step 5).

Fig. 6

Maturity steps of the IT integration of the initial accreditation process (left side) and corresponding self-evaluations regarding the achievement of the respective step (right side), summarized as box plot (outside values not plotted)

Discussion

As seen in other parts of the world the number of CABs accredited by the eight participating European ABs is growing roughly linearly with the macroeconomic indicators GDP and population, while for some specific types of CABs significant variations can be identified, often caused by differences in national regulations. But not all variations in the ratio of the types of CABs accredited in the eight participating countries are related to regulatory issues. As seen above accredited PT provider first show up only in the larger economies, indicating that for this new field of accreditation business decisions predominate.

As the IT integration of the accreditation process and the usage of IT solutions for the provision of information revealed quite a large variation in the respective process maturity, this issue was discussed intensely during the Results Workshop. All participants agreed on the need for IT systems for running their processes. Different approaches to the design as well as the financing of new IT systems were discussed, including the solutions and limitations of online application systems. Regarding the online application procedures, the lack of a unique, unambiguous description of all scopes of accreditation listed in the annexes to the accreditation certificates was identified as a burden. Due to the specific requests of the CAB clients or governmental authorities, equal competencies are sometimes described in different ways, making an online accessible list of competencies from which to choose during the application process confusing. A notable point was the question of whether written accreditation documents are required or whether a database entry managed by the AB is sufficient. While most participants print accreditation documents, one AB abolished them completely and now only provides information on the accreditation status and scope of CABs through an internet-accessible database. It was also discussed whether synergies could be exploited in the programming of software solutions between the ABs, but especially due to language barriers such cooperation was not considered feasible.

For the analysis of the overall process maturity, all self-evaluations related to the maturity of the measures are summarized in 10 categories, each of which represents either solely one of the five Enabler criteria of the EFQM model or a major aspect of one of these criteria, adapted to the specific processes of ABs. The means of the achievement levels associated to each process maturity step were calculated and assigned to one of the 10 categories. The distribution of the means for these 10 categories is shown in Fig. 7 as box plot, and the categories are sorted by the median of the respective box plot. The sorting shows that categories with a focus on internal processes such as leadership, strategy, or competence management are characterized by relatively high levels of achievement in the respective maturity steps. In contrast, processes that are characterized by stronger interaction with external stakeholders such as customer orientation, accreditation process, and product development process have the lowest levels of achievement.

Fig. 7

Mean values of the self-evaluated achievement level of the maturity steps related to one of the 10 categories of measures, summarized as box plot and sorted by the median of each category. n is the number of evaluations in the respective category used for calculation. Outside values not plotted

According to the measures, the achievement levels related to the results were summarized in nine categories, each representing a major aspect of one of the four Results criteria of the EFQM model, adapted to ABs. The distribution of the mean of the achievement level of the self-evaluations related to each ideal-state description, categorized in nine aspects, is shown as box plot in Fig. 8. The self-evaluations related to achieved results have a similar trend as those of the measures: the lowest values are attained in terms of meeting customer and stakeholder expectations and for future viability, results that are strongly related to interactions with external stakeholders. High values, on the other hand, can be identified for results associated with the daily business of the ABs, such as business process performance, financial performance, and performance of assessors/employees. The best values of all categories and a low spread characterize the self-evaluations of the reputation of the accreditations issued by the respective AB.

Fig. 8

Mean values of the self-evaluated achievement level of the ideal state descriptions related to one of nine categories of results, summarized as box plot and sorted by the median of each category. n is the number of evaluations in the respective category used for calculation. Outside values not plotted

Conclusions

The first application of the PMBT with eight European ABs proved its usability as a tool for organizing benchmarking between organizations of different sizes running comparable processes under different circumstances, but not acting as competitors. The tool provides a framework for a systematic analysis of selected processes based on a holistic management model. As described above, the participants are encouraged to share experiences and discuss practices in a targeted way, thereby validating the applicability of the PMBT approach. The successful application of the PMBT approach has given evidence that the comparison of different operational practices using a structured, systematic procedure is a powerful tool to identify potentials for improvement and to rate the current operational performance of an organization.

In addition to numerous findings the participating ABs had about best practices for specific processes, an overall trend towards process maturity was identified, pointing to an area of improvement: Processes with a predominantly internal focus, which are mainly characterized by the interaction of internal personnel, are rated by the ABs as relative mature. In contrast, processes that are strongly determined by interactions with external stakeholders are evaluated as less mature. To optimize their services, ABs should consider involving external stakeholders more in their thinking and strategic planning. The analysis of the distribution of work between external assessors and experts and internal personnel indicates that ABs which allocate 20–25 % of their work to externals are less often forced to ask their personnel for overtime than ABs relying less or more on external expertise.

There is a strong positive correlation between the size of the country's economy and the number of accreditations issued by the national AB, but significant variations are observed in the ratio of the different types of accredited CABs. Although it is not possible to make statistically valid assumptions about the European accreditation landscape due to the very small sample and the sample selection (see below), this distribution may point to a general trend. A closer analysis reveals two reasons for the variation: for the established types of CABs such as inspection bodies and medical laboratories, variations are mostly related to different national regulatory approaches, while new types of conformity assessment are evolving first in larger economies, indicating that economic operators rather than regulatory reasons are the relevant driver here. Further research is needed to verify this and clarify the reasons for the different national approaches.

The study has some limitations: only eight ABs representing less than 1/4 of the EA full members and only about 1/3 of the accreditations issued by EA full members participated in the benchmarking, and the participants were not chosen by random or targeted selection but volunteered. Additionally, a relevant proportion of the data is based on self-evaluations, limiting the comparability of organization-specific process maturities. Therefore, statistically proven conclusions about the European ABs as population are not possible.

Outlook: a second application of the PMBT with ten participating European ABs was successfully carried out with data collection in summer 2019 and workshop in January 2020. First analyzes and the positive feedback of the participants confirm the suitability of the PMBT and indicate the validity of the results presented above.