1 Introduction

The Covid-19 pandemic has clearly demonstrated how essential digital technologies are to modern economies and societies, enabling public services, schools, universities to function, and businesses to maintain their viability even in times of lockdowns and restrictions. The Digital Economy and Society Index (DESI), first published by the European Commission in 2015, monitors and assesses the digital transformation in the European Union and its member states through a quantitative measurement system and related reports. The DESI itself is a composite index that combines several ICT-related indicators with predefined weights, using a scoring model to rank countries in order to monitor the development of the digital economy and society. The DESI framework measures digitalization performance in five principal dimensions: Connectivity, Human Capital (digital skills), Use of Internet, Integration of Digital Technology (use of digital technology by businesses) and Digital Public Services (see Table 1).

Table 1 Dimensions of DESI.
Full size table

The TOPSIS method is essentially a very simple, geometric approach based on decision theory that attempts to eliminate procedures based on data transformation. The method consists of three consecutive steps. (It is often summarized in six steps, but this is based on the three basic methods we describe.) In the first step, the scale problem between the data is addressed with a normalization transformation, adjusting the values measured on different scales to a notionally common scale. That normalization can be based on Euclidean distance on the unit sphere or the data can be transformed to a [0,1] interval with an affine transformation. The resulting normalized data is then weighted by a weight vector. The weights can be subjective, given a priori, or objectively determined from the statistical properties of the available data by theoretical or mathematical statistical considerations. A third type of weights is called integrated weights systems (Odu 2019; Trzaskalik 2023). Finally, in the third step, the TOPSIS efficiency is calculated using the ratio of the distance between the normalized, weighted data to ideal and nadir (negative ideal) points, the order of which gives the ranking. In this paper, we address the research questions of how different normalization methods affect the determination of weights, and how similar (dissimilar) our rankings with objective (TOPSIS) weights are to each other and to the rankings of the original DESI scoring model. The purpose of the research is to test the robustness of the European Commission’s ranking and its sensitivity to the methods of normalization and weighting.

The paper will consist of five sections. In the following (second) section, a brief overview of the DESI methodology is given, as well as the four data normalization methods used for the TOPSIS method, and the five objective weights used. The third chapter presents the mathematical details of the methodology used for the TOPSIS method. The next section presents the results. In the interests of brevity, the twenty rankings themselves are not presented, only the vectors of objective weights, as well as the similarity indices of Pearson and Kendall's tau-b correlation coefficients and dissimilarity indices of Garuti compatibility and Euclidean distance function of the solutions with the DESI index, which shows the robustness of the results. Finally, we summarize the results and present our conclusions in the last section.

2 Literature review

The literature review consists of two parts. First, the focus is on the DESI index and the related I-DESI index and their applications in scientific literature. Then the procedures and applications of the TOPSIS method are described.

2.1 The DESI and I-DESI indices

Bánhidi et al. (2020) analyze the five principal dimensions of DESI using multivariate statistical methods. The authors first investigate the linear relationships between the dimensions using simple Pearson and partial correlation analysis and factor analysis, with a focus on possible causal relationships based on the partial correlation analysis. The objects (EU member states) in the data set are grouped with cluster analysis, then grouped and ranked using Multidimensional Scaling (MDS), and the resulting rankings are compared to those based on the European Commission’s original scoring model. The results of the study support the European Commission's thesis that the five main dimensions of DESI are interrelated, but also suggest that at least one of them could be redundant (based on the multicollinearity in the data set).

Tokmergenova et al. (2021) carry out a similar analysis focusing on the multicollinearity between dimensions and the statistical relationships between them using data from the International Digital Economy and Society Index (I-DESI), which can be considered an international extension of the DESI. Their results show a high degree of multicollinearity between the dimensions.

Bánhidi et al. (2019) investigate the stability of rankings based on the same I-DESI data set. To do so, the authors use the basic DEA and the DEA Common Weights (DEA/CWA) methods, as well as a one-dimensional version of MDS, which they consider suitable for ranking decision-making units (DMUs – in this case, countries). Their results show that these ranking methods offer a similar solution to the European Commission's original scoring model, but that the ranking of some countries (e.g., Russia) shows more significant variation depending on the method chosen.

The research questions of Bánhidi et al. (2021) also relate to the stability of the I-DESI ranking and the position of Russia, and their results (using the DEA/CI method) show that the Russian Federation achieved respectable results in digital economic and social development relative to Eastern and Southern member states of the European Union.

Moroz (2017) discusses the main (international) indicator systems used to measure and compare the digital development of countries, analyzing Poland's position and the dynamics of its development based on two selected measurement systems, the DESI and the World Economic Forum's Networked Readiness Index (NRI). His results show that Poland's position is relatively unfavorable, with a low level of development and digital competitiveness, coupled with a low growth (catch-up) rate.

Kotarba (2017) also discusses composite indices of digital development, including the DESI. He suggests that digital development can be measured at five levels (stages): society, sectors, firms and customers, in addition to the economy as a whole. The article discusses the similarities and differences between the main indicator systems, and makes suggestions for their improvement.

The work of Laitsou et al. (2020) uses the DESI and its five main dimensions to evaluate the digital performance of the Greek economy and uses the Gompertz model to predict how Greece can converge to leading EU countries in terms of digital development. They predict that, although the country faces a number of challenges on both the demand and supply sides of digital markets, with the right government policies, it could catch up with the EU average by 2030.

2.2 Data normalization and objective weights in TOPSIS method

Data normalization is the first important step in the TOPSIS method. The aim of normalization is to bring the data according to the criteria to the same length scale, which filters out differences in magnitude between the different dimensions. In Chakraborty and Yeh (2009), Papathanasiou and Ploskas (2018), Sarraf and McGuire (2021), and Vafaei et al. (2018, 2021), numerous data normalization methods are presented. Four of them are examined in the present study. Three of these methods are linear, while the last, fourth, uses Euclidean distance. The transformation of criteria also depends on whether the criteria are of benefit or cost type. In the case of the DESI dimensions, all of them can be characterized as benefit criteria, i.e., the country with the highest score is the best country according to each dimension. After data normalization, the weights are determined.

There are three main families of methods for determining weights: subjective, objective and integrated. Subjective weights are given a priori (based on expert judgement), and are used inter alia by the original DESI scoring model. Our analyses are carried out using the objective weighting method (Nasser et al. 2019; Odu 2019; Şahin 2021). This is called objective because it is based on the statistical property of the available data set. The weights are therefore endogenously determined from the data, so that no external influence affects their magnitude. Five of the known objective weighting methods are used. The weighting methods are described later (in Sect. 3.2).

3 Using of TOPSIS in ranking of countries (DMUs)

A short overview of the TOPSIS method was given in the Introduction, which is summarized in Table 2. The three steps described are illustrated by the methods we use. In the detailed description of the TOPSIS methodology, it is assumed that the decision-making units (DMUs) and their associated criteria are available for evaluation. However, it is also assumed that the criteria are of the benefit type, i.e., the decision units with the highest value are the best according to the given criterion. We can do this because the five dimensions of the DESI are benefit variables.

Table 2 The TOPSIS method in three steps
Full size table

3.1 Data normalization

In the first step, the normalization of the basic data is performed. Suppose that the data for criterion j according to each decision-making unit i, i.e., EU member state are contained in the vector xi = \({({x}_{ij})}_{j=1}^{m}\) (for the concrete data set comprising the five principal dimensions of DESI, see Table 11 in the Appendix). Suppose there are n countries (DMUs) and m dimensions (criteria). Then the types of data transformation are as shown in Table 3.

Table 3 Normalizations approaches used in the analysis
Full size table

The minimal and maximal values of criterion j are \({x}_{j}^{min}\) and \({x}_{j}^{max}\). With this transformation, the values of each criterion for each country were transformed to the interval [0,1]. Let the value of the new vectors be yi. The determination of the weight vectors is based on these yi vectors, which will be discussed in the next section.

3.2 Determination of the objective weights

In the second step, knowing the values of the individual transformed variables, in our case dimensions, the weights are determined. Table 4 provides the mathematical description of the five objective weighting methods. In Table 4, \({\sigma }_{j}\) represents the standard deviation, \({V}_{j}\) the variance and \({r}_{kj}\) the correlation coefficient between the kth and jth criteria.

Table 4 Objective weighting methods used in the analysis
Full size table

3.3 Identifying TOPSIS efficiencies

The weighted normalized values are denoted by zji, which is equal to.

$${z}_{ji}={w}_{i}\bullet {y}_{ji}.$$

The ideal and nadir points are then determined using the zji values.

Finally, in the third step, the efficiency index is used for the weighted data to determine using the ideal (Ii) and nadir (Ni) points, which are calculated in the following way:

$${I}_{i}=\underset{j=\mathrm{1,2},\dots ,n}{\mathrm{max}}{z}_{ji}, {N}_{i}=\underset{j=\mathrm{1,2},\dots ,n}{\mathrm{min}}{z}_{ji},(i =\mathrm{ 1,2},\dots ,m)$$

The distance of the jth country from the ideal and nadir is determined as follows:

$${d}_{j}^{I}=\sqrt{\sum_{i=1}^{n}{\left({z}_{ji}-{I}_{i}\right)}^{2}}, {d}_{j}^{N}=\sqrt{\sum_{i=1}^{n}{\left({z}_{ji}-{N}_{i}\right)}^{2}},(j =\mathrm{ 1,2},\dots ,n).$$

A final calculation is the determination of the TOPSIS efficiency Ej, which shows the ratio of the distance from the two aforementioned points:

$${E}_{j}=\frac{{d}_{j}^{N}}{{d}_{j}^{I}+{d}_{j}^{N}}, (j =\mathrm{ 1,2},\dots ,n).$$

The value of the Ej indicator illustrates the efficiency. The closer this TOPSIS indicator is to one, the more efficient the DMU (in our case, the country), is considered to be. This also means that the more efficient a country is (with respect to its digital development), the closer it is to the ideal point and the further it is from the nadir.

After a brief description of the TOPSIS method, we describe the results of our calculations performed on the data set. We omit the detailed calculations, only the objective weights, and the correlation between TOPSIS efficiencies and the DESI indicator are presented in the following Tables in the next section. Table 12 in the Appendix shows three orders, namely EU-DESI, MW-MAX and SD-MaxMin.

4 Main results: TOPSIS weights and the similarity of rankings

For the concrete calculations, we used the DESI data set introduced in the first section (European Commission 2020). We evaluated the digital development of the European Union, which still had 28 member states in 2020, along the five main dimensions of DESI. Four normalisation and five objective weighting procedures were combined to obtain the TOPSIS efficiencies and to establish the rankings of EU member states. In total, twenty TOPSIS models were tested. Two research questions can be asked in this context:

  1. 1.

    How do the different normalization procedures affect the development of weights?

  2. 2.

    What is the linear relationship between TOPSIS efficiencies according to the twenty possible methods and the DESI scoring model? Also, how do the twenty rankings based on the TOPSIS efficiencies relate to the original one based on the European Commission’s scoring model?

The two research questions are answered in the following. Similarity and dissimilarity indices are used to compare the rankings of each TOPSIS model. Similarity indicators are correlation coefficients known from statistics. Dissimilarity indices rely on the properties of distance functions. We use two dissimilarity indicators to illustrate the similarity between rankings:

  • The Euclidean distance and

  • The Garuti compatibility index (Garuti 2020).

4.1 Comparison of TOPSIS weights

Table 5 shows the total of seventeen possible objective weights (maximal elements in each column and the last row are marked in bold). The mean weight objective weighting method does not depend on the normalization procedure, so there will be only seventeen types of weights.

Table 5 Objective weights and normalization methods used in the analysis
Full size table

The weights are first sorted by the Connectivity dimension. This showed that the CRITIC objective weighting method resulted in fairly similar weight vectors regardless of the normalization procedure used in the first step. These weight vectors were also quite similar to the subjective weights of the European Commission, except that the Human Capital dimension was given significantly less weight and the Digital Public Services dimension more weight. In addition, according to these weights, the Connectivity dimension is slightly more important and the Integration of Digital Technologies is somewhat less important.

The remaining thirteen objective weighting methods other than CRITIC were then assigned to three groups with roughly similar results, which were then ranked in descending order along the Integration of Digital Technology dimension. In this dimension, there is a significant gap between the richer core member states in the north and west and the poorer periphery in the south and east. In the first group, weights for this dimension were above 0.400, substantially determining TOPSIS efficiencies. This group includes the weights of the Entropy Method and the Statistical Variance Procedure: there are three Entropy and two SVP procedures in this group.

In the second group, an Entropy Method weighting was used, but with the Max–Min normalization procedure. This was done because the Human Capital dimension’s weight in this TOPSIS model, at 0.336, is significantly higher than the Human Capital dimension weights obtained by the other methods.

Finally, the other results could not be classified so clearly and can therefore be considered as one heterogeneous group. Groups are distinguished by grey and white backgrounds.

The MDS statistical method was used to test the validity of the results, using SPSS29 software. (Borg et al. 2018) The resulting map is shown in Fig. 1. The map is considered to be very good because the stress value is 0.02144 and the R square of the distances is 0.99855, which shows a perfect mapping to the plane.

Fig. 1
figure 1

Source: Own compilation

MDS map of objective weights.

Full size image

After comparing the weights, the TOPSIS rankings are compared with the DESI indices.

4.2 Comparison of rankings with correlation similarity indices

Pearson correlation is used first, which has the disadvantage that it only compares TOPSIS efficiencies with the DESI index, but does not tell us anything about the ranking between countries.

Table 6 shows the Pearson correlations between each TOPSIS model and the DESI scoring model. It is immediately apparent that all correlation coefficients are greater than 0.900, suggesting that the results all of our models are similar to those of the original scoring model. However, two methods, namely the MW-MAX and SD-MaxMin models, stand out with correlation coefficients of 0.997.

Table 6 Pearson correlation between DESI Scoring model and TOPSIS models.
Full size table

The high Pearson correlation between the efficiencies does not reflect the correlation between the rankings, so we also estimated the Kendall’s tau-b rank correlation. (Samara and Randles, 1988) The results are shown in Table 7. This indicator also falls between minus one and plus one. However, a strong relationship between two ranks is found when the absolute value of the correlation coefficient is greater than 0.700. Our calculations show that all orders have a strong relationship with the DESI order, but the same two orders also proved to be the best for the Pearson correlation, so the two types of correlation give almost identical results. The highest values here are also shown in bold.

Table 7 Kendall’s tau-b correlation between DESI Scoring model and TOPSIS models.
Full size table

After the similarity indices, the dissimilarity indices are used to compare the relationship between the rankings.

4.3 Comparison of rankings with distance-type dissimilarity indices

Recently, the Garuti compatibility index has become popular, as dissimilarity indicator. It has numerous applications, especially in the AHP method. In practical problems it is mainly used to solve logistical problems. (Duleba et al. 2021; Duleba and Szádoczki 2022, and Orbán-Mihálykó et al. 2023).

The Garuti index defines the distance between two vectors by the following relation: The Garuti index defines the distance between two vectors \(x \mathrm{and} y\) by the following relation:

$$G\left(x,y\right)=\sum_{i=1}^{n}\frac{{x}_{i}+{y}_{i}}{2}\bullet \frac{{min}_{i}({x}_{i},{y}_{i})}{{max}_{i}({x}_{i},{y}_{i})}.$$

The values of the Garuti indices are then mapped to an ordinal scale, as shown in Table 8, which measures how compatible the vectors are with each other. The Garuti values between DESI sequences are demonstrated in Table 9.

The highest value is then considered the most compatible. This value, marked in grey, is the value given by the CRITIC weighting. However, it should also be noted that even the best compatibility will be no better than “moderate”, as can be seen by comparing Tables 8 and 9.

Table 8 Scale of value for G compatibility index.
Full size table
Table 9 Garuti compatibility indices between DESI Scoring model and TOPSIS models.
Full size table

Table 10 shows the Euclidean distance dissimilarity indices. The distances result in much better discriminability than correlation similarity indices. In this case, too, the CRITIC weighting and the order formed by the MAX normalization procedure give the order most similar to the DESI order.

Table 10 Euclidean distances between DESI Scoring model and TOPSIS models.
Full size table

5 Conclusion

In conclusion, among the seventeen possible vectors of objective weights, we could identify three distinct homogeneous groups and a more heterogeneous leftover group based on their MDS map. In particular, all weight vectors obtained with the CRITIC weighting procedure, and all but one obtained with the ENTROPY procedure exhibited a high level of similarity regardless of the normalization method used in the first step.

Moreover, the normalization and weighting methods chosen had little effect on the resulting rankings, which all exhibited strong or very strong linear association with each other according to both the Pearson and the Kendall rank correlation coefficients. The rankings obtained with the SUM and Vector normalization methods, in particular, were almost identical regardless of the weighting method used in the second step.

The TOPSIS rankings were also generally similar to those obtained with the original DESI scoring model, confirming the latter’s robustness. The value of Kendall’s tau-b correlation coefficients all exceeded a threshold value of 0.7 and the results of the MW-MAX and SD-MaxMin models were almost identical to those of the original scoring model (according to their Pearson correlation coefficients and Kendall’s tau-b coefficients). In terms of policy implications, our results suggest that despite its somewhat arbitrary weights, the original DESI scoring model provides a reasonably accurate assessment of member states’ digital readiness and their “pecking order”, as the rankings are not particularly sensitive to the weights.

There are, of course, several limitations to our research that could also guide future research directions. We have only used the five main dimensions of the DESI 2020 report as the base data for our models, but the research could be extended to the level of sub-dimensions or their constituent indicators. In addition, the methodology could be applied to data from previous or subsequent DESI reports, especially given that the most recent DESI reports publish only four equally weighted principal dimensions (excluding the former Use of Internet dimension).

Table 11 Basic data set (DESI principal dimensions).
Full size table
Table 12 Ranking of EU-DESI, MW-MAX and SD-MaxMin.
Full size table