Keywords

1 Introduction

In more and more industries, agile values [1] are changing the way people work together and influence people’s mindsets. This can be seen in the increasing spread of agile process models [2]. The establishment of the term Agile in software development began with the Agile Manifesto in 2001 [3]. As a reaction to the influence of the increasing trend of digitalization in software development, the Agile Manifesto defines the values and principles for a new approach to the development of digital products, which differs from traditional software development [3]. Agile transformation is understood as a development away from traditional process models towards agile process models in the development of digital products [4]. In the 1980s, Takeuchi and Nonaka [5] had already stated that a sequential phases approach to product development is not well suited because of the lack of flexibility. Digitalization leads to an increasing dynamic of user requirements, which in turn leads to ever shorter development cycles and shorter product launch times. These developments impact the way in which users are involved in the development process [4, 6]. In response to these changes, the solution approach postulated in the Agile Manifesto requires a shift in the focus of software development from process and project to people and product [1].

Agility is a mindset that must be transferred to the specific context of its user, and so, strongly depends on the individual situation. The emergence of an agile way of working takes place here in the context of agile transformation through the adaptation of agile values [1]. An important part of the Agile mindset is continuous improvement based on feedback. For this purpose, measurement is pursued in many subject areas. Research has already dealt with the measurement of agility in recent years. Many of the approaches are concerned with measuring the process using artefacts and workflows [7]. However, the Agile values are not taken into account by these approaches. Agile is more than a process; hence it is important to start with people working together to measure agility.

In this paper we present our questionnaire for measuring agility. Our research is guided by the following research question:

‘How can agility be measured at the team level?’

Our understanding of agility is influenced by the agile values [1]. We decided to focus on the team level because the team is responsible for value delivery in agile product development [8].

Our questionnaire was developed by means of scientific methods. Therefore, the current state of research in the field of measuring agility must be sufficiently considered. For this reason, a literature review was conducted. This literature review serves as the starting point of the construction process for the questionnaire, which must consistently follow the rules of science in order to be able to make generalizable statements about the reality of experience.

The paper is structured as follows: Sect. 2 provides an overview of related work.

Section 3 presents the research method used to develop the questionnaire to measure agility. Section 4 presents the results including the complete questionnaire. Section 5 discusses the significance of the results and limitations. At the end, Sect. 6 concludes with a summary of this paper and our future research projects.

2 Related Work

We used a literature review to identify models for measuring agility. We looked at models that measure agility across the levels of organization, team, and individual to obtain a comprehensive picture of related work. Furthermore, we analyzed related work in terms of their methodology for measuring agility. The results are presented in the following.

On the one hand, we identified and evaluated seven maturity models [7, 9,10,11,12,13,14]. to measure agility in the development of digital products. On the other hand, we evaluated two questionnaires [15, 16].

The authors of the models justify their creation primarily with the realization that no suitable model for agile maturity levels had yet been established at the time of creation [7, 11, 14]. In this respect, it is evident that the identified models are in an early phase of development and have not been consistently and actively developed since their creation [17]. So far, none of the maturity models examined has been sufficiently evaluated and empirically proven.

We can conclude that further conceptual and empirical research is required to enable valid application of the models [16, 17]. In general, these publications make it clear that there is a need for structured approaches to support agile transformation. This view is confirmed in other sources [3, 16]. The need for agile transformation support along with the lack of validation of existing models confirms the need for research. The Agile Manifesto was used as a basis for developing the identified maturity models [7, 9, 10]. In general, the determination of the agile maturity level in the identified models is largely independent of the process model; it is based on the basic concept of agility [17]. Although they originated independently, many of the maturity models examined have a comparable structure. However, it can be seen that the maturity models emerged against fundamentally different theoretical backgrounds and intentions [17]. Accordingly, the requirements that must be achieved with increasing maturity levels are strongly related to the individual context in which the models were defined. Therefore, no uniform, hierarchical model of the requirements for agility across the maturity models could be gained from the analysis of the maturity models [14]. In addition to the context-dependent applicability, the different theoretical backgrounds in the creation of the models examined also mean that the results are not comparable across models when applied [17]. Even though all maturity models depict agility and most have a similar number of maturity levels, it cannot be said that users of different models with the same maturity level are equally advanced in the agile transformation.

The questionnaires (see [15, 16]) studied offer an approach to determining progress in agile transformation that is independent of maturity levels. The scales defined by the questionnaires correspond to the construct of process fields in the maturity models. They delineate different sub-areas of the domain of agility from each other [18]. A hierarchical prioritization of the queried agility requirements is, therefore, not necessary.

The analysis of the identified models has shown that the models each show different ways to achieve agile maturity, while the questionnaires aim to determine agile maturity.

3 Research Method

This paper presents our questionnaire for measuring agility at the team level. Our aim is to develop a standardized questionnaire to measure agility. The results of the measurement should enable suitable measures to be derived and prioritized and the agile transformation to be driven forward in a targeted manner, taking into consideration the agile values.

This section first explains the development of the questionnaire. The development of the questionnaire was carried out by considering the research question ‘How can agility be measured at the team level?’. We, therefore, define the requirements below:

  • Determining of a representative overall impression of the current state of agility at the team level

  • Focusing on agile values independent of the application of methods

  • Supporting the agile transformation process

  • Covering all dimensions of the concept of agile transformation

  • Talking up the least possible time and effort for testing (maximum 10 min)

  • Considering the user-specific context of the participant

The aforementioned requirements were derived from the issues we identified in the related work (see Sect. 2). In order to be able to advance the agile transformation in a targeted manner with appropriate measures, a comprehensive overall impression of the current state of agility is required. By focusing on the agile values, the application of the questionnaire is not limited by the use of a specific agile approach. Furthermore, the need for a short testing time is justified by the conception, that the measurement of agility needs to be done in an agile way. We think that a complex and time-consuming method for measuring agility is in conflict with the concept of agility itself. To take into account, that every agile transformation differs based on the specific context, in which the transformation takes part, this specific context needs to be assessed in the questionnaire.

The development of the questionnaire is a continuous learning process. In the beginning, we started with a literature review (see Sect. 2) and an expert survey in order to identify a list of potential items for our questionnaire (see Sect. 3.1). Then, after the first version of the questionnaire was created, we conducted a pretest (see Sect. 3.2). Afterwards, we adapted the questionnaire to the context of public administration and evaluated it (see Sect. 3.3). We also conducted another expert survey to review the comprehensibility of the items and the assignment to the six defined dimensions of agility (see Sect. 3.4).

3.1 Initial Construction of the Questionnaire

The maturity models and questionnaires for implementing and measuring agile transformation identified in the literature review (see Sect. 2) form the starting point for the construction of the questionnaire. Each of these models defines elementary questions for assessing the current state of agile transformation, which are referred to as indicators in the following. The 539 indicators identified were completely reworded in the construction of the questionnaire so that they could be considered as potential items. After sorting out duplicate and irrelevant indicators, the remaining 386 indicators were assigned to agile values. Afterwards, the indicators defining a common behavior were grouped. These groups were then combined and reformulated into a potential item that reflects the underlying behavior of the assigned indicators. This process resulted in 83 possible candidate items of the questionnaire (see Fig. 1).

Fig. 1.
figure 1

Process of construction of the questionnaire

Full size image

Through an empirical selection based on an expert survey with five experts the 28 most relevant items were selected as assessment questions for the questionnaire. The aim of these assessment questions is to evaluate the current state of agility.

These 28 items were then assigned to the defined six dimensions of agility (see Sect. 4.1). The illustration of this mapping is shown in Sect. 4.3.

The assessment of agility through the assessment questions is initially independent of the specific context of the questionnaire user. In order to take the user-specific context into account, weighting questions were added to the assessment questions. Furthermore, demographic questions to capture the demographic situation of the user were included in the questionnaire. Based on the implementation of a case study, the demographic questions were reduced from five to two questions (see Sect. 4.2). The result of the construction process is a questionnaire comprising 36 items.

3.2 Pretest

Subsequent to the construction of the questionnaire, a pretest was conducted. The pretest represents a prototypical testing of the questionnaire before the actual data collection is carried out; it is an indispensable prerequisite for preparing the main survey [19].

This pretest serves to determine the quality of the items and dimensions as a result of the construction process of the questionnaire. For this pretest, the authors constructed an online version of the questionnaire in the German language. The online questionnaire contains all three groups of questions demographic questions, assessment questions, and weighting questions, and was extended with an introduction that clarifies the context of use for the participants, it contains all three groups of questions. Sixty participants from different companies took part in the pretest. The companies belong to different industries (e.g. insurance, consulting, software development, e-commerce). The data from these participants was used to analyze the quality of the questionnaire.

The statistical evaluations of the results of this pretest confirmed that we were on the correct path in our development and, so, could continue with further studies.

3.3 Evaluation in the Public Administration Sector

The development of the questionnaire aims to ensure that the questionnaire can be used in any user-specific context in the future. In order to evaluate the questionnaire in a broader context, it has already been adapted for use in public administrations.

The digital transformation impacts the way products are developed, leadings to an increased focus on project work in public administrations. In the context of public administration, user-centricity is a central aspect in the development of digital products because of the diversity of the target group. The agile paradigm supports a high degree of user-centricity through its focus on people and the product. The employees of the public administration increasingly show a readiness for agile practices and implicitly for topics of user involvement.

For the use of the questionnaire in public administrations in Germany, an expert survey was conducted with 26 experts in three iterations, with the aim of adapting the wording of the items to the user-specific context. Based on the expert survey conducted, it was shown that the questionnaire can be adapted to the user specific context. Furthermore, the dimension team-centered was also renamed self-organized. This renaming will now be adopted for the dimension in the following. The adaptations showed that the development of the questionnaire is a continuous learning process; further adaptations are necessary.

After adapting the questionnaire to the context of public administration, we were able to apply the questionnaire in a first case study to three teams of a public administration in Germany and successfully determine the current state of agility within these teams.

The results of the survey show a representative overview of the current state of agility at the team level. The case study also showed that due to team sizes between five and eight team members, the anonymity of the survey based on individual demographic questions could no longer be guaranteed. Therefore, the items of the demographic questions were reduced (see Sect. 4.2).

3.4 Expert Survey

During a further expert survey in two iterations, the items of the assessment questions of the questionnaire were reviewed again and linguistically adapted. In the first iteration, five experts were asked about the comprehensibility of the items. They were furthermore, given the task to assign the items to one or more of the six dimensions of agility, as we defined them (see Sect. 4.1). In this first iteration, four of the 28 items of the assessment questions were rated as not clearly understandable. Furthermore, the items were not clearly assigned to the dimensions. These results were carried over into a second iteration. Within the second iteration, the four items that were not clearly understandable could be adjusted in their wording in conversation with six agile experts. The experts judged the divergent assignment of the items to the dimensions to be dependent on the linguistic understanding of the items, the ambivalence of the items and the experience horizon of the experts from the first iteration. This divergent assignment shows the relevance of the need for further research.

4 Results

The following section presents our questionnaire to measure agility at team level. Owing to the adaptations already carried out, the questionnaire presented in this paper is a Version 2.0. Three groups of questions were defined: demographic questions, assessment questions, and weighting questions. The 36 defined items (see Sect. 3.1) were assigned to these three question groups.

4.1 Defining Six Dimensions of Agility

In order to develop a suitable questionnaire that takes agile values into account, we defined six dimensions of agility. For the definition of the dimensions, the agile values were compared with the traditional values. Based on the agile expression of the defined value pairs, the six dimensions communicative, change-affine, iterative, self-organized, product-driven, and improvement-oriented have been defined for the questionnaire. The construction was based on a mixed strategy with aspects of intuitive, rational, criteria-oriented, and factor-analytical construction. These dimensions may change in the future as we identify the need for them, based on future study results.

The meaning of the six dimensions for the agile approach - as distinguished from the plan-based approach in respect of the development of the questionnaire - is defined as follows:

Communicative

For the sequential processing of the plan, the plan-based approach defines a formal process that must be strictly followed in the implementation of the plan. In the agile transformation, this focus shifts. Here, the focus is on direct communication, both within the development team and with the customer.

Change-Affine

In the plan-based approach, a change in requirements represents an unforeseen deviation from the plan. This can only be integrated into the plan with great effort and, so, it results in a negative attitude towards changes. In the agile approach, the product and requirements are reviewed by the customer in several feedback cycles in order to avoid developing the product without taking the customer’s needs into account. Identified changes are seen as adding value to the product for the customer, resulting in an open attitude towards change.

Iterative

Since a change in requirements is not expected in the plan-based approach, the product is created through the linear processing of the initially defined plan. In contrast, the agile approach provides for iterative development with regular reassessment of open requirements. The requirements that have to be implemented in the iteration are processed by the team in accordance with the pull principle.

Self-organized

The plan-based approach is characterized by a strong hierarchy-centricity in respect of project management. A leading authority takes the decisions in the project and delegates the tasks to the development team. In the agile approach, the development team itself accepts a high degree of responsibility in the project. The team works in a self-organized way and can take decisions independently of a leading authority.

Product-Driven

The plan-based approach is characterized by a high degree of documentation. In the agile approach, less value is placed on documentation. The focus here is on the product to be created. Much of the documentation can be replaced by direct communication. The close cooperation, therefore, enables a stronger product centricity, with consideration of the customer’s benefit.

Improvement-Oriented

The strong orientation towards the project plan results in the plan-based approach in the fact that deviations from the plan must be answered for. The resultant apportionment of blame can be avoided in the agile approach through constant cooperation. Regular retrospectives are used to constantly try to improve the project approach in order to generate a product with high customer benefit.

4.2 Demographic questions

In order to capture the demographic context of the participants, five demographic questions were first constructed. The case study showed that because of the size of a team, anonymity in the survey results cannot be maintained (see Sect. 3.3). Based on these findings from the case study, the demographic questions were reduced to Item 1 and Item 2 (see Table 1).

Table 1. Demographic questions
Full size table

These demographic questions can be used to verify whether the self-assessment of competence and working methods correlates with the results of the survey.

In the questionnaire, the demographic questions are asked at the beginning of the survey, since they are a suitable introduction to the survey.

4.3 Assessment Questions

The 28 items selected by the experts in the construction process (see Sect. 3.1) were assigned to the previously defined dimensions (see Sect. 4.1), as shown in Table 2.

These items are used in the questionnaire to assess the agility of a team. The first column of the following tables defines a unique abbreviation for each item, which is used to reference the items.

Table 2. Assessment questions
Full size table

Except for the communicative dimension, which comprises eight items, all dimensions include four items. The items of each dimension form a psychometric scale. According to Döring and Bortz [20], a psychometric scale is a summary of items that together measures a characteristic of a complex construct. By forming a scale, the theoretical construct described by the dimension can be captured more precisely than by querying it with a single item [20].

The items are formulated as statements. For every statement the participant is asked to indicate the extent to which the behaviour required by the items apply to thinking and acting in his or her current developmental environment. According to Moosbrugger and Kelava [21], a discretely graded rating scale is suitable for answering this type of question. Regarding the scale points, a 7-point Likert scale was chosen. On the scale, the participant is asked to choose between the extremes ‘totally agree’ and ‘totally disagree’. The individual scale points are verbalized as follows: totally agree, agree, rather agree, neutral, rather disagree, disagree, totally disagree.

The scale has an odd number of scale points, which means that neutral middle category is present. The participant is, therefore, not forced to give a tendency, as would be the case with a scale with an even number of scale points [19]. With an uneven number of scale points, however, there is a risk that the mean category will not be used exclusively in the sense of the mean characteristic expression, but also if comprehension problems occur or the answer is refused. This mixture of response behaviour, known as confounding, is to be relativized by the additional option of no response on the response scale [21].

Regarding scale points, a number of five or seven is suggested in the literature for odd scales [19, 22]. A fewer number of scale points leads to a loss of information, as the participant cannot give the answer with sufficient differentiation [19]. More scale points can lead to cognitive overload, as the participant is no longer able to differentiate meaningfully between the response options [21]. Based on the results of Finstad [22], it was decided to use the higher differentiability of a 7-point scale.

4.4 Weighting Questions

In order to fulfil the objective of taking the user-specific context into account, the questionnaire was supplemented with weighting questions. For each of the six dimensions of the questionnaire, an additional item was defined that asks about the importance of the dimension for the user. Accordingly, the items of the scales can be weighted in a manner specific to the context. For the formulation of the weighting questions, a concise description of the behaviour of the team required by the dimension was chosen.

The questions are listed in Table 3. As with the assessment questions, a unique abbreviation is given.

Table 3. Weighting questions
Full size table

Analogous to answering the assessment questions, the weighting questions are rated on a 7-point Likert scale. In contrast to the assessment questions, however, the questions do not ask for agreement but for importance. For this reason, the naming of the scale points for the weighting questions is chosen as follows: particularly important, important, rather important, neutral, rather unimportant, unimportant, particularly unimportant. To preclude any confusion, the scale was also supplemented here with the option of no answer.

We have learned in a case study that it is valuable to have an introductory workshop before carrying out the questionnaire. On the one hand, the team can build a shared understanding as regards the objectives of the questionnaire and to define common goals among the team. On the other hand, it is helpful in terms of reducing bias during the questionnaire study, because the dimensions can be discussed in order to build a shared understanding regarding the wording and meaning.

5 Discussion and Limitations

In the previous sections, we presented our questionnaire for measuring agility at the team level as well as the construction process of our questionnaire. In the following, we want to discuss the strengths and weaknesses of the questionnaire.

During our journey to create our questionnaire, we conducted several studies (see Sects. 3.2, 3.3 and 3.4) and discussed it with many experts. We learned that our questionnaire is able to assess the agile values with the assistance of the six dimensions (communicative, change-affine, iterative, self-organized, product-driven, improvement-oriented: see Sect. 4.1). Moreover, the additional weighting questions allow us to include the respective context of the respondent as well as define the objective of the survey. In our case study, we learned that we can analyze how the current state of agility is perceived. For example, one study showed that a team has very different perceptions in respect of the understanding of agility and agile working. The difference in these perceptions in cooperating teams leads to misconceptions about the way of work. Such a work environment can lead to a breakdown of trust and commitment between teams, which furthermore leads to dysfunctionalities according to Lencioni [23].

We also found that the implementation of the survey must be accompanied by an expert. The implementation of the questionnaire should be carried out in a moderated fashion. On the one hand, it is important to reach a shared understanding of the objective. On the other hand, it is important to interpret the results correctly and derive measures based on the measurement. Furthermore, the moderated implementation of the questionnaire causes a stronger analysis of the contents and, so, leads to a reflection of one’s own work as well as the work in the team. For this reason, we are currently working on a process model for the application of the questionnaire [24, 25]. In summary, this means that practitioners and companies can use the questionnaire to optimize collaboration within their teams. Optimizing collaboration has a positive effect on the value delivery of the team and, hence, on the success of a company.

In addition, our questionnaire for measuring agility has some limitations. The expert survey has shown that the items of the assessment questions (see Sect. 4.3) cannot be clearly assigned to the defined dimensions. This ambiguous assignment of the items to the defined dimensions can be caused by the different kinds of linguistic understanding of the items, the environment, the experts’ horizon of experience, or also the ambiguity of the items. The expert surveys showed that adjustments with regard to the linguistic formulation were repeatedly necessary. We are currently in the process of conducting further studies, in which we will investigate the assignment of the items to the dimensions by means of statistical analysis.

6 Conclusion and Future Work

This paper presented an overview of the current state of research on the development of a questionnaire to measure agility at the team level. Our questionnaire for measuring agility includes 36 items, assigned to these three question groups (2 demographic questions, 28 assessment questions, and 6 weighting questions). In order to develop a suitable questionnaire that takes the agile values into account, we used the six dimensions communicative, change-affine, iterative, self-organized, product-driven, and improvement-oriented.

Firstly, we provided an overview of the related work. We identified and analyzed seven maturity models and two questionnaires for measuring agility based on a literature review. In addition, the research method for the development of the questionnaire was specified.

We presented our research method, including the initial construction of the questionnaire, a pretest, a case study, and further expert surveys.

By conducting the expert surveys, an optimization in the wording of the developed items of the assessment questions was already achieved. The pretest and the case study already show that the questionnaire is suitable for measuring agility at the team level and that the current state of agility can be captured. Furthermore, the case study showed that the results of the questionnaire can be used to identify dysfunctionalities in teams.

In future work, we want to create a standardized tool for supporting the agile transformation of an organization. We are already conducting further studies on this in Spain. On the one hand, the aim is to develop a standardized questionnaire for measuring agility. Conducting the expert survey and the case study shows that the development of this instrument is a continuous learning process. Further research is required to develop this instrument into a standard. On the other hand, we want to elaborate on our process model for using this questionnaire within the agile transformation.