Keywords

1 Introduction

Transformation from a traditional software engineering process to an agile process is not well understood. This area of agile transformation is relatively new, and researchers are working on different aspects to gain new understanding. Industry organizations that transformed from waterfall to agile implemented multiple strategies such as altering quality assurance practices [1, 2] and training staff in agile methods [3, 4]. However, organizations must consider additional factors when scaling such transformations.

In this paper, we present a tertiary study in scaled agile transformations. Our focus is on large projects defined by the number of people working on the project team. We are not aware of another tertiary study specific to scaled agile transformation. This study identifies gaps in the literature of scaled agile transformation that will help the community to identify potential research avenues in future.

We could not identify tertiary studies that present a meta-analysis of relevant secondary studies. Additionally, we observed there are not many secondary studies compared with more mature research areas. The goal for this study is to collate instead of synthesize; in this way we hope to identify gaps as opportunities for further study and understanding. As a relatively new area, researchers have been trying to answer different questions related to the scaled agile transformation. The contribution of this paper is to assemble evolving early large-scale agile transformation evidence.

The main goal of this tertiary study is to synthesize the research goal and findings of peer-reviewed secondary studies in scaled agile transformation. Under this goal, we addressed two specific questions:

  • RQ1: What success factors and challenges in scaled agile transforms have been identified by prior secondary studies?

  • RQ2: What gaps exist in existing studies that should be prioritized by the research community moving forward?

2 Research Methodology

We use [5] and [6] as guidelines for performing this tertiary study. The research methodology was primarily conducted by the first author and reviewed by the second author.

2.1 Search Process

The steps of the search process are shown in Fig. 1 and summarized as follows:

  1. 1.

    Digital libraries from IEEE (29), ACM (12), SpringerLink (35), and ScienceDirect (46) were searched for journal, conference, and workshop papers.

  2. 2.

    Duplicate studies were removed manually.

  3. 3.

    For a tertiary study, we only consider secondary studies for analysis. Specifically, only systematic literature reviews (SLRs) were included beyond this step.

  4. 4.

    The abstracts from step 3 were read. Reviewers searched for keywords and for verification that the study was an SLR on the topic of scaled agile transformation. Manual review was necessary as an initial keyword scan on terms such as ‘agile’, ‘large’, ‘scale’, ‘change’ OR ‘transformations’ resulted in too many false positives.

  5. 5.

    The final set of papers was determined by a full-text manual review of the studies from step 4, conducted by both authors, focusing on research questions and paper quality. Paper quality review may be subjective [5]; we considered the publication venue, date of publication, citations of the paper, and citation indices of the venue and authors. Further, we also considered whether enough primary studies were included in the secondary study to validate claims of significance. Disagreements between the authors on inclusion or exclusion were resolved through discussion.

Fig. 1.
figure 1

Search process for identifying papers to include in the tertiary study.

Full size image
Table 1. Final set of papers included in the study.
Full size table

2.2 Summary of Included Studies

Agile adoption has rapidly increased over the past two decades. A few preliminary studies and calls for more research in scaled agile transformation started appearing in the 2000s as practitioners in large organizations are moved towards large-scale agile adoption (cf. [7,8,9,10]). Scholarly study of this phenomenon largely started appearing since 2015. Given the relatively small number of studies that met our inclusion criteria, we summarize these papers here.

Dikert, Paasivaara & Lassenius [12] present an extensive secondary study of 52 primary studies on the challenges and success factors of agile transformation. The study reported 35 challenges classified into 9 categories and 29 success factors classified into 11 categories. As per the authors, the most important success factors were management support, choosing the right agile model, mindset, and alignment with the organization’s value. The challenges included resistance to change, lack of training, and misunderstanding agile. This study is very influential and a widely referenced work, however, all of the primary studies used for evidence were from 2010 and earlier, and almost all describe transformation from a waterfall-like process to an agile process.

Uludag et al. [13] conducted a structured literature review of 73 papers related to the challenges of scaling agile from a stakeholder perspective. The study reported 79 challenges in 11 categories. The top 3 challenges were coordinating multiple teams working on the same project, considering dependencies in integration, and coordination among geographically distributed teams. This recent study reports new challenges in scaled agile transformations. Notably, it claims the majority of challenges in large-scale agile development (38 of 79) still exist and are “typical”. The stakeholder perspective focuses on development team roles, with only a few higher-level roles included.

An action research approach to a meta-review was taken by Kalenda, M., Hyna, P., & Rossi, B [11]. The authors identified 8 common features of scaling frameworks SAFe and LeSS [10] and used this to drive a focused literature review. 12 papers were selected, 10 of which mapped to at least one of the 8 common features, and then described challenges and success factors of each paper. The paper is influential due to its recency and use of scaling frameworks (whose awareness and adoption are becoming more prevalent in industry) as an organizing principle. However, the authors acknowledge this approach is not comprehensive, so there may be more evidence in the literature not included in the study. We also wonder if the approach of identifying common practices between SAFe and LeSS is appropriate, due to scaling agile not being a prescriptive formulaic process, and also because multiple frameworks were excluded (notably DaD [10] and SoS [10]). Nevertheless, this paper is highly influential and an ambitious action research for understanding scaled agile transformation.

Klünder et al. [14] answer 4 questions on large-scale agile transformations. On the first question ‘Does any transformation model for large companies exist that-in particular-preserves already existing SPLs?’ The authors state that no model can be used to transform large organizations into agile. The second question ‘What preconditions should be met before starting transformation toward in a large company?’ indicates success factors reported by studies which are the same as [11], [12], and [17] such as management commitment, training, knowledge, and one additional precondition that is risk planning. Question 3 ‘What tasks are recommended to be fulfilled during the agile transformation on development team and management level?’ reports that the distribution of tasks and setting up the infrastructure are key steps that need to be done by development teams and management during the transformation phase. Question 4 ‘What tasks are required on organizational level to finalize the transformation in a large company?’ reports that management should start the transformation with a pilot team so they can get feedback to improve the pilot team’s transformation and also other teams.

A different set of challenges related to scaling agile software development has been reported by Ozkan & Tarhan [15]. The study reports physical dependencies, fragmentation, feudalism, narrow focus on product, construction, and bottlenecks from one to many. This paper is relatively new, but reported some challenges that we could not find any other studies we analyzed.

Shameem et al. [16] reports success factors for large scaled agile projects. The authors report a set of 15 success factors grouped into six categories. The paper also classified these into two major categories, client and vendor. This provides a broader picture of agile processes and factors related to its respective success factors.

Saeeda et al. [17] reports that 24% of the research states that documentation is one of the limitations of agile, 22% reported time period issues and 14% talks about budget overflow. It also reports that 33% of studies report communication as a challenge and 25% report distributed teams as a challenge. The authors also report that researchers are working to find the limitations of agile scalability and its remedial ways.

2.3 Data Extraction

We extracted detailed information from the 7 studies in Table 1 including research goals and questions, findings, discussion, and limitations and reviewed this information for our analysis. Our analysis focused on the success factors and/or challenges in scaled agile transformation presented in each paper, though we note this was not always the primary focus of every study. For example, study [14] presents literature review findings from primary studies a bit differently; this paper identifies preconditions and tasks from the primary studies that exist for large-scale agile transformation to be successful. We mapped preconditions and tasks to success factors to facilitate analysis of these studies. Tables 2 and 3 below shows 23 different challenges and 22 success factors reported by the 7 studies from Table 1.

Table 2. Challenges in Scaled Agile Transformations Reported by Prior Studies.
Full size table
Table 3. Success Factors in Scaled Agile Transformations Reported by Prior Studies
Full size table

We manually combined similar ideas with different verbiage into one for the purpose of this study. For example, ‘Change Resistance’ from [12] and ‘Dealing with doubts in people about changes’ [13] have been combined as ‘Resistance to change’.

2.4 Limitations

There are challenges in conducting a tertiary review in a topic as recent and fluid as scaled agile transformation. First, there is a lack of general agreement on what is scale. The term can describe the size of an organization, the size of software projects, the breadth of application and system domains, or the range of organizational roles participating in the transformation. Second, the recency of industry adoption and published research presented a challenge both for identifying relevant literature and for applying a systematic process for analysis. Admittedly, we had to soften our inclusion criteria somewhat to identify even the small number of recent studies due to this limitation. Analyzing the papers from a common perspective was also difficult as often the studies focused on different aspects. For example, [13] focused on stakeholder perspectives, [11] started with scaled agile frameworks (SAFe and LeSS), [14] focused on software product line engineering, and [15] focused on a design perspective. Therefore, our identification of common success and/or challenge factors is bounded by the perspectives of the included secondary studies. Finally, this study was conducted by one Ph.D. student as the primary researcher, and a single secondary researcher. In a situation where the research is sparse and there is an above average reliance on subjective interpretation due to the subject matter, a third researcher may have improved the arbitration process.

3 Analysis and Discussion

In this section we present the answers to the research questions.

  • RQ1: What success factors and challenges in scaled agile transforms have been identified by prior secondary studies?

From Tables 2 and 3 we see that 5 out 7 studies identified challenges while 4 out 7 studies presented success factors in scaled agile transformations. Two studies [11] and [17] reported transformation frameworks and limitations. We could find only one study [14] that suggests organization support is a key factor in scaled agile transformation.

Figure 2 (left) shows the coverage of challenges by different studies as listed in Table 2. In this format, we can see there is not widespread agreement on the challenges, though the few that are agreed upon include coordination, employee mindset, management support, resistance to change and quality assurance.

Fig. 2.
figure 2

Challenges (left) and Success Factors (right) reported by previous studies

Full size image

Figure 2 right shows the coverage of success factors by different studies listed in Table 3. There is somewhat more agreement in factors here as compared to challenges, though again there are still some (such as training, coordination, training and knowledge) that are emphasized in most all, if not all, studies.

  • RQ2: What gaps exist in current studies that should be prioritized by the research community moving forward?

The answer to this question will shed light on the future research avenues. Scaled agile transformation is a relatively new area so we could only identify a few relevant studies. Most of the studies focus on challenges and success factors of scaled agile transformation. Given the scarcity of literature and the answers from RQ1 these are the research gaps we identified that the community needs to address to move forward.

There are some challenges to scale agile projects that have been reported by a significant number of studies (Table 2). However, there are still many challenges that appeared only in one of the seven studies in Table 1 (CH7, CH8, CH10, CH11, CH13–CH22). Success factors are similar; some were reported by multiple studies (Table 3) while others (SF8–SF11, SF13, SF14, SF16–SF22) were reported by only one study.

In our opinion the identification of challenges and success factors by these studies offer guidance to real-world practitioners and identify areas for future research. Further research is needed to identify common perspectives as more software engineering organizations go from “agile-in-the-small” to “agile-in-the-large” transformation. The ultimate goal is to coalesce understanding into a reference framework for practitioners such a machine learning or statistical model. The goal of the framework would be to help practitioners to make decisions during scaling agile. These are the possible avenues that we think need to be explored.