Keywords

1 Introduction

More and more large organizations are now adopting agile enterprise-wide. However, the scaling of agile to a larger undertaking, creates an entire new set of problems and challenges ranging from resistance to change to the problems associated with hierarchical management and organizational boundaries. Leffingwell [1] groups the challenges of scaling agile into two broad categories: (1) those inherent to the methods themselves, “because of the fixed rule bases and assumptions built into the methods” (p. 87); and (2) challenges imposed by the enterprise that “will prevent the successful application of the new methods” (p. 87). More recently, Dikert, Paasivaara and Lassenius [2] surveyed 52 publications describing 42 industrial cases and reported 35 challenges grouped in nine categories: change resistance, lack of investment, agile difficult to implement, coordination challenged in multi-team environment, different approaches emerge in a multi-team environment, hierarchical management and organizational boundaries, requirement engineering challenges, quality assurance challenges, integrating non-development functions. Surprisingly, very little empirical research has been done on how to alleviate these challenges [3].

However, consultancy firms have developed various frameworks and models to address these challenges to implement scaled agile: Disciplined Agile Framework [4], Large Scale Scrum (LeSS) [5] and Nexus [6]. The most popular framework for scaling agile remains SAFe [6,7,8].

All these frameworks are building on the values and principles of the agile manifesto [9], among others the deployment and support of self-organized teams. Although much has been researched on teams, particularly in the field of psychology, much remains to be investigated in the context of agile deployment. The objective of this paper is therefore to contribute to the field, based on a research question proposed, during last year’s workshop on scaling agile at XP 2018 [10]: “What is the right degree of team autonomy in different contexts (and how to measure it)?”

2 Literature Review

Parker, Holesgrove, and Pathak [11] define a self-organized team “as a self-regulated, semi-autonomous small group of employees whose members determine, plan and manage their day-to-day activities and duties under reduced or no supervision.” (p. 112) and the labels “autonomous teams” have been used as synonyms for “self-organizing teams,” “self-managing teams” and “empowered teams” [10].

Teams have been studied for decades, primarily in the field of psychology. Literature abounds on numerous topics such as: leadership, roles, phases, performance, team dynamics, etc. More and more researchers reuse some of that literature to study software development teams using agile approaches, for example to relate autonomy to team performance [12], empowerment and outcomes in software development organizations [13], productivity of self-organized teams [11] and job satisfaction and/or motivation [14].

However, according to Kakar [15] “no approach or instrument has yet been designed to measure and compare self-organization between teams. Second, although conceptually the difference in adoption of self-organization in agile versus plan-driven methods has been discussed previously in the literature, the levels of self-organization between these two major paradigms of software development have not been objectively compared” (p. 208). In the case presented in this paper, the issue was the assessment of the teams and the level of authority delegated to them by the senior management. This is what Cao, Mohan, Xu and Ramesh classified as “sources of structure” in their proposed framework.

3 Methodology

The development and deployment of a new model to assess teams in order to grant them the right level of autonomy to release software was observed in a large South African bank’s IT department. Twenty employees in the department were interviewed. The interviews were conducted by the two researchers themselves. The twenty interviewees comprised of four people from business who were direct customers of IT and ultimately the agile process, two people from the agile portfolio office, four portfolio project managers, the CFO of the IT department, two CIOs within the IT department, three release train engineers, one COO and three agile coaches.

Most of the interviews were directed towards understanding the link between the IT initiatives and the organization’s strategy. However, during the course of the research many interviewees kept referring to an internal program to assess a team’s autonomy with respect to the deployment of software. There were two interviews with the people responsible for the deployment and improvement of engineering practices. The interviews were coded in ATLAS.ti to assist in the analysis and summary of the approach. The objective of this paper is primarily to present and share an experience report from the perspective of the bank with the understanding that their approach might be of interest to other organizations.

4 Case Description

SA Bank (Note: This is a fictitious name to protect the identity of the bank), a financial institution, is one of the largest African financial institutions by assets. SA Bank is among the largest organizations in South Africa by market capitalization. It offers a wide range of banking and financial services to millions of personal customers in 20 African countries. The company employs over 6000 people in their IT.

SA Bank decided to start deploying agile practices in their IT department. They redesigned their software development completely and restructured around self-organizing teams responsible for products rather than specific activities. These teams would also be in close contact with business representatives and would be entirely responsible for the development, testing, deployment, and maintenance of their products. The intention was to reduce the number of handoffs and simplify the development process dramatically. The SAFe deployment was strongly committed to and supported by executives and line managers across the board.

At the time of the interviews, SA Bank had deployed SAFe 4.5 but used only three of the four layers [8, 16]: They did not use the “large solution” layer but had the intention to deploy it in coming months.

Semi-permanent self-organizing agile development teams were put in place to support the planning, prioritization, harmonization, synchronization, development and release of the features/systems. The term “semi-permanent” is used to refer to teams being maintained as permanent as possible with the exception of changes related to turnover, punctual member swaps due to competence requirements, etc. The agile teams use many of the Scrum tools and artefacts such as backlogs, burn-down charts and Kanban walls.

5 Background Leading to the Introduction of Earn Your Wings

An incident in March 2017 triggered the introduction of the “Earn Your Wings” program. The teams were used to systematically obtain authorization to release software from a committee, composed of senior managers, called the change advisory board (CAB). One of the teams wanted to deploy a change into production. The changes would have been deployed using a fully automated pipeline and the team was convinced that the software would not be faulted. That release happened to occur in a high-care period. This meant that the team required authorizations from the production manager, from the business responsible, then from the area senior management head sign-off and then finally a sign-off from the CAB.

One of the people responsible for process improvement and SAFe deployment started to challenge the governance and approval process. How could they be talking about team autonomy and team self-organization while at the same time imposing a very cumbersome approval process?

After a number of focus groups, interviews and surveys, they realized that this old approval process actually had three negative side effects (what they called reverse incentive behaviors):

  1. 1.

    The CAB was shielding people from consequence management. If a team was deploying a change into production, so long as they received CAB sign-off, they felt relieved from taking responsibility if anything went wrong.

  2. 2.

    The second reverse incentive was low-low changes. Only low impact and low risk changes were self-governed. If changes were categorized as low-low, the teams did not have to go to CAB. Consequently, more and more teams were categorizing everything as low-low with the risk of deploying faulty high-risk/high-impact packages.

  3. 3.

    If the approval process is tedious, teams tend to package the release in larger bundles and decrease the release frequency

6 Earn Your Wings

In order to reverse these negative side effects, the team in charge of improving the deployment process at SA Bank derived a detailed evaluating and rating scheme. This section summarizes what the bank representatives presented to the researchers. They used the theme of a pilot’s ability to fly an aircraft using five levels, from lowest to highest:

  • Level 1 Red Bull: This is the lowest level for which the highest level of authorization is required. This comes from the image that “Red Bull gives you wings” but actually you are not really able to fly. In other words, the team (and the supported software) is at the lowest level of autonomy.

  • Level 2 Hot Air Balloon: Although you can get into the air, you are reliant on fire, which can easily go up in flames. You are restricted by conditions as to whether it can take off or not. You carry very little safety gear.

  • Level 3 Small helicopter: You have freedom of movement but with extra caution. Individually you decide if this is trusted and if the risk is worth taking. Although you can get somewhere, with relative ease, you do not have the best safety gear if something goes wrong

  • Level 4 Dinky Plane: While you have the freedom to go wherever you want, there are situations in which you would choose not to fly. A trusted form of transport, with a little extra convincing

  • Level 5 Private Plane (highest): Freedom to go wherever you want, whenever you want. The most trusted form of transport. Air traffic control simply coordinates but does not question.

The level is determined based on Team Fly-ability and Elevation Safety described in detail in this section.

6.1 Team Fly-Ability

Team fly-ability includes two elements “Maturity of engineering practices” (60%) and “Ability to manage traceability and risk through ease of recovery” (40%).

Maturity of Engineering Practice

The teams subjectively assess their own level of Maturity of Engineering Practices based on: the level of automation, the level of testing, the size of work package and an assessment to whether acceptance criteria are met. In practice, this is performed through a tool called Continuum using elements such as coding practices, continuous integration, incident management, release management, quality assurance and risk management. If teams decide to overrate themselves, they take on the risk of something going wrong.

Ease of Recovery

The ease of recovery evaluates the ability to manage traceability and risk, the time to recover and the ability to roll back and restore. This is assessed, using a tool called Remedy, based on the Mean Time to Recover. The objective is to maintain the down time under the agreed levels contracted in the Service Level Agreements.

6.2 Elevation Safety

Elevation Safety is based 100% on historic data on deployment performance and severity of incidents with some consideration of the application dynamics and criticality being taken into account.

Historic Data on Deployment Performance and Severity of Incidents

This is based on the performance of the previous five deployments answering questions such as: Have you caused incidents within the two weeks after your deployment due to the change? Have you had an avoidable production incident? What was the frequency of deployments?

Application Dynamics and Criticality

This is based on the chief information officer (CIO)’s list of the most critical applications with respect to: number of users, rate of change, number of dependent systems, the number of countries impacted.

6.3 Overall Assessment on a Scale of Five Levels

Using the combination of team fly-ability and elevation safety described in the previous sections, the overall level of autonomy (from Red Bull to Private Plane) is assessed using the grid in Fig. 1.

Fig. 1.
figure 1

Overall assessment of the team autonomy

Full size image

The wings are assigned to teams, not individuals. The teams are continually assessed using real-time data (for example on the quality of their prior releases).

6.4 Authorizations Required to Deploy

The release and deployment authorizations required, depends on the level of the team as follows. This varies from required authorization of the CAB for all changes, for Level 1 (Red Bull), to complete autonomy for releasing and deployment at Level 5 (Private plane). In the case of level 5, the CAB would just be informed and would not be involved in the decision. There are various degrees of authorizations required for intermediate team level medium or above. Depending on the level, the release and deployment constraints are specified e.g. ability to release during the freeze periods, time of the day when deployment is allowed, frequency of deployment (per week), artefacts required.

7 Conclusion

At the time of the interviews, 70 teams had “earned their wings.” The objective was to deploy to approximately 300 teams. Consequently, the organization was working on the improvement of engineering practices to support teams. They also continued to automate data collection to support team evaluation. As far as the SA Bank was concerned, the main benefits of this program had been: (1) Improved accountability of teams (2) reduced duration of the CAB (3) reduced attempts to find workarounds and loopholes and (4) coordination done at the right level i.e. teams govern each other as opposed to management doing it. Release management does not question the releases; they just make sure that there is no mid-air collisions (like air traffic controllers).

Although this research is based on a limited set of interviews, the authors wanted to share how SA bank had used an objective assessment of the autonomy and maturity of the teams as a good example of how an organization tried to answer the research question, “What is the right degree of team autonomy in different contexts (and how to measure it)?” Their objective, in doing so, was to improve the deployment process both in terms of quality and the time required to approve the releases.