Keywords

1 Introduction

The government utilizes ICT for several purposes. Naturally, ICT helps in improving the efficiency and effectiveness of executing the internal primary processes within a government organization. For that purpose, government organizations develop their own ICT facilities for their primary and supporting processes, or they purchase those facilities externally. In addition, ICT is essential for a government organization’s contact with other government organizations, businesses and citizens. Finally, ICT often connects the interaction of the outside world with the digitalized processes within the government organization. Therefore, ICT is the enabler of e-Government in all three of these previously mentioned domains [7, 30].

E-Government is referred to as the technology-enabled transformation of government, and therefore governments’ best hope to reduce costs, whilst promoting economic development, increasing transparency in government, improving service delivery and public administration, and facilitating the advancement of an information society [35]. The purpose of e-Government is thus to work towards a more effective and more efficient government organization by utilizing new technologies and by involving citizens and businesses and, in doing so, listening to their needs [8, 20].

Citizens are therefore ‘agents’, so to speak, who, together with government representatives, and even together with business representatives, shape the services and processes of the government. This is in line with the principles of New Public Governance (NPG), in which the government is seen as part of a large and synchronized network of parties within society as a whole. It is essential that the interface between society and the total government system works properly, both for the political and the administrative system [20].

When it comes to e-Government, people often refer to e-Participation and, in particular, the various possibilities that this new technology offers [24, 30]. Macintosh [12] argues that e-Participation has several overarching objectives next to these technologies, including engaging a wider audience, and providing relevant information in a format that is both more accessible and more understandable. However, transparency and accountability do not receive as much attention [13]. This article specifically focuses on these issues. In addition to the involvement of citizens in making demands and creating IT facilities for the government, citizens also take on the additional role of monitoring the development process and therefore the quality of the facilities.

The question answered in this article is whether citizens are being sufficiently involved in monitoring the quality of the production processes and the implementation of facilities for e-Government. This would mean that citizens are involved during quality tests and especially during evaluations.

The involvement of citizens is relevant because society is setting increasingly high standards for the government regarding transparency, effectiveness, efficiency, possible influence and services provided [27]. For example, citizens want to know what information about them has been recorded by the government, and they want to know where ‘their’ tax money went [21]. People want this information to be easily accessible, preferably on a variety of mobile devices [9]. In addition, citizens also want to cooperate and offer their input for the solutions they are getting, an ambition that is in line with the public administrative theory of New Public Governance (NPG).

Safeguarding quality is relevant, because projects used to create IT facilities and the corresponding processes fail on a regular basis. Audits, quality checks and evaluations are subsequently used to see what can be learned from the projects, as well as what went wrong. As a result, a great deal of research has been done into success and fail factors of such ICT projects for both the public and private sectors [28]. These studies indicate that only a third of projects result in the desired end products within the specified time frame and within the budget. A third of the projects is terminated prematurely, and half of all projects end up costing nearly twice as much as was initially projected.

As a result of the disappointing performance of ICT projects within the government, governments focus more on monitoring by third parties. This corresponds to the attention for additional evaluations and audits, which Power and Clarke call the ‘audit explosion’ within the public domain [3, 22].

However, evaluations of ICT projects and the methods used for those evaluations are often lacking in quality. Nijland [15] states that evaluation methods are lacking in terms of management of costs and benefits of ICT. In the study into aspects of ICT projects within the Dutch government, it turns out that the most important learning experiences are not found in ICT but in related disciplines, in the processes of the organization itself, and in the effect of the organization on its environment – in other words: the business processes and citizens [36].

2 Theoretical Background

In order to better indicate the role of the citizens, the public administrative approach of New Public Governance (NPG) will prove useful. New Public Governance (NPG) is connected to the pluralistic and fragmented complexity of the twenty-first century [16]. While its predecessor, New Public Service (NPS), focuses more on the civilian’s role and the optimal performance of the government organization, NPG focuses more on managing the government organization’s environment. Through NPG, government managers are asked to manage while directing their attention outside of the organization, and to act in inter-organizational environments.

In addition to collaborating with citizens and offering services to citizens, the focus of NPG is also on negotiation about the added value and the management of networks and mutual relationships. This is in line with the observation that government organizations are starting to turn into network organizations within government organization chains [7], away from the vision that governments should treat citizens more like their customers, and focus on efficiency, effectiveness and economy as the New Public Management used to focus on [10].

The governmental chains exist in order to provide added value to citizens and businesses through products and services [25, 32]. E-Government can facilitate the process by providing full participation by citizens and businesses regarding both the creation and evaluation of the required facilities.

In this study, evaluation reports of IT-projects have been studied, in order to find evidence of citizen involvement in these evaluation. For the definition of evaluation, we chose the definition used by Stufflebeam and Shinkfield [29]: “Evaluation is the systematic assessment of an object’s merit, worth, probity, feasibility, safety, significance and/or equity.” The core of this definition in its operationalization is the process-oriented, systematic research approach to evaluation. The definition used by Stufflebeam and Shinkfield [29] mentions a ‘systematic assessment’, exactly in the same way that Scriven [26] talks about evaluation as “…the process of determining the merit, worth and value of things, and evaluations are the products of that process.”

The definitions describe a logical phasing of a number of steps in performing evaluations. In other words, an evaluation with a systematic research approach will follow steps like defining an objective, a problem and a research question, the construction of the theoretical framework, and subsequently defining a type of study, e.g. a descriptive, exploratory or evaluative study. Based on that, a choice will be made as to what population will be studied, and what method of information gathering and data analysis will be used. The final steps of such a study are the report and a phase of evaluation, recommendation and formulating follow-up steps [6].

Because evaluation research is a form of practical research, and because it usually involves a complex problem, an evaluation study will be conducted using an intervention cycle [34]. Such a cycle for evaluation consists of the following steps:

  • A motivation that leads to performing an evaluation, and from which several concerns for the evaluation arise.

  • The decision to perform an evaluation. The starting point for the evaluation is the moment the originator decides to have an evaluation performed.

  • The formulation and scope of the evaluation. In doing so, the evaluator lays the foundation for their evaluation study. In the formulation phase, the criteria and substantive requirements to be used as preconditions for the evaluation are formulated as well [23].

  • Choosing the evaluation approach, evaluation type and evaluation techniques. These are the choices the evaluator makes regarding the research method and approach from a scientific framework.

  • Performing the evaluation, using the research and evaluation framework.

  • Finalizing the evaluation. This step focuses on the importance of the end result, the way in which for example the report is handed to the originator of the evaluation, as well as the characteristics of the report [23].

  • Monitoring the evaluation process as a whole, an activity that is performed during all steps of the evaluation. There is a particular focus on choosing the moment of evaluation and the (role of the) evaluator. This monitoring activity also initiates iterations in the process and the learning circles for the evaluator.

In an overview:

Every process step comes with iterations during its execution, and iteration may take place after every step leading back to the previous step. The iterations may relate to continuous insight into the evaluation itself (‘single loop’), or to the norms, criteria and preconditions within which the evaluation takes place (‘double loop’), or to the conditions from the context that affect the evaluation (‘triple loop’). With every iteration of this process, the evaluator will consider whether a subsequent step is essential and what changes need to take place.

The process of evaluation has nothing to do with the moment of evaluation: evaluations can take place both ex ante and ex post. The evaluation process can be used at any given time and has its own duration and dynamics. This process is independent from the ICT project being evaluated.

3 Research Method

A meta-evaluation

Citizen involvement is one of the aspects in evaluations of projects with an ICT component within the government. In order to study the overall quality of these evaluations, the evaluation reports of these projects have been studied. Obviously, not everything of value is put in evaluation reports. However, these reports give an indication of the evaluation as carried out. These reports are available upon request and can be studied based on a perceptual framework.

This approach can be more easily repeated as well, making additional and follow-up studies easier to carry out. By studying the evaluation reports, they are effectively subjected to a meta-evaluation. The meta-evaluation features two underlying levels for assessing evaluations:

  1. 1.

    Assessment of the process of the performed evaluation. Examples include the nature of the decision-making process surrounding an evaluation, the assignment description for the evaluation, the design of the evaluation study and the report.

  2. 2.

    Assessing whether the project evaluations meet content quality standards, e.g.:

    1. (a)

      Aspects in the process itself, the product resulting from a process, and the way in which people within the ICT project act and collaborate.

    2. (b)

      Assessing whether enough attention was paid to the relationship between the project and its environment during the evaluation.

In the study presented here, evaluation reports were requested from government organizations that apply ICT in their organizational processes and services: departments, government organizations that act at the federal level, and large municipalities. The criteria for the selection of the reports for this study are (a) that the reports are publicly available and (b) that the selected reports are about projects with an impact on computerized processes that took place between 2002 and 2010.

The population was determined based on two basic pieces of information. Firstly, the progress reports of large government projects sent to Parliament were studied. These include quality tests and evaluations per department and per project, listed by name. Secondly, an estimation of the possible number of projects was made for non-departmental government organizations and contracting organizations and municipalities, since a population for those could not be determined.

The requesting of these reports was based on a stratified random sample, after which the selected projects were requested. In case only the organizations were known, and not the reports or the projects, we approached the organizations. In these requests, we specifically stated that all reports will be treated confidentially, and that research results cannot be traced back to the report, the project or the organization. In total, this approach resulted in 88 evaluation reports for research purposes.

Assessment framework

In order to study the evaluation reports, specific criteria have been defined for every phase of the defined process (Fig. 1), derived from (a) literature about evaluation, and (b) literature of four related disciplines: (1) administrative and organizational science, (2) behavioural science, (3) accountancy and (IT) auditing, and (4) quality management.

Fig. 1.
figure 1

Evaluation as an iterative process

Full size image

Taken together, these criteria form the foundation of an extensive perceptual framework, consisting of 21 aspects with a variety of focus points. A detailed description of the framework can be found in: Bronsgeest [5]. Some of these focus points are about the extent to which citizens are involved when it comes to safeguarding the quality of the ICT-facilities during their creation.

Content analysis

In order to analyse the selected reports, we carried out a document analysis, using content analysis techniques. Document analysis is “…the techniques used to categorise, investigate, interpret and identify the limitations of physical sources, most commonly written documents whether in the private or public domain” [19]. Document analysis is often seen as a valuable addition to information obtained from interviews, for example, especially when talking about social studies: “In policy research, almost all likely sources of information, data, and ideas fall into two general types: documents and people” [1]. Document analysis can be carried out in multiple ways, namely through analytical reading, content analysis and quantitative analysis [4]. All three of these techniques were used within this study.

Document analysis as applied within this study employs content analysis techniques. With content analysis, the source or text is viewed as a piece of information for drawing conclusions about the meaning of these sources or text within a certain context. Berg [2] uses the following definition of content analysis, derived from Holsti: “Content analysis is any technique for making inferences by systematically and objectively identifying special characteristics of messages.” According to Berg, this qualitative definition is broadly applicable on all kinds of written material, spoken material and visual material. Neuendorf [14] uses the term ‘content analysis’ and adds that the systematic and objective analysis is also quantitative. An objective analysis then becomes possible by determining explicit rules or criteria for the selection of these messages or research subjects before analysing the data [11]. When it comes to these definitions, it is essential for an analysis to be carried out using fixed procedures and methods, and for the approach to be repeatable.

In document analysis and content analysis, there is a methodological difference between manifest and latent content. Manifest content means the elements physically present, that can be observed and counted. Latent content means the interpretation of texts, e.g. its symbolic function or the underlying meaning of texts [2]. In this study’s analysis, the manifest content of the evaluation reports was used.

Research approach

For the approach of document analysis, we used the basic model of the scientific method applied for a content analysis approach, as presented by Neuendorf [14]:

  1. 1.

    Determine what content will be analysed. In this study, the 88 evaluation reports.

  2. 2.

    Define and operationalize the variables to be studied. These have been included as focus areas within an operationalized and indexed perceptual framework of 21 focus areas. E-Government features in operationalization in parts, as follows:

    1. (a)

      The decision to have an evaluation carried out. The question is whether there is motivation from a political issue, questions from Parliament, or societal attention or unrest.

    2. (b)

      Which theoretical framework was used for this evaluation? In other words, are there any indications in the reports for, e.g., elements and instruments of New Public Governance?

    3. (c)

      Choosing evaluation techniques: Were approaches such as co-creation, participation taken into account, and were specific tools used to conduct research and to obtain, analyse and structure information?

    4. (d)

      The execution of a project, and the extent to which the content was analysed regarding what citizens think considering their participatory role and, if possible, how they affected the (project-oriented) creation, implementation, and monitoring of the projects. Topics such as usability, accessibility and various working methods like panels, sounding boards and design sessions are also part of the process.

  3. 3.

    Sampling of the content. Sampling is the selection of evaluation reports.

  4. 4.

    Train the coder(s). After initial training of the researcher, this was performed by carrying out a pilot of six cases, using a ‘convenience selection’. In this pilot, evaluation reports were assessed by the researcher using the perceptual framework. The results were subsequently provided to eight informants for the purpose of assessment and improving reliability.

  5. 5.

    Coding. The researcher then coded the other evaluation reports.

  6. 6.

    Calculating reliability. We started with determining face validity and content validity. This was done by testing the perceptual framework in a workshop with external experts. The experts are people with at least 10 years of experience in both public and private organizations. Additionally, the reliability of the perceptual framework was tested using the intercoder reliability [14]. A person other than the researcher went through two of the cases. The achieved intercoder reliability as a representation of the percentage of similarities (percent agreement, PAo) was calculated. In the analysis carried out, PAo = .76. The formula for this calculation is as follows:

    figure a

    where A is the number of similarities between the coders and n is the total number of components coded by the coders.

  7. 7.

    Tabulation and reports. This was done in the form of the doctoral thesis and various articles.

4 Results

The results of the 88 meta-evaluations provide insight into the extent to which citizens are involved in the evaluation of e-Government projects, and to what extent the approach chosen facilitated this.

Decision-making process

Regarding the decision-making surrounding an evaluation, it is clear that many reports are not compiled with a political motive or a motive from within the organization itself. Many reports have a content-oriented motivation, and the decision to have an evaluation carried out was made in only 35 out of 88 reports. The role of the citizen in this decision-making process, e.g. as the person requesting an evaluation through representatives, social institution or lobby, could not be demonstrated. The relationship between the project and the outside world is limited, as the results show:

  • In 83 out of 88 reports, there is no political issue, and no relationship with a political or social development was described.

  • In 19 out of 88 reports, the evaluation was carried out because an audit plan was drafted within the organization, or because agreements were made for carrying out an evaluation prior to the project.

  • For 12 out of 88 reports, we can see that the project featured an independent decision to have an evaluation carried out. This happened in the final stages of the project in all cases, in order to demonstrate the project results. Of the 12 times that a project was evaluated independently, there were indications on four occasions that the goal of doing so was to learn from the project.

Theoretical basis

Regarding the theoretical framework used, concepts of recent public administrative theories are virtually absent in evaluation reports. Elements such as (parts of) a network approach and attention for the projects’ environments are featured in a very limited fashion. Stakeholder management is present in a very limited fashion and, if it is present, it mainly focuses on the stakeholders within a government organization. Broad attention to the organization’s objectives in relation to the (social) environment could not be demonstrated in the evaluation reports reviewed. This means that, as expected, we did not find any evidence for the application of elements of New Public Governance (NPG), even though the reports were compiled in a period in which such elements could be reasonably assumed to have been fairly well known within the government.

Method of evaluation

In addition to the research approach, another point to consider in addition to the research approach is which of the techniques used for data collection and data analysis to use in the research design. E-Participation was not used, but other techniques were. The most important techniques used were:

  • In 48 out of 88 reports, interviews with individuals of (and surrounding) the project were conducted.

  • In 33 out of 88 reports, a document study was carried out. In 32 situations, this happened prior to the interviews.

  • In 19 out of 88 reports, group sessions were held in the form of focus groups (5), workshops (6) or group discussions (8).

  • In 10 out of 88 evaluations, questionnaires were used.

Demands of citizens

In the meta-evaluation, we looked into whether evaluation reports paid attention to the extent to which the project itself looked at what citizens thought of it from their participatory role. No evidence was found: regarding the collection of demands of citizens and businesses, no framework was included in the reviewed evaluation reports. In the project evaluations, attention to topics like usability or accessibility, or the desires and demands of citizens regarding the internal processes and facilities of the government was seldom paid.

Degree of influence of citizens

Very little evidence was found regarding the opportunities citizens have to affect both the (project-oriented) creation, implementation and monitoring of the projects in the evaluation reports. Focus on various working methods, such as citizen panels, sounding board and design sessions in evaluations was not found. References to e-Government and supporting means to realize it are not mentioned in the evaluations either.

Service provision to citizens

Improved service to citizens was featured as a result to be evaluated in only five of the evaluation reports. In 13 evaluation reports, it was mentioned that the organization within which the project is positioned has a core task in the area of improved service provision to citizens. A core task from the organization gives projects some direction. The presence of a clear core task should therefore be studied in the evaluation. However, our study shows that evaluations of projects pay very little attention to it. As a result, this focus area is difficult to measure and demonstrate in the evaluations.

5 Conclusions

The focus of many evaluations follows the lines of Administrative Theory [17]. The central focus is on the idea of the government as an efficient organization, with hierarchical management and strong administrative management of responsibilities and authorizations. The corresponding evaluations are traditional in their approach and execution. The 88 evaluations we studied have a signature ‘inside-in’ or ‘inside-out’ approach. No evidence for an explicit ‘outside-in’ view for decision-making was found. Political issues or a clear connection to a social development were hardly found at all. Our conclusion, therefore, is that the majority of the reports have more of an inward focus regarding the decision-making process surrounding evaluations, and consequently do not focus on the involvement of citizens, or their explicit desires and needs.

Additionally, we found little to no evidence in the studied evaluation reports of (elements of) a network approach and attention for the projects’ environments. These are all important aspects of New Public Governance (NPG). Stakeholder management is present in a very limited fashion as an evaluation topic and, if it is present, it mainly focuses on the stakeholders within a government organization. Through NPG, government managers are asked to manage while directing their attention outside of the organization, and to act in inter-organizational environments. In addition to collaborating with citizens and offering services to citizens, the focus is also on negotiation about the added value and the management of networks and the mutual relationships.

As a result, and as expected, we found no evidence for the application of elements from New Public Governance (NPG). Elements of New Public Service (NPS) could not be found either, despite the fact that this public administrative movement has existed for a long time. The role of the citizen as a ‘civilian’, and issues like co-creation with the outside world, are not a topic of discussion in evaluations.

E-Participation also requires a different working method from government organizations. It was barely observed at all in the study. For example, citizens can offer their input and cooperate during the process of determining demands regarding IT facilities, or they may be involved during the creation process.

In addition, it is conceivable that citizens also play a bigger role in monitoring or even evaluating such projects. There are various intervention moments and techniques that can be used during such a process. Some working methods include requirements engineering, user tests and other types of input like expert opinions. Additionally, there are instruments for e-Participation, including methods to maintain contact with citizens, industry associations, professional organizations and knowledge institutes via the internet and other social media.

The general conclusion is therefore twofold. Firstly, the role citizens play in creating ICT facilities, though the means available to them via e-Participation, is limited. Citizens are not involved in the beginning, during, or after the completion of projects. Citizens are only involved in a limited number of usability tests, or in a campaign for the introduction of a new process or product.

Secondly, the question as to whether citizens are sufficiently involved in monitoring and evaluating the quality of the creation processes and the implementation of facilities for e-Government remains unanswered. This topic was not discussed in the evaluations reviewed. Monitoring projects, from the perspective of retrospective monitoring (i.e. not working with learning and looking forward in mind), is most common in the evaluations.

6 Recommendations and Discussion

Our most important recommendation to quickly improve evaluations from an e-Participation perspective is to start with the so-called low hanging fruit. That is to say, issues that can be targeted quickly and easily, and that can be solved to create short-term improvements. Within the government, for instance, an idea would be to direct attention outward in addition to directing it inward, e.g. by hiring professionals from government networks, industry and consumer organizations and professional organizations to obtain different viewpoints and better and broader input in projects and evaluations.

We also recommend using modern and creative working methods to involve project staff and citizens in evaluations. A lot of experience was gained regarding co-creation, and there are ample opportunities to use ICT applications and social media, both inside and outside of government organizations, to involve the various stakeholders and users. These are the forms of e-Participation that are clearly visible within the European and international context [18]. This is about involving people who work with the government’s new ICT systems, as well as the citizens and businesses that experience the effects of government actions.

The application of evaluation in which citizens play a role (e-Evaluation) offers additional benefits. Since evaluations can be carried out more easily with modern electronic participatory means, is will become more easy to execute evaluations during different project stages. In addition, by using the resources of e-Government, boundaries of social status, position and area of expertise disappear. Everybody can be involved and can participate in evaluations, adding new insights and feedback. Even when disregarding the discussion of whether everyone has sufficient access to the digital channels provided by the government, participatory evaluation would be an improvement over the current method of evaluation.

Finally, the use of e-Evaluation could also result in a shift in the way in which the government executes projects. From the perspective of NPS and NPG, these projects use involved citizens before and during the project, as well as during the evaluation of said projects. Firstly, this results in different products for the purpose of interacting with citizens and businesses. And secondly, it results in a different way of implementing ICT products and related processes in the government organization and people using the ICT products [31]. This is also in line with the changes in project management and the way these projects create solutions for a variety of facilities: they are becoming increasingly multidisciplinary and agile [33]. This is exactly the type of approach where more intense involvement of citizens in drafting demands and monitoring quality is appropriate.

The study into the evaluation of projects with an ICT component within the government indicates, that the execution of projects and project evaluations in collaboration with citizens occurred infrequently, and that there is still much room for improvement in many aspects.