Keywords

1 Introduction

The emergence of Participatory Budgeting (PB) programs in the public institutions were often linked to the increasing pressures imposed by the stakeholders including the governments, citizens and non-governmental organizations (NGOs). The central target is to provide opportunities for citizens to deliberately negotiate over the utilization and allocation of public money [1]. Besides fostering citizens’-government engagement and increasing transparency in public service delivery, subliminally PB has potentials to educate and empower the citizens, especially those niches. Traditionally, low-income and low-level of formal education citizens refrained themselves from involving in the government-related decision-making activities. By empowering them, the government could expand outreach and enhance the quality of democracy [1]. Acknowledging these potentials, PB could be a tool for more inclusive and accountable governance, contributing towards a higher public value [2, 3]. Many public institutions use social media (e.g. Facebook, Twitter) to engage with their citizens [4, 5]. Nevertheless, such alternatives constraint the citizens’ empowerment process [6, 7] due to the absence of real-time process and the “virtuous circles” [8, 9]. Hence, the need for a digitally-enabled PB platform triggers, leading to the development of EMPATIA (Enabling Multichannel Participation through ICT Adaptations for Participatory Budgeting ICT-enabled platform).

EMPATIA was developed under the framework of European Union’s innovation and research programme i.e. Horizon 2020. Its ultimate aim is to benefit global society and democracy [10]. The platform integrates the existing e-government platforms, and adds new features such as auto generation of data and visualization, voting systems and opinion mining to accommodate the desired PB function and enhance performance. It was hoped that these features would entice its adoptions, thus help to achieve the policy objective. However, the failure of past e-government platforms due to the scarcity of adoptions had triggered a risk to EMPATIA. To mitigate such risk, the critical performance indicators need to be identified. Hence, an objective and a cohesive evaluation metrics is needed to facilitate such action – which is what this study attempts to propose. The rest of this paper is organised as follows. Following this introduction, the paper presents the conceptual background discussing the significance of digitally enabled platform and participatory budgeting. Then, a description of the methodological approach was outlined, entailed by the discussion on the EMPATIA’s evaluation metrics. Finally, the paper concludes by highlighting the expected impact of the EMPTIA platform and way to move forward.

2 Conceptual Background

Digitally enabled service transformations offers vast potentials to create self-sustaining change in a broad range of connected technology, universal culture, and closely-linked communities [11, 12]. Explicit growth of digitally enabled services in public institutions was flourished by the expectancies to enhance governance process, increase citizens’ participations, and break the siloes between public institutions [12]. Such trend is also affiliated to the impact of communication technology advancements [3]. Many do not realized that the phenomenon partly caused by the New Public Management (NPM) movement [12]. NPM had redefined public services and government-citizens’ relationship, raising expectations on service standard and quality [13]. The scenario increases adoption of private sectors’ practices, including computer-aided services that subliminally deter democratic values by treating a citizen as a customer and imposing charges for rendered services. Next, the focus was shifted towards the quality of governance, where the concept of “participation” was introduced [14]. Such concept urged revitalization of the public sector’s roles through “partnerships” between government-citizens to improve social well-being and the quality of public administration, leading to the emergence of the participatory budgeting (PB) concept [13]. Meanwhile, the evolution of technology, increasing demands/expectations of the civil societies and complex political inferences prospers the growth of digitally-enabled services in public institutions [15]. The growth was also due to the over-arching economy pressures, which has channelled the focus of service delivery from public value (PV) creation (evaluated against services, outcome and trust) to cost feasibility [16]. Along the same line, the PV concept was argued as mimicking the definition of ‘perceived value’ in the marketing discipline. As such, it was claimed that the PV theory was unfit to evaluate the actual ‘value’ created by the digitally-enabled services – as the value was partly attributed to the information quality, system quality and service quality. In this perspective, the value is an antecedent determining satisfaction and trust among the users that will lead to re-use intention and increases usage level that characterized the IS success [16].

The e-voting feature in EMPATIA is claim as the main PV determinant. Besides promoting citizens’ empowerment, such feature facilitates political inclusiveness, while reducing contentment. Scholars have different views on how the introduction of an online platform for participation will affect the citizens, politically. Evidence shows that the introduction of an online platform for participation will encourage citizens’ participations [17]. Nevertheless, the digital divide advocators suggest that due to unequal access to the internet, the online PB platform will increase the involvement number of “economically advantaged groups” (i.e. people from the middle class to high-income groups) – who are already politically active, hence has a null effect on the politically-excluded group [18]. Instead, it will further empower the groups who are already resourceful and determined to participate. As the effect varies, this aspect worth an assessment. To enable the strategy development for the ‘politically excluded’ groups, citizens’ demographics information (e.g. income group, age, education level) and their motivation underpinning the system usage need to be retrieved. On top of the “inclusion” issue, the rising tide of discontentment in the global political landscape has aroused anti-politics orientation among the citizens [19]. This scenario creates the feeling of unrepresented in politic, unheard by the politicians, excluded due to social class and constrained opportunity to influence political decisions [20]. These feelings are often expressed in negative sentiments towards politicians, political institutions or politic per se, creating a gap between citizens and politically linked process, institutions or individuals [21]. Noting its ability to facilitate transparency and efficient execution of institutional roles, hypothetically, such issue can be reduced by PB implementation [22] – hence, should also be included in the metrics for evaluation.

3 Methodology

The overall methodological approach adopted to develop the EMPATIA’s evaluation metric was based on qualitative approach. It was stated the use of a single strategy limits data richness about certain phenomenon, and therefore the combination of strategies in a single research allows the weaknesses of certain strategy to be complemented by others [23]. Thus, a combined strategy of state-of-the-art literature review (secondary research), expert views mining and focus groups were used to collect data on the potential key performance indicators that will allow objective assessment of the online PB platform from the users’ point of view. Besides rich in data, such combination also facilitates the data triangulation process, which underpins the rigorousness of research [24].

The process started with a comprehensive state-of-the-art literature review (SOTA LR) and archival research on the relevant public sector - technology adoption and diffusion reports and publications. Using the thematic analysis approach, the investigation’s results were classified into two types of indicators - i.e. technical and non-technical, followed by an evaluation on its context-suitability against PB platform (i.e. EMPATIA). Next, a group of 20 experts dominating various roles across public and private sectors in several European countries were identified and invited to present their views towards PB platform in the two series of online ‘expert mining sessions’. They include the renown IS/IT/e-Government/Public Administration scholars, public sector employees (i.e. council staff and policy makers) and private sector practitioners (e.g. IS/IT consultants/contractors/consortiums) who have vast experience and specialities related to the PB platform. The findings from SOTA LR were used to stimulate the experts’ discussions and help to clarify certain issues. The outputs from the two sessions were used to structure an agenda for the subsequent focus group. Although it is not necessarily required [25], it was acknowledged that the agenda had elucidated themes for probing during the focus group sessions, which were held in three countries. Each session involves ten participants consisting the municipality staff, consortium members and public representatives (i.e. citizens). Finally, the findings were analyzed and triangulated to form a list of KPIs (and sub-indicators) that could facilitate an objective assessment of the PB platform’s performance.

4 Proposed Evaluation Metric

The process of making decisions pertaining budget allocations is part of the government’s primary role. Hence, the implementation of a multi-channel digitally enabled PB platform, or in this context EMPATIA - implied that such role is about to be heavily determined by the citizens. In this case, the proposed evaluation metric for the platform should be more incline towards the citizens’ interests. As stated in the earlier section, the platform was aimed to define a new form of democracy for the 21st-century nations and will be available freely for everyone to use. For such purpose, EMPATIA will be piloted in three different countries – each with unique requirements, scenarios, and users. Ideally, the platform should be assessed differently (i.e. according to their specific context). Nevertheless, the development of a generic evaluation metric is required as a performance baseline. Hence, this section presents the evaluation metric for EMPATIA’s performance assessment, developed based on the synthesis of findings derived from data collection strategies that were described in Sect. 2. The metric is displayed in Table 1. While the technical indicator focuses on the operation and performance of the platform’s “network” and “architecture”, the non-technical indicator evaluates the social-economy, behavioral, political and process aspects of the platform.

Table 1. KPIs for EMPATIA platform
Full size table

4.1 Technical KPIs

Data syntheses suggest that ‘technical indicator’ is highly important for the measurement of the platform’s technical-related performance. It is particularly critical, since the platform will be implemented across various contexts. Besides enabling the troubleshooting process, the indicator helps to determine the root of discrepancies in four dimensions: performance, usability, maintenance, and monitoring – adapted from the “Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models”, as outlined in ISO/IEC 25010:2011. Although SQuaRE proposes the assessment of all characteristics its two models of “quality in use” (focus on the outcome of interaction) and “product quality” (focus on the software’s static properties and system’s dynamic properties), findings suggest that only five characteristics (from the “product quality” model) are relevant to the research context [26]. The technical KPIs for EMPATIA are listed in Table 2.

Table 2. Technical KPIs selected for EMPATIA
Full size table

4.2 Non-technical Indicator: Behavioral

The behavioral indicator is one of the non-technical KPIs selected to indicate the PB platform’s performance. The sub-indicators under the behavioral indicator category are proposed based on the findings of the literature review. It was discovered that various models/theories were introduced to facilitate investigations on the reasons underpinning technology acceptance among the users. Nevertheless, user satisfaction remains as a central focus in all approaches. Against this backdrop, a theory known as ‘Unified Theory of Acceptance and Use of Technology (UTAUT) was developed to integrate eight ‘technology acceptance and use’ theories such as Theory of Reasoned Action (TRA) and Technology Acceptance Model (TAM), and used to explain the relationship between users’ intentions and their subsequent usage behavior [27].

Although many argue that the IS design, implementation strategy, and usage level are the important determinants for IS success, Bailey and Pearson [28] suggest that ‘user satisfaction’ is the key driver for the IS usage, which leads to its success. Next, Technology Acceptance Model (TAM) that encapsulates ‘user involvement concept’ was introduced [29]. TAM demonstrates how perceived usefulness and ease of use; as well as attitudes and behavioral intentions resulted into IS adoption. This indicates that user involvement is key to IS success. Such insight was incorporated in the Information System Success Model (ISSM), where success was attributable to the information quality, system quality, use, user satisfaction, individual impact and organizational impact [30]. According to ‘perceptions regarding information privacy’ theory, system security and information privacy are two other critical factors affecting users satisfaction on IS usage, thus worth evaluation [31]. Hence, it is proposed that the PB platform behavioral model to include concepts from UTAUT, ISSM and ‘perceptions regarding information privacy’ theory, and used these concepts as sub-indicators for evaluation purpose.

4.3 Non-technical Indicator: Socio-Economy

In a study that investigates citizens’ perspective toward digitally enabled public service, three indicators i.e. technical, social and economy were combined to assess the socio-economy impact [see 32]. As this evaluation metric will be used in the same context of digitally-enabled service, the same indicators will be used to evaluate the PB platform performance from the socio-economy perspective. However, since the technical evaluation will be conducted separately, the metric for socio-economy impact will be limited to social and economy indicators, as outlined in Table 3.

Table 3. Proposed KPIs for EMPATIA’s socio-economy assessment
Full size table

4.4 Non-technical Indicator: Political

Political indicator is important to enable the evaluation of the PB platform’s impact on the evolution of internal and external efficacy of users and overall trust in democratic institutions. Such evaluation can be performed by assessing the “inclusiveness” (i.e. to investigate participants’ profiles), and “Political Alienation” (i.e. to investigate participants’ “incapability” and “discontentment”) [32]. Two types of political alienation are identified as incapability and discontentment [33]. Incapability refers to efficacy (either internal efficacy i.e. citizens’ self-assessments of their own political judgments, or external efficacy i.e. citizens’ perceptions on how they influence the political decisions), and discontentment (i.e. negative affectation towards political objects, explaining why people believed that certain government policies are biased, or why they distrust political authorities). Restoration of trust towards government’s integrity in the decision-making processes is vital to remedy these situations [33]. As a platform that encourages transparency in public-budgeting processes, PB is advocated as a tool to fix the integrity issues, and subsequently enhance trust towards the government. Thus, a metric to evaluate such political aspects of PB platform performance, should consist of two main types of indicators – (1) internal and external efficacy and (2) anti-politics (measures of trust), where comparison of the users’ attitudes post PB platform usage with the baseline attitude of the random population in the same region is required to allow an objective assessment.

4.5 Non-technical Indicator: Process

Process indicator is a required to assess the level of process standardization during the pilot implementation. It was proposed that the process indicator should outline the basic requirements for the pilot implementation (i.e. to start the pilot), and criteria to exit the pilot (i.e. to stop the pilot) against user and process perspectives. Hence, the aspects to be considered in the evaluation metric under this indicator are listed in Table 4.

Table 4. Proposed indicators for EMPATIA’s process assessment
Full size table

5 Conclusions

The ‘participatory budgeting’ (PB) concept was introduced against the backdrop of a contemporary public administration to provide a platform for citizens to involve in the public budgeting process. In general, PB was targeted to improve the quality of public services and social well-being. With the emergence of new technology, the digitally enabled PB platform has sheds new lights in fostering better citizens-government engagement and broadening social-political impacts (e.g. improve economy conditions, enhance education level and flourish democracies). Nevertheless, the attainment of these desires is conditioned by the platform’s success – measured by its level of usage. Usage is determined by series of the cause-effect relationship. The process started with gaining value from the services, which raises satisfaction level among the users, producing the ‘re-use intentions’. Hence, value creation (often linked to the service performance) is essential to ensure the PB platform’s success. To do so, the KPIs representing the PB platform’s performance need to be identified, followed by the metric for evaluation. Since the existing models or theories offer limited scope of evaluation, this study proposes a new evaluation metric for the PB platform, outlining the technical and non-technical KPIs, developed against the context of EMPATIA. The metric served as a basis for the evaluation’s instruments development in a near future, where the platform’s performance result will be obtained. Besides signposting numerous potential practical and theoretical insights, the evaluation’s outcomes will assist interventions, thus help to promote the success of the digitally enabled PB platform.