Evaluation in Pilot-Based Research Projects: A Hierarchical Approach

  • Thomas ZeffererEmail author
Conference paper
Part of the Lecture Notes in Business Information Processing book series (LNBIP, volume 372)


The evaluation of project results is a crucial part of most research projects in the information-technology domain. In particular, this applies to projects that develop solutions for the public sector and test them by means of pilot applications. In these projects, involved stakeholders require meaningful evaluation results to derive lessons learned and to steer future research activities in the right directions. Today, most projects define and apply their own evaluation schemes. This yields evaluation results that are difficult to compare between projects. Sometimes, inconsistent evaluation schemes are even applied within a project. In these cases, even evaluation results of a single project lack comparability. In absence of coherent evaluation schemes, lessons learned from conducted evaluation processes cannot be aggregated to a coherent holistic picture and the overall gain of research projects remains limited. To address this issue, we propose and introduce an evaluation scheme for arbitrary pilot-based research projects targeting the public sector. The proposed scheme follows a hierarchical approach. Concretely, it organizes evaluation criteria on different layers of abstraction. Furthermore, the proposed scheme describes in detail necessary process steps to carry out evaluations using the defined criteria. This way, the proposed scheme enables in-depth evaluations of research projects and their pilot applications. At the same time, it assures that obtained evaluation results remain comparable anytime. The hierarchical evaluation scheme introduced in this article has been successfully applied to the international research project FutureTrust, demonstrating its practical applicability, and showing its potential also for future research projects.


Evaluation scheme Project evaluation Hierarchical evaluation model 



This work has been supported by the FutureTrust project (N.700542) funded under the programme H2020-EU.3.7. - Secure societies - Protecting freedom and security of Europe and its citizens (2013/743/EU of 2013-12-03).


  1. 1.
    Asosheh, A., Nalchigar, S., Jamporazmey, M.: Information technology project evaluation: an integrated data envelopment analysis and balanced scorecard approach. Exp. Syst. Appl. 37(8), 5931–5938 (2010). Scholar
  2. 2.
    De Cock, D., Wouters, K., Preneel, B.: Introduction to the Belgian EID card. In: Katsikas, S.K., Gritzalis, S., López, J. (eds.) EuroPKI 2004. LNCS, vol. 3093, pp. 1–13. Springer, Heidelberg (2004). Scholar
  3. 3.
    Doran, G.T.: There’s a SMART way to write managements’s goals and objectives. Manag. Rev. 70(11), 2 (1981). EBSCOhost. Scholar
  4. 4.
    Eilat, H., Golany, B., Shtub, A.: R&d project evaluation: an integrated dea and balanced scorecard approach. Omega 36(5), 895–912 (2008). Scholar
  5. 5.
    European Commission: Digital Agenda for Europe (2018). Accessed 05 July 2018
  6. 6.
    European Commission: H2020-EU.3.7. - Secure societies - Protecting freedom and security of Europe and its citizens (2018). Accessed 05 July 2018
  7. 7.
    European Commission: Shaping the Digital Single Market (2018). Accessed 05 July 2018
  8. 8.
    European Union: Regulation (EU) No 910/2014 of the European Parliament and of the Council of 23 July 2014 on electronic identification and trust services for electronic transactions in the internal market and repealing Directive 1999/93/EC (2018). Accessed 05 July 2018
  9. 9.
    Khan, Z., Ludlow, D., Caceres, S.: Evaluating a collaborative it based research and development project. Eval. Prog. Planning 40, 27–41 (2013). Scholar
  10. 10.
    Knall, T., Tauber, A., Zefferer, T., Zwattendorfer, B., Axfjord, A., Bjarnason, H.: Secure and privacy-preserving cross-border authentication: the STORK pilot ‘SaferChat’. In: Andersen, K.N., Francesconi, E., Grönlund, Å., van Engers, T.M. (eds.) EGOVIS 2011. LNCS, vol. 6866, pp. 94–106. Springer, Heidelberg (2011). Scholar
  11. 11.
    Leitold, H., Hollosi, A., Posch, R.: Security Architecture of the Austrian Citizen Card Concept. In: 18th Annual Computer Security Applications Conference, 2002. Proceedings, pp. 391–400 (2002).
  12. 12.
    Leitold, H., Posch, R.: STORK - technical approach and privacy. In: Bus, J., Crompton, M., Hildebrandt, M., Metakides, G. (eds.) Digital Enlightenment Yearbook 2012, pp. 289–306 (2012)Google Scholar
  13. 13.
    Martens, T.: Electronic identity management in Estonia between market and state governance. Identity Inf. Soc. 3(1), 213–233 (2010). Scholar
  14. 14.
    Rath, C., Roth, S., Bratko, H., Zefferer, T.: Encryption-based second authentication factor solutions for qualified server-side signature creation. In: Kő, A., Francesconi, E. (eds.) EGOVIS 2015. LNCS, vol. 9265, pp. 71–85. Springer, Cham (2015). Scholar
  15. 15.
    Suzic, B., Reiter, A.: Towards secure collaboration in federated cloud environments. In: 2016 11th International Conference on Availability, Reliability and Security (ARES), pp. 750–759, August 2016.
  16. 16.
    Tauber, A., Zwattendorfer, B., Zefferer, T.: Stork: pilot 4 towards cross-border electronic delivery. In: Verlag, T. (ed.) Electronic Government and Electronic Participation - Joint Proceedings of Ongoing Research and Projects of IFIP EGOV and ePart 2011, pp. 295–301 (2011)Google Scholar
  17. 17.
    Tou, J.T.: Information systems. In: von Brauer, W. (ed.) GI 1973. LNCS, vol. 1, pp. 489–507. Springer, Heidelberg (1973). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Secure Information Technology Center Austria (A-SIT)GrazAustria

Personalised recommendations