Abstract
This paper reports on a reliability analysis of a Software-mediated Process Assessment (SMPA) decision support tool as it was used at three sites. The tool was used at two sites as part of its development process. Following its evaluation, further development included the formulation of assessment questions and knowledge items to complete the requirements for all ISO/IEC 20000 processes. The SMPA was deployed at a third site, a large financial services firm. These three case studies brought up the issue of the reliability of the capability ratings and the value of the reliability measure (coefficient of variation) incorporated in the SMPA. Further data collection when undertaking assessments is required to have better assurance of the results given the variability of subjects. In addition the future of ISO/IEC 15504/ISO/IEC 330xx and ISO/IEC 20000 and how the SMPA will need to change to maintain alignment is raised.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
ISO/IEC: ISO/IEC 20000-1:2011 – Information Technology – Service Management – Part 1: Service Management System Requirements. International Organisation for Standardisation, Geneva (2011)
OGC: ITIL Service Strategy. TSO for the Office of Government Commerce, London (2011)
Winkler, T.J., Wulf, J., Brenner, W.: The relevance of IT service management maturity for IT alignment and its strategy and organization size contingencies. In: First AIS-Journals Joint Author Workshop in ECIS. AIS Electronic Library (AISeL) (2014)
ISO/IEC: ISO/IEC 33001:2016 Information technology - Process assessment - Concepts and terminology. In: JTC IT-015 (ed.). International Organization for Standardization, Geneva (2015)
SCAMPI Upgrade Team: Appraisal Requirements for CMMI® Version 1.3 (2011)
Parasuraman, A., Zeithaml, V.A., Berry, L.L.: A conceptual model of service quality and its implications for future research. J. Mark. 49, 41–50 (1985)
Lepmets, M., Cater-Steel, A., Gacenga, F., Ras, E.: Extending the IT service quality measurement framework through a systematic literature review. J. Serv. Sci. Res. 4, 7–47 (2012)
Jia, R., Reich, B.H.: IT service climate—an essential managerial tool to improve client satisfaction with it service quality. Inf. Syst. Manage. 28, 174–179 (2011)
Spath, D., Bauer, W., Praeg, C.-P.: IT service quality management: assumptions, frameworks and effects on business performance. In: Praeg, C.-P., Spath, D. (eds.) Quality Management for IT Services-Perspectives on Business and Process Performance, pp. 1–21. IGI Global, Hershey (2011)
Radice, R.A., Harding, J.T., Munnis, P.E., Phillips, R.W.: A programming process study. IBM Syst. J. 24, 79–90 (1985)
Humphrey, W.S.: Managing the Software Process. Addison-Wesley, Reading (1990)
ISO/IEC: ISO/IEC 15504.1:2005 Information technology - Process assessment - Concepts and vocabulary (2005)
Rout, T.P., El Emam, K., Fusani, M., Goldenson, D., Jung, H.W.: SPICE in retrospect: developing a standard for process assessment. J. Syst. Softw. 80, 1483–1493 (2007)
Rout, T.P.: A new framework for standardisation and certification for process assessment. Softw. Qual. Prof. 14 (2011)
Rout, T.P.: Critical design decisions in the development of the standard for process assessment. In: Woronowicz, T., Rout, T., O’Connor, R.V., Dorling, A. (eds.) SPICE 2013. CCIS, vol. 349, pp. 247–251. Springer, Heidelberg (2013)
Mesquida, A., Mas, A., Amengual, E., Calvo-Manzano, J.: IT service management process improvement based on ISO/IEC 15504: a systematic review. Inf. Softw. Technol. 54, 239–247 (2012)
Barafort, B., Betry, V., Cortina, S., Picard, M., St-Jean, M., Renault, A., Valdès, O.: ITSM Process Assessment Supporting ITIL. Van Haren Publishing, Zaltbommel (2009)
Renault, A., Cortina, S., Barafort, B.: Towards a maturity model for ISO/IEC 20000-1 based on the TIPA for ITIL process capability assessment model. In: Rout, T., O’Connor, R.V., Dorling, A. (eds.) SPICE 2015. CCIS, vol. 526, pp. 188–200. Springer, Heidelberg (2015)
Lloyd, V.: ITIL Continual Service Improvement. The Stationery Office, London (2011)
Straub, D.W.: Validating instruments in MIS research. MIS Q. 13, 147–168 (1989)
MacKenzie, S.B., Podsakoff, P.M., Podsakoff, N.P.: Construct measurement and validation procedures in MIS and behavioral research: integrating new and existing techniques. MIS Q. 35, 293–334 (2011)
Lord, F.M., Novick, M.R.: Statistical Theories of Mental Test Scores. Addison-Wesley Publishing, Reading (1968)
ISO/IEC: ISO/IEC 15504-2:2004 – Information Technology – Process Assessment – Part 2: Performing an Assessment. International Organization for Standardization, Geneva (2004)
ISO/IEC: ISO/IEC TS 15504-8:2012 - Information Technology - Process Assessment - Part 8: An Exemplar Process Assessment Model for IT Service Management. International Organization for Standardization, Geneva (2012)
Shrestha, A., Cater-Steel, A., Toleman, M., Rout, T.: Towards transparent and efficient process assessments for IT service management. In: Mitasiunas, A., Rout, T., O’Connor, R.V., Dorling, A. (eds.) SPICE 2014. CCIS, vol. 477, pp. 165–176. Springer, Heidelberg (2014)
Deutskens, E., de Ruyter, K., Wetzels, M.: An assessment of equivalence between online and mail surveys in service research. J. Serv. Res. 8, 346–355 (2006)
ISO/IEC: ISO/IEC TS 15504-9:2011 – Information Technology – Process Assessment – Part 9: Target Process Profiles. International Organisation for Standardisation, Geneva (2011)
Pries-Heje, J., Baskerville, R., Venable, J.R.: Strategies for design science research evaluation. In: 16th European Conference on Information Systems. (2008)
Hevner, A.R., March, S.T., Park, J., Ram, S.: Design science in information systems research. MIS Q. 28, 75–106 (2004)
ISO/IEC: ISO/IEC 25010:2011 – Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models. International Organisation for Standardisation, Geneva (2011)
Shrestha, A., Cater-Steel, A., Toleman, M., Rout, T.: Evaluation of software mediated process assessments for IT service management. In: Rout, T., O’Connor, R.V., Dorling, A. (eds.) SPICE 2015. CCIS, vol. 526, pp. 72–84. Springer, Heidelberg (2015)
ISO/IEC: FDIS 33071 - Information technology – Process assessment – An integrated process capability assessment model for Enterprise processes. International Organisation for Standardisation (2016)
Acknowledgements
This work is supported by an Australian Research Council (ARC) Linkage grant. We thank Mr. Paul Collins of Assessment Portal Pty. Ltd. for his involvement and support in providing the platform to implement the DSS tool. We also appreciate the opportunity provided by staff at the three case study organisations.
Note. ITIL® is a Registered Trade Mark of AXELOS Limited.
CMMI® is the Registered Trademark of Carnegie Mellon University.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Shrestha, A., Cater-Steel, A., Toleman, M., Rout, T., Behari, S. (2016). The Issue of Reliability in Software Mediated Process Assessments. In: Clarke, P., O'Connor, R., Rout, T., Dorling, A. (eds) Software Process Improvement and Capability Determination. SPICE 2016. Communications in Computer and Information Science, vol 609. Springer, Cham. https://doi.org/10.1007/978-3-319-38980-6_15
Download citation
DOI: https://doi.org/10.1007/978-3-319-38980-6_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-38979-0
Online ISBN: 978-3-319-38980-6
eBook Packages: Computer ScienceComputer Science (R0)