An Experimental Evaluation of the Understanding of Safety Compliance Needs with Models
Context: Most safety-critical systems have to fulfil compliance needs specified in safety standards. These needs can be difficult to understand from the text of the standards, and the use of conceptual models has been proposed as a solution. Goal: We aim to evaluate the understanding of safety compliance needs with models. Method: We have conducted an experiment to study the effectiveness, efficiency, and perceived benefits in understanding these needs, with text of safety standards and with UML object diagrams. Results: Sixteen Bachelor students participated in the experiment. Their average effectiveness in understanding compliance needs and their average efficiency were higher with models (17% and 15%, respectively). However, the difference is not statistically significant. The students found benefits in using models, but on average they are undecided about their ease of understanding. Conclusions: Although the results are not conclusive enough, they suggest that the use of models could improve the understanding of safety compliance needs.
KeywordsSafety-critical system Safety standard Safety compliance needs Model Understanding Comprehension Experiment
The research leading to this paper has received funding from the AMASS project (H2020-ECSEL grant agreement no 692474; Spain’s MINECO ref. PCIN-2015-262) and the AMoDDI project (Ref. 11130583). We also thank the subjects that participated in the experiment.
- 1.Abdulkhaleq, A, Wagner, S.: A controlled experiment for the empirical evaluation of safety analysis techniques for safety-critical software. In: EASE 2015, pp. 16:1–16:10 (2015)Google Scholar
- 6.de la Vara, J.L., et al.: Do models improve the understanding of safety compliance needs? Insights from a pilot experiment. In: ESEM, pp. 32:1–32:6 (2016)Google Scholar
- 10.ESA. Software engineering and standardisation (2006). http://www.esa.int/TEC/Software_engineering_and_standardisation/TECBUCUXBQE_0.html
- 11.Labunets, K., et al.: An experimental comparison of two risk-based security methods. In: ESEM, pp 163–172 (2013)Google Scholar
- 15.Razali, R., et al.: Experimental comparison of the comprehensibility of a UML-based formal specification versus a textual one. In: EASE (2007)Google Scholar
- 16.Salman, I., et al.: Are students representatives of professionals in software engineering experiments? In: ICSE (2015)Google Scholar
- 17.Sharafi, Z., et al.: An empirical study on the efficiency of graphical vs. textual representations in requirements comprehension. In: ICPC (2013)Google Scholar