Abstract
Constraint solvers are complex pieces of software and are notoriously difficult to debug. In large part this is due to the difficulty of pinpointing the source of an error in the vast searches these solvers perform, since the effect of an error may only come to light long after the error is made. In addition, an error does not necessarily lead to the wrong result, further complicating the debugging process. A major source of errors in a constraint solver is the complex constraint propagation algorithms that provide the inference that controls and directs the search. In this paper we show that metamorphic testing is a principled way to test constraint solvers by comparing two different implementations of the same constraint. Specifically, specialised propagators for the constraint are tested against the general purpose table constraint propagator. We report on metamorphic testing of the constraint solver Minion. We demonstrate that the metamorphic testing method is very effective for finding artificial bugs introduced by random code mutation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The authors have experienced both of these conditions being violated in both their own, and other, solvers.
References
Apt, K.: Principles of Constraint Programming. Cambridge University Press, New York (2003)
Boussemart, F., Hemery, F., Lecoutre, C., Sais, L.: Boosting systematic search by weighting constraints. In: ECAI 2004, pp. 482–486 (2004)
Brummayer, R., Biere, A.: Fuzzing and delta-debugging SMT solvers. In: Proceedings of the 7th International Workshop on Satisfiability Modulo Theories, SMT 2009, pp. 1–5. ACM, New York (2009)
Budd, T.A.: Mutation analysis of program test data. Ph.D. thesis, New Haven, CT, USA (1980)
Debruyne, R., Bessière, C.: Some practicable filtering techniques for the constraint satisfaction problem. In: Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI 1997), pp. 412–417 (1997)
Feydy, T., Somogyi, Z., Stuckey, P.J.: Half reification and flattening. In: Lee, J. (ed.) CP 2011. LNCS, vol. 6876, pp. 286–301. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23786-7_23
Gecode Team: Gecode: Generic constraint development environment (2006). http://www.gecode.org
Gent, I.P., Jefferson, C., Miguel, I.: Watched literals for constraint propagation in minion. In: Benhamou, F. (ed.) CP 2006. LNCS, vol. 4204, pp. 182–197. Springer, Heidelberg (2006). https://doi.org/10.1007/11889205_15
Gent, I.P., Jefferson, C., Linton, S., Miguel, I., Nightingale, P.: Generating custom propagators for arbitrary constraints. Artif. Intell. 211, 1–33 (2014). http://www.sciencedirect.com/science/article/pii/S000437021400023X
Gent, I.P., Jefferson, C., Miguel, I.: Minion: a fast scalable constraint solver. In: ECAI, vol. 141, pp. 98–102 (2006)
Gent, I.P., Miguel, I., Nightingale, P.: Generalised arc consistency for the alldifferent constraint: an empirical survey. Artif. Intell. 172(18), 1973–2000 (2008)
Gent, I., Jefferson, C., Miguel, I., Nightingale, P.: Data structures for generalised arc consistency for extensional constraints. In: AAAI (CPPOD-19-2006-A), pp. 191–197 (2007). http://www.aaai.org/Papers/AAAI/2007/AAAI07-029.pdf
Jefferson, C., Kadioglu, S., Petrie, K.E., Sellmann, M., Živný, S.: Same-relation constraints. In: Gent, I.P. (ed.) CP 2009. LNCS, vol. 5732, pp. 470–485. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04244-7_38
Jefferson, C., Moore, N.C.A., Nightingale, P., Petrie, K.E.: Implementing logical connectives in constraint programming. Artif. Intell. 174(16–17), 1407–1429 (2010)
Jefferson, C., Nightingale, P.: Extending simple tabular reduction with short supports. In: IJCAI (2013)
Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Trans. Softw. Eng. 37(5), 649–678 (2011)
King, K.N., Offutt, A.J.: A fortran language system for mutation- based software testing. Softw. Pract. Exp. 21(7), 685–718 (1991). https://onlinelibrary.wiley.com/doi/abs/10.1002/spe.4380210704
Lecoutre, C., Saïs, L., Tabary, S., Vidal, V.: Reasoning from last conflict(s) in constraint programming. Artif. Intell. 173(18), 1592–1614 (2009). http://www.sciencedirect.com/science/article/pii/S0004370209001040
Nightingale, P.: The extended global cardinality constraint: an empirical survey. Artif. Intell. 175(2), 586–614 (2011)
Nightingale, P., Gent, I.P., Jefferson, C., Miguel, I.: Short and long supports for constraint propagation. J. Artif. Int. Res. 46(1), 1–45 (2013). http://dl.acm.org/citation.cfm?id=2512538.2512539
Reger, G., Suda, M., Voronkov, A.: Testing a saturation-based theorem prover: experiences and challenges. In: Gabmeyer, S., Johnsen, E.B. (eds.) TAP 2017. LNCS, vol. 10375, pp. 152–161. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61467-0_10
Rossi, F., van Beek, P., Walsh, T. (eds.): Handbook of Constraint Programming. Elsevier, New York (2006)
Schulte, C., Tack, G.: Weakly monotonic propagators. In: Gent, I.P. (ed.) CP 2009. LNCS, vol. 5732, pp. 723–730. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04244-7_56. http://www.gecode.org/paper.html?id=SchulteTack:CP:2009
Chen, T.Y., Cheung, S.C., Yiu, S.M.: Metamorphic testing: a new approach for generating next test cases. Technical report HKUST-CS98-01 (1998)
Acknowledgements
We thank EPSRC for funding this work via the grants EP/P015638/1 and EP/P026842/1. Dr Jefferson holds a Royal Society University Research Fellowship.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Akgün, Ö., Gent, I.P., Jefferson, C., Miguel, I., Nightingale, P. (2018). Metamorphic Testing of Constraint Solvers. In: Hooker, J. (eds) Principles and Practice of Constraint Programming. CP 2018. Lecture Notes in Computer Science(), vol 11008. Springer, Cham. https://doi.org/10.1007/978-3-319-98334-9_46
Download citation
DOI: https://doi.org/10.1007/978-3-319-98334-9_46
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-98333-2
Online ISBN: 978-3-319-98334-9
eBook Packages: Computer ScienceComputer Science (R0)