Metamorphic Testing of Constraint Solvers

  • Özgür AkgünEmail author
  • Ian P. Gent
  • Christopher Jefferson
  • Ian Miguel
  • Peter Nightingale
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11008)


Constraint solvers are complex pieces of software and are notoriously difficult to debug. In large part this is due to the difficulty of pinpointing the source of an error in the vast searches these solvers perform, since the effect of an error may only come to light long after the error is made. In addition, an error does not necessarily lead to the wrong result, further complicating the debugging process. A major source of errors in a constraint solver is the complex constraint propagation algorithms that provide the inference that controls and directs the search. In this paper we show that metamorphic testing is a principled way to test constraint solvers by comparing two different implementations of the same constraint. Specifically, specialised propagators for the constraint are tested against the general purpose table constraint propagator. We report on metamorphic testing of the constraint solver Minion. We demonstrate that the metamorphic testing method is very effective for finding artificial bugs introduced by random code mutation.



We thank EPSRC for funding this work via the grants EP/P015638/1 and EP/P026842/1. Dr Jefferson holds a Royal Society University Research Fellowship.


  1. 1.
    Apt, K.: Principles of Constraint Programming. Cambridge University Press, New York (2003)CrossRefGoogle Scholar
  2. 2.
    Boussemart, F., Hemery, F., Lecoutre, C., Sais, L.: Boosting systematic search by weighting constraints. In: ECAI 2004, pp. 482–486 (2004)Google Scholar
  3. 3.
    Brummayer, R., Biere, A.: Fuzzing and delta-debugging SMT solvers. In: Proceedings of the 7th International Workshop on Satisfiability Modulo Theories, SMT 2009, pp. 1–5. ACM, New York (2009)Google Scholar
  4. 4.
    Budd, T.A.: Mutation analysis of program test data. Ph.D. thesis, New Haven, CT, USA (1980)Google Scholar
  5. 5.
    Debruyne, R., Bessière, C.: Some practicable filtering techniques for the constraint satisfaction problem. In: Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI 1997), pp. 412–417 (1997)Google Scholar
  6. 6.
    Feydy, T., Somogyi, Z., Stuckey, P.J.: Half reification and flattening. In: Lee, J. (ed.) CP 2011. LNCS, vol. 6876, pp. 286–301. Springer, Heidelberg (2011). Scholar
  7. 7.
    Gecode Team: Gecode: Generic constraint development environment (2006).
  8. 8.
    Gent, I.P., Jefferson, C., Miguel, I.: Watched literals for constraint propagation in minion. In: Benhamou, F. (ed.) CP 2006. LNCS, vol. 4204, pp. 182–197. Springer, Heidelberg (2006). Scholar
  9. 9.
    Gent, I.P., Jefferson, C., Linton, S., Miguel, I., Nightingale, P.: Generating custom propagators for arbitrary constraints. Artif. Intell. 211, 1–33 (2014). Scholar
  10. 10.
    Gent, I.P., Jefferson, C., Miguel, I.: Minion: a fast scalable constraint solver. In: ECAI, vol. 141, pp. 98–102 (2006)Google Scholar
  11. 11.
    Gent, I.P., Miguel, I., Nightingale, P.: Generalised arc consistency for the alldifferent constraint: an empirical survey. Artif. Intell. 172(18), 1973–2000 (2008)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Gent, I., Jefferson, C., Miguel, I., Nightingale, P.: Data structures for generalised arc consistency for extensional constraints. In: AAAI (CPPOD-19-2006-A), pp. 191–197 (2007).
  13. 13.
    Jefferson, C., Kadioglu, S., Petrie, K.E., Sellmann, M., Živný, S.: Same-relation constraints. In: Gent, I.P. (ed.) CP 2009. LNCS, vol. 5732, pp. 470–485. Springer, Heidelberg (2009). Scholar
  14. 14.
    Jefferson, C., Moore, N.C.A., Nightingale, P., Petrie, K.E.: Implementing logical connectives in constraint programming. Artif. Intell. 174(16–17), 1407–1429 (2010)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Jefferson, C., Nightingale, P.: Extending simple tabular reduction with short supports. In: IJCAI (2013)Google Scholar
  16. 16.
    Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Trans. Softw. Eng. 37(5), 649–678 (2011)CrossRefGoogle Scholar
  17. 17.
    King, K.N., Offutt, A.J.: A fortran language system for mutation- based software testing. Softw. Pract. Exp. 21(7), 685–718 (1991). Scholar
  18. 18.
    Lecoutre, C., Saïs, L., Tabary, S., Vidal, V.: Reasoning from last conflict(s) in constraint programming. Artif. Intell. 173(18), 1592–1614 (2009). Scholar
  19. 19.
    Nightingale, P.: The extended global cardinality constraint: an empirical survey. Artif. Intell. 175(2), 586–614 (2011)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Nightingale, P., Gent, I.P., Jefferson, C., Miguel, I.: Short and long supports for constraint propagation. J. Artif. Int. Res. 46(1), 1–45 (2013). Scholar
  21. 21.
    Reger, G., Suda, M., Voronkov, A.: Testing a saturation-based theorem prover: experiences and challenges. In: Gabmeyer, S., Johnsen, E.B. (eds.) TAP 2017. LNCS, vol. 10375, pp. 152–161. Springer, Cham (2017). Scholar
  22. 22.
    Rossi, F., van Beek, P., Walsh, T. (eds.): Handbook of Constraint Programming. Elsevier, New York (2006)zbMATHGoogle Scholar
  23. 23.
    Schulte, C., Tack, G.: Weakly monotonic propagators. In: Gent, I.P. (ed.) CP 2009. LNCS, vol. 5732, pp. 723–730. Springer, Heidelberg (2009). Scholar
  24. 24.
    Chen, T.Y., Cheung, S.C., Yiu, S.M.: Metamorphic testing: a new approach for generating next test cases. Technical report HKUST-CS98-01 (1998)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Özgür Akgün
    • 1
    Email author
  • Ian P. Gent
    • 1
  • Christopher Jefferson
    • 1
  • Ian Miguel
    • 1
  • Peter Nightingale
    • 1
  1. 1.School of Computer ScienceUniversity of St AndrewsSt AndrewsUK

Personalised recommendations