Abstract
The accuracy of an inferred grammar is commonly computed by measuring the percentage of sequences that are correctly classified from a random sample of sequences produced by the target grammar. This approach is problematic because (a) it is unlikely that a random sample of sequences will adequately test the grammar and (b) the use of a single probability value provides little insight into the extent to which a grammar is (in-)accurate. This paper addresses these two problems by proposing the use of established model-based testing techniques from the field of software engineering to systematically generate test sets, along with the use of the Precision and Recall measure from the field of information retrieval to concisely represent the accuracy of the inferred machine.
This work has been funded by the AutoAbstract EPSRC grant EP/C511883/1.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Angluin, D., Smith, C.H.: Inductive inference: Theory and methods. Computing Surveys 15(3), 237–269 (1983)
Berg, T., Grinchtein, O., Jonsson, B., Leucker, M., Raffelt, H., Steffen, B.: On the correspondence between conformance testing and regular inference. In: Cerioli, M. (ed.) FASE 2005. LNCS, vol. 3442, pp. 175–189. Springer, Heidelberg (2005)
Bogdanov, K., Holcombe, M., Ipate, F., Seed, L., Vanak, S.: Testing methods for X-Machines: A review. Formal Aspects of Computer Science 18, 3–30 (2006)
Bongard, J., Lipson, H.: Active coevolutionary learning of deterministic finite automata. Journal of Machine Learning Research 6, 1651–1678 (2005)
Bradley, A.: The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition 30(7), 1145–1159 (1997)
Chow, T.: Testing Software Design Modelled by Finite State Machines. IEEE Transactions on Software Engineering 4(3), 178–187 (1978)
Davis, J., Goadrich, M.: The relationship between precision-recall and ROC curves. In: ICML. ACM International Conference Proceeding Series, vol. 148, pp. 233–240. ACM, New York (2006)
Dupont, P.: Regular grammatical inference from positive and negative samples by genetic search: the GIG method. In: Carrasco, R.C., Oncina, J. (eds.) ICGI 1994. LNCS, vol. 862, pp. 236–245. Springer, Heidelberg (1994)
Dupont, P., Lambeau, B., Damas, C., van Lamsweerde, A.: The QSM algorithm and its application to software behavior model induction. Applied Artificial Intelligence 22, 77–115 (2008)
Dupont, P., Miclet, L., Vidal, E.: What is the search space of the regular inference? In: Carrasco, R.C., Oncina, J. (eds.) ICGI 1994. LNCS, vol. 862, pp. 25–37. Springer, Heidelberg (1994)
Fonseca, C., Fleming, P.: An overview of evolutionary algorithms in multiobjective optimization. Evolutionary Computation 3(1), 1–16 (1995)
Gill, A.: Introduction to the Theory of Finite State Machines. McGraw-Hill, New York (1962)
Gold, E.: Language identification in the limit. Information and Control 10, 447–474 (1967)
Groce, A., Peled, D., Yannakakis, M.: Adaptive model checking. Logic Journal of the IGPL 14(5), 729–744 (2006)
Lang, K.: Random DFA’s can be approximately learned from sparse uniform examples. In: COLT, pp. 45–52 (1992)
Lang, K., Pearlmutter, B., Price, R.: Results of the Abbadingo One DFA learning competition and a new evidence-driven state merging algorithm. In: Honavar, V.G., Slutzki, G. (eds.) ICGI 1998. LNCS (LNAI), vol. 1433, pp. 1–12. Springer, Heidelberg (1998)
Lee, D., Yannakakis, M.: Principles and Methods of Testing Finite State Machines - A Survey. Proceedings of the IEEE 84, 1090–1126 (1996)
Lo, D., Khoo, S.: QUARK: Empirical assessment of automaton-based specification miners. In: WCRE, pp. 51–60. IEEE Computer Society, Los Alamitos (2006)
Oncina, J., Garcia, P.: Inferring regular languages in polynomial update time. In: Pérez de la Blanca, N., Sanfeliu, A., Vidal, E. (eds.) Pattern Recognition and Image Analysis. Series in Machine Perception and Artificial Intelligence, vol. 1, pp. 49–61. World Scientific, Singapore (1992)
Parekh, R., Honavar, V.: Grammar Inference, Automata Induction, and Language Acquisition, ch. 29. Marcel Dekker, USA (2000)
Tomita, M.: Dynamic construction of finite-state automata from examples using hill-climbing. In: Proceedings of the Fourth Annual Cognitive Science Conference, Ann Arbor, Mi, pp. 105–108 (1982)
Trakhtenbrot, B., Barzdin, Y.: Finite Automata, Behavior and Synthesis. North Holland, Amsterdam (1973)
Tu, K., Honavar, V.: Unsupervised learning of probabilistic context-free grammar using iterative biclustering. Technical Report 00000572, Dept. Computer Science, Iowa State University (May 2008)
van Rijsbergen, C.J.: Information Retrieval. Butterworth-Heineman, Newton (1979)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Walkinshaw, N., Bogdanov, K., Johnson, K. (2008). Evaluation and Comparison of Inferred Regular Grammars. In: Clark, A., Coste, F., Miclet, L. (eds) Grammatical Inference: Algorithms and Applications. ICGI 2008. Lecture Notes in Computer Science(), vol 5278. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-88009-7_20
Download citation
DOI: https://doi.org/10.1007/978-3-540-88009-7_20
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-88008-0
Online ISBN: 978-3-540-88009-7
eBook Packages: Computer ScienceComputer Science (R0)