Abstract
One approach to improve the accuracy of classifications based on generative models is to combine them with successful discriminative algorithms. Fisher kernels were developed to combine generative models with a currently very popular class of learning algorithms, kernel methods. Empirically, the combination of hidden Markov models with support vector machines has shown promising results. So far, however, Fisher kernels have only been considered for sequences over flat alphabets. This is mostly due to the lack of a method for computing the gradient of a generative model over structured sequences. In this paper, we show how to compute the gradient of logical hidden Markov models, which allow for the modelling of logical sequences, i.e., sequences over an alphabet of logical atoms. Experiments show a considerable improvement over results achieved without Fisher kernels for logical sequences.
Chapter PDF
Similar content being viewed by others
References
Anderson, C.R., Domingos, P., Weld, D.S.: Relational Markov Models and their Application to Adaptive Web Navigation. In: Proceedings of the Eighth International Conference on Knowledge Discovery and Data Mining, KDD (2002)
De Raedt, L., Kersting, K.: Probabilistic Logic Learning. ACM-SIGKDD Explorations 5(1), 31–48 (2003)
Fine, S., Singer, Y., Tishby, N.: The hierarchical hidden markov model: analysis and applications. Machine Learning 32, 41–62 (1998)
Fischer, J., Kersting, K.: Scaled CGEM: A Fast Accelerated EM. In: Lavrač, N., Gamberger, D., Todorovski, L., Blockeel, H. (eds.) ECML 2003. LNCS (LNAI), vol. 2837, pp. 133–144. Springer, Heidelberg (2003)
Gärtner, T.: Kernel-based Learning in Multi-Relational Data Mining. ACMSIGKDD Explorations 5(1), 49–58 (2003)
Ghahramani, Z., Jordan, M.: Factorial hidden Markov models. Machine Learning 29, 245–273 (1997)
Hargbo, J., Elofsson, A.: Hidden markov models that use predictied secondary structure for fold recognition. Proteins: Structure, Function, and Genetics 36, 68–76 (1999)
Haussler, D.: Convolution kernels on discrete structures. Technical report, Department of Computer Science, University of California at Santa Cruz (1999)
Horváth, T., Wrobel, S., Bohnebeck, U.: Relational Instance-Based learning with Lists and Terms. Machine Learning 43(1/2), 53–80 (2001)
Hubbard, T., Murzin, A., Brenner, S., Chotia, C.: SCOP: a structural classification of proteins database. NAR 27(1), 236–239 (1997)
Jaakkola, T.S., Haussler, D.: Exploiting generative models in discriminative classifiers. In: Adv. in Neural Information Processing Systems, vol. 11, MIT Press, Cambridge (1999)
Joachims, T.: Learning to Classify Text Using Support Vector Machines: Methods, Theory and Algorithms. Kluwer, Dordrecht (2002)
Kersting, K., De Raedt, L., Raiko, T.: Logical hidden markov models (2004) (submitted)
Kersting, K., Raiko, T., Kramer, S., De Raedt, L.: Towards discovering structural signatures of protein folds based on logical hidden markov models. In: Proceedings of the Pacific Symposium on Biocomputing, PSB 2003 (2003)
Kersting, K., Raiko, T., De Raedt, L.: A Structural GEM for Learning Logical Hidden Markov Models. In: Working Notes of the Second KDD-Workshop on Multi-Relational Data Mining, MRDM 2003 (2003)
Lloyd, J.W.: Foundations of Logic Programming, 2nd edn. Springer, Heidelberg (1989)
Rabiner, L.R.: A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Proceedings of the IEEE 77(2), 257–286 (1989)
Salakhutdimov, R., Roweis, S., Ghahramani, Z.: Optimization with EM and Expectation-Conjugate-Gradient. In: Proceedings of the Twentieth International Conference on Machine Learning, ICML 2003 (2003)
Schölkopf, B., Smola, A.J.: Learning with Kernels. MIT Press, Cambridge (2002)
Taskar, B., Abbeel, P., Koller, D.: Discriminative Probabilistic Models for Relational Data. In: Proceedings of the Eighteenth Conference on Uncertainty in Artificial Intelligence, UAI 2002 (2002)
Tsuda, K., Kawanabe, M., Rätsch, G., Sonnenburg, S., Müller, K.-R.: A new discriminative kernel from probabilistic models. In: Adv. in Neural Information Processing Systems, MIT Press, Cambridge (2002)
Tsuda, K., Kin, T., Asai, K.: Marginalized kernels for biological sequences. Bioinformatics (2002)
Watkins, C.: Kernels from matching operations. Technical report, Department of Computer Science, Royal Holloway, University of London (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kersting, K., Gärtner, T. (2004). Fisher Kernels for Logical Sequences. In: Boulicaut, JF., Esposito, F., Giannotti, F., Pedreschi, D. (eds) Machine Learning: ECML 2004. ECML 2004. Lecture Notes in Computer Science(), vol 3201. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30115-8_21
Download citation
DOI: https://doi.org/10.1007/978-3-540-30115-8_21
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-23105-9
Online ISBN: 978-3-540-30115-8
eBook Packages: Springer Book Archive