Abstract
In supervised learning feature vectors are often implicitly mapped to a high-dimensional space using the kernel trick with quadratic costs for the learning algorithm. The recently proposed random Fourier features provide an explicit mapping such that classical algorithms with often linear complexity can be applied. Yet, the random Fourier feature approach remains widely complex techniques which are difficult to interpret. Using Matrix Relevance Learning the linear mapping of the data for a better class separation can be learned by adapting a parametric Euclidean distance. Further, a low-rank representation of the input data can be obtained. We apply this technique to random Fourier feature encoded data to obtain a discriminative mapping of the kernel space. This explicit approach is compared with a differentiable kernel vector quantizer on the same but implicit kernel representation. Using multiple benchmark problems, we demonstrate that a parametric distance on a RBF encoding yields to better classification results and permits access to interpretable prediction models with visualization abilities.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
The random Fourier features are generated such that the respective RBF kernel is approximated.
- 2.
Note that all vectors in DK-GMLVQ still live in the same D-dimensional space.
- 3.
Often the RBF encoding is considered as a silver bullet, but if it fails a controlled inspection framework can be very useful.
References
A.Rahimi, Recht, B.: Random features for large-scale kernel machines. In: Platt, J.C., Koller, D., Singer, Y., Roweis, S.T. (eds.) Advances in neural information processing systems 20. In: Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems. Curran Associates, Inc. (2007). http://books.nips.cc/papers/files/nips20/NIPS2007_0833.pdf
Arlt, W., Biehl, M., Taylor, A.: Urine steroid metabolomics as a biomarker tool for detecting malignancy in adrenal tumors. J. Clin. Endocrinol. Metab. 96, 3775–3784 (2011)
Biehl, M., Bunte, K., Schleif, F.M., Schneider, P., Villmann, T.: Large margin linear discriminative visualization by matrix relevance learning. In: Proceedings of IJCNN 2012, 1873–1880 (2012)
Biehl, M., Hammer, B., Schleif, F.M., Schneider, P., Villmann, T.: Stationarity of matrix relevance LVQ. In: Proceedings of IJCNN 2015. p. to appear (2015)
Biehl, M., Hammer, B., Verleysen, M., Villmann, T. (eds.): Similarity based clustering - recent developments and biomedical applications. In: Lecture Notes in Artificial Intelligence, vol. 5400. Springer (2009)
Bojer, T., Hammer, B., Schunk, D., von Toschanowitz, K.T.: Relevance determination in Learning Vector Quantization. In: Verleysen, M. (ed.) European Symposium on Artificial Neural Networks, pp. 271–276 (2001)
Bunte, K., Schleif, F.M., Biehl, M.: Adaptive learning for complex-valued data. Proceedings of ESANN 2012, 387–392 (2012)
Bunte, K., Schneider, P., Hammer, B., Schleif, F.M., Villmann, T., Biehl, M.: Limited rank matrix learning, discriminative dimension reduction and visualization. Neural Netw. 26, 159–173 (2012)
Cover, T., Hart, P.: Nearest neighbor pattern classification. IEEE Trans. Inf. Theory 13(1), 21–27 (1967)
Crammer, K., Gilad-Bachrach, R., Navot, A., Tishby, A.: Margin analysis of the LVQ algorithm. In: Advances in Neural Information Processing Systems, vol. 15, pp. 462–469. MIT Press, Cambridge, MA (2003)
Gisbrecht, A., Hammer, B., Mokbel, B., Sczyrba, A.: Nonlinear dimensionality reduction for cluster identification in metagenomic samples. In: IV, pp. 174–179 (2013)
Hammer, B., Villmann, T.: Generalized relevance learning vector quantization. Neural Netw. 15(8–9), 1059–1068 (2002)
Kaden, M., Lange, M., Nebel, D., Riedel, M., Geweniger, T., Villmann, T.: Aspects in classification learning—review of recent developments in learning vector quantization. Found. Comput. Decision Sci. 39, 79–105 (2014)
Kästner, M., Nebel, D., Riedel, M., Biehl, M., Villmann, T.: Differentiable kernels in generalized matrix learning vector quantization. In: 11th International Conference on Machine Learning and Applications, ICMLA, pp. 132–137. IEEE (2012). http://dx.doi.org/10.1109/ICMLA.2012.231
Kohonen, T.: Learning Vector Quantization for pattern recognition. Technical report TKK-F-A601, Helsinki Univeristy of Technology, Espoo, Finland (1986)
Kohonen, T.: Self-Organizing Maps. Springer, Berlin, Heidelberg (1997)
Lee, J.A., Verleysen, M.: Nonlinear Dimensionality Reduction. Springer, Berlin (2007)
van der Maaten, L.J.P., Hinton, G.: Visualizing High-Dimensional Data Using t-SNE (2008)
Mendenhall, M.J., Merényi, E.: Relevance-based feature extraction for hyperspectral images. IEEE Transactions on Neural Networks. 19(4), 658–672 (2008)
Micchelli, C.A., Xu, Y., Zhang, H.: Universal kernels. J. Mach. Learn. Res. 6, 2651–2667 (2006). http://www.jmlr.org/papers/v7/micchelli06a.html
Mylavarapu, S., Kaban, A.: Random projections versus random selection of features for classification of high dimensional data. In: UKCI, pp. 305–312. IEEE (2013)
Newman, D.J., Hettich, S., Blake, C.L., Merz, C.J.: UCI repository of machine learning databases. http://archive.ics.uci.edu/ml/ (1998)
Oja, E.: Neural networks, principal components, and subspaces. J. Neural Syst. 1, 61–68 (1989)
Pöllä, M., Honkela, T., Kohonen, T.: Bibliography of self-organizing map (som) papers: 2002–2005 addendum. TKK Reports in Information and Computer Science, Helsinki University of Technology Report TKK-ICS-R23 (2009)
Sato, A., Yamada, K.: Generalized learning vector quantization. In: Touretzky, D.S., Mozer, M.C., Hasselmo, M.E. (eds.) Advances in Neural Information Processing Systems, vol. 8. Proceedings of the 1995 Conference, pp. 423–429. MIT Press, Cambridge, MA, USA (1996)
Schleif, F.M., Villmann, T., Hammer, B.: Prototype based fuzzy classification in clinical proteomics. Int. J. Approx. Reasoning 47(1), 4–16 (2008)
Schneider, P., Biehl, M., Hammer, B.: Adaptive relevance matrices in learning vector quantization. Neural Comput. 21(12), 3532–3561 (2009)
Schneider, P., Schleif, F.M., Villmann, T., Biehl, M.: Generalized matrix learning vector quantizer for the analysis of spectral data. In: Verleysen, M. (ed.) European Symposium on Artifiacal Neural Networks. Bruges, Belgium, Apr 2008
Schölkopf, B.: The kernel trick for distances. In: Advances in Neural Information Processing Systems, vol. 13, Papers from Neural Information Processing Systems (NIPS), pp. 301–307 (2000)
Strickert, M., Witzel, K., Mock, H.P., Schleif, F.M., Villmann, T.: Supervised attribute relevance determination for protein identification in stress experiments. In: Proceedings of Machine Learning in Systems Biology (2007)
Villmann, T., Haase, S., Kaden, M.: Kernelized vector quantization in gradient-descent learning. Neurocomputing 147, 83–95 (2015). http://dx.doi.org/10.1016/j.neucom.2013.11.048
Villmann, T., Schleif, F.M., Hammer, B.: Comparison of relevance learning vector quantization with other metric adaptive classification methods. Neural Netw. 19, 610–622 (2006)
Weinberger, K., Blitzer, J., Saul, L.: Distance metric learning for large margin nearest neighbor classification. In: Weiss, Y., Schölkopf, B., Platt, J. (eds.) Advances in Neural Information Processing Systems 18, pp. 1473–1480. MIT Press, Cambridge, MA (2006)
Weston, J., Mukherjee, S., Chapelle, O., Pontil, M., Poggio, T., Vapnik, V.: Feature selection for SVMs. In: Leen, T.K., Dietterich, T.G., Tresp, V. (eds.) NIPS, pp. 668–674. MIT Press (2000)
Acknowledgments
Marie Curie Intra-European Fellowship (IEF): FP7-PEOPLE-2012-IEF (FP7-327791-ProMoS) is greatly acknowledged. This work has been partially funded by the Belgian FRS-FNRS project 7.0175.13 DRedVis.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Bunte, K., Kaden, M., Schleif, FM. (2016). Low-Rank Kernel Space Representations in Prototype Learning. In: Merényi, E., Mendenhall, M., O'Driscoll, P. (eds) Advances in Self-Organizing Maps and Learning Vector Quantization. Advances in Intelligent Systems and Computing, vol 428. Springer, Cham. https://doi.org/10.1007/978-3-319-28518-4_30
Download citation
DOI: https://doi.org/10.1007/978-3-319-28518-4_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-28517-7
Online ISBN: 978-3-319-28518-4
eBook Packages: EngineeringEngineering (R0)