Abstract
We study the problem of classifying data in a given taxonomy when classifications associated with multiple and/or partial paths are allowed. We introduce an incremental algorithm using a linear-threshold classifier at each node of the taxonomy. These classifiers are trained and evaluated in a hierarchical top-down fashion. We then define a hierachical and parametric data model and prove a bound on the probability that our algorithm guesses the wrong multilabel for a random instance compared to the same probability when the true model parameters are known. Our bound decreases exponentially with the number of training examples and depends in a detailed way on the interaction between the process parameters and the taxonomy structure. Preliminary experiments on real-world data provide support to our theoretical results.
The first and third author gratefully acknowledge partial support by the PASCAL Network of Excellence under EC grant no. 506778.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Azoury, K.S., Warmuth, M.K.: Relative loss bounds for on-line density estimation with the exponential familiy of distributions. Machine Learning 43(3), 211–246 (2001)
Cesa-Bianchi, N., Conconi, A., Gentile, C.: A second-order perceptron algorithm. In: Kivinen, J., Sloan, R.H. (eds.) COLT 2002. LNCS (LNAI), vol. 2375, p. 121. Springer, Heidelberg (2002)
Cesa-Bianchi, N., Conconi, A., Gentile, C.: Learning probabilistic linear-threshold classifiers via selective sampling. In: Schölkopf, B., Warmuth, M.K. (eds.) COLT/Kernel 2003. LNCS (LNAI), vol. 2777, pp. 373–387. Springer, Heidelberg (2003)
Devroye, L., Györfi, L., Lugosi, G.: A Probabilistic Theory of Pattern Recognition. Springer, Heidelberg (1996)
Dumais, S.T., Chen, H.: Hierarchical classification of web content. In: Proceedings of the 23rd ACM International Conference on Research and Development in Information Retrieval, pp. 256–263. ACM Press, New York (2000)
Granitzer, M.: Hierarchical Text Classification using Methods from Machine Learning. PhD thesis, Graz University of Technology (2003)
Hofmann, T., Cai, L., Ciaramita, M.: Learning with taxonomies: classifying documents and words. In: Nips 2003: Workshop on syntax, semantics, and statistics (2003)
Hoeffding, W.: Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association 58, 13–30 (1963)
Horn, R.A., Johnson, C.R.: Matrix Analysis. Cambridge University Press, Cambridge (1985)
Kschischang, F.R., Frey, B.J., Loeliger, H.: Factor graphs and the sum-product algorithm. IEEE Trans. of Information Theory 47(2), 498–519 (2001)
Koller, D., Sahami, M.: Hierarchically classifying documents using very few words. In: Proc. 14th ICML, pp. 170–178. Morgan Kaufmann Publishers, San Francisco (1997)
McCallum, A.K., Rosenfeld, R., Mitchell, T.M., Ng, A.Y.: Improving text classification by shrinkage in a hierarchy of classes. In: Proc. 15th ICML, pp. 359–367. Morgan Kaufmann Publishers, San Francisco (1998)
Mladenic, D.: Turning yahoo into an automatic web-page classifier. In: Proc. 13th European Conference on Artificial Intelligence, pages, pp. 473–474 (1998)
Novikov, A.B.J.: On convergence proofs on perceptrons. In: Proc. of the Symposium on the Mathematical Theory of Automata, vol. XII, pp. 615–622 (1962)
Rifkin, R., Yeo, G., Poggio, T.: Regularized least squares classification. In: Advances in Learning Theory: Methods, Model and Applications. NATO Science Series III: Computer and Systems Sciences, volume 190, pp. 131–153. IOS Press, Amsterdam (2003)
Rosenblatt, F.: The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review 65, 386–408 (1958)
Ruiz, M.E., Srinivasan, P.: Hierarchical text categorization using neural networks. Information Retrieval 5(1), 87–118 (2002)
Shawe-Taylor, J., Williams, C., Cristianini, N., Kandola, J.S.: On the eigenspectrum of the gram matrix and its relationship to the operator eigenspectrum. In: Cesa-Bianchi, N., Numao, M., Reischuk, R. (eds.) ALT 2002. LNCS (LNAI), vol. 2533, pp. 23–40. Springer, Heidelberg (2002)
Sun, A., Lim, E.P.: Hierarchical text classification and evaluation. In: Proc. 2001 International Conference on Data Mining, pp. 521–528. IEEE Press, Los Alamitos (2001)
Vovk, V.: Competitive on-line statistics. International Statistical Review 69, 213–248 (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Cesa-Bianchi, N., Conconi, A., Gentile, C. (2004). Regret Bounds for Hierarchical Classification with Linear-Threshold Functions. In: Shawe-Taylor, J., Singer, Y. (eds) Learning Theory. COLT 2004. Lecture Notes in Computer Science(), vol 3120. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-27819-1_7
Download citation
DOI: https://doi.org/10.1007/978-3-540-27819-1_7
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-22282-8
Online ISBN: 978-3-540-27819-1
eBook Packages: Springer Book Archive