Abstract
Multimodal scene understanding is an integral part of human-robot interaction (HRI) in situated environments. Especially useful is category-level recognition, where the the system can recognize classes of objects of scenes rather than specific instances (e.g., any chair vs. this particular chair.) Humans use multiple modalities to understand which object category is being referred to, simultaneously interpreting gesture, speech and visual appearance, and using one modality to disambiguate the information contained in the others. In this paper, we address the problem of fusing visual and acoustic information to predict object categories, when an image of the object and speech input from the user is available to the HRI system. Using probabilistic decision fusion, we show improved classification rates on a dataset containing a wide variety of object categories, compared to using either modality alone.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bolt, R.: “Put-that-there”: Voice and gesture at the graphics interface. In: SIGGRAPH 1980. Proceedings of the 7th Annual Conference on Computer Graphics and interactive Techniques, pp. 262–270. ACM Press, New York (1980)
Chang, C., Lin, C.: LIBSVM: A library for support vector machines, Software (2001), http://www.csie.ntu.edu.tw/~cjlin/libsvm
Fei-Fei, L., Fergus, R., Perona, P.: Learning generative visual models from few training examples: In: an incremental Bayesian approach tested on 101 object categories. IEEE. CVPR, Workshop on Generative-Model Based Vision (2004)
Fergus, R., Fei-Fei, L., Perona, P., Zisserman, A.: Learning Object Categories from Google’s Image Search. In: Proc. of the 10th Inter. Conf. on Computer Vision, ICCV (2005)
Frome, A., Singer, Y., Malik, J.: Image Retrieval and Recognition Using Local Distance Functions. In: Proceedings of Neural Information Processing Systems (NIPS) (2006)
Grauman, K., Darrell, T.: The Pyramid Match Kernel: Discriminative Classification with Sets of Image Features. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), Beijing, China (October 2005), http://people.csail.mit.edu/jjl/libpmk/
Grauman, K., Darrell, T.: Approximate Correspondences in High Dimensions. In: Proceedings of Advances in Neural Information Processing Systems (NIPS) (2006)
Grauman, K., Darrell, T.: Pyramid Match Kernels: Discriminative Classification with Sets of Image Features. MIT Technical Report MIT-CSAIL-TR-2006-020, 2006. The Journal of Machine Learning (2006) (to appear)
Haasch, A., Hofemann, N., Fritsch, J., Sagerer, G.: A multi-modal object attention system for a mobile robot, Intelligent Robots and Systems (2005)
Kaiser, E., Olwal, A., McGee, D., Benko, H., Corradini, A., Li, X., Cohen, P., Feiner, S.: Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality. In: Proceedings of the 5th International Conference on Multimodal Interfaces (ICMI) (2003)
Murphy, K., Torralba, A., Eaton, D., Freeman, W.T.: Object detection and localization using local and global features. In: Sicily workshop on object recognition, LNCS(unrefered) (2005)
Potamianos, G., Neti, C., Gravier, G., Garg, A., Senior, A.: Recent Advances in the Automatic Recognition of Audio-Visual Speech. In: Proc. IEEE (2003)
Roy, D., Gorniak, P., Mukherjee, N., Juster, J.: A Trainable Spoken Language Understanding System for Visual Object Selection. In: Proceedings of the International Conference of Spoken Language Processing (2002)
Russell, B., Torralba, A., Murphy, K., Freeman, W.T.: LabelMe: a database and web-based tool for image annotation. MIT AI LAB MEMO AIM-2005-025 (2005)
Zhang, H., Berg, A., Maire, M., Malik, J.: SVM-KNN: Discriminative Nearest Neighbor Classification for Visual Category Recognition. In: proceedings of CVPR (2006)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Saenko, K., Darrell, T. (2008). Object Category Recognition Using Probabilistic Fusion of Speech and Image Classifiers. In: Popescu-Belis, A., Renals, S., Bourlard, H. (eds) Machine Learning for Multimodal Interaction. MLMI 2007. Lecture Notes in Computer Science, vol 4892. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-78155-4_4
Download citation
DOI: https://doi.org/10.1007/978-3-540-78155-4_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-78154-7
Online ISBN: 978-3-540-78155-4
eBook Packages: Computer ScienceComputer Science (R0)