Semantic Object Search Using Semantic Categories and Spatial Relations between Objects

  • Patricio Loncomilla
  • Marcelo Saavedra
  • Javier Ruiz-del-Solar
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8371)


In this work, a novel methodology for robots executing informed object search is proposed. It uses basic spatial relations, which are represented by simple-shaped probability distributions describing the spatial relations between objects in space. Complex spatial relations can be defined as weighted sums of basic spatial relations using co-occurrence matrices as weights. Spatial relation masks are an alternative representation defined by sampling spatial relation distributions over a grid. A Bayesian framework for informed object search using convolutions between observation likelihoods and spatial relation masks is also provided. A set of spatial relation masks for the objects “monitor”, “keyboard”, “system unit” and “router” were estimated by using images from Label-Me and Flickr. A total of 4,320 experiments comparing six object search algorithms were realized by using the simulator Player/Stage. Results show that the use of the proposed methodology has a detection rate of 73.9% that is more than the double of the detection rate of previous informed object search methods.


Semantic search Informed search Co-occurrence matrix 


  1. 1.
    Garvey, T.D.: Perceptual strategies for purposive vision, Technical report, SRI International, vol. 117 (1976)Google Scholar
  2. 2.
    Wixson, L., Ballard, D.: Using intermediate object to improve efficiency of visual search. Int. J. Comput. Vis. 18(3), 209–230 (1994)CrossRefGoogle Scholar
  3. 3.
    Ye, Y., Tsotsos, J.K.: Sensor Planning for 3D Object Search. Computer Vision and Image Understanding 73-2, 145–168 (1999)CrossRefGoogle Scholar
  4. 4.
    Shubina, K., Tsotsos, J.: Visual search for an object in a 3d environment using a mobile robot. Computer Vision and Image Understanding 114(5), 535–547 (2010)CrossRefGoogle Scholar
  5. 5.
    Kollar, T., Roy, N.: Utilizing object-object and object-scene context when planning to find things. In: Proc. of the 2009 IEEE Int. Conf. on Robotics and Automation, ICRA 2009 (2009)Google Scholar
  6. 6.
    Viswanathan, P., Meger, D., Southey, T., Little, J.J., Mackworth, A.: Automated Spatial-Semantic Modeling with Applications to Place Labeling and Informed Search. In: Proc. Canadian Conf. on Computer and Robot Vision (2009)Google Scholar
  7. 7.
    Russell, B., Torralba, A., Murphy, K., Freeman, W.: Labelme: A database and web-based tool for image annotation. International Journal of Computer Vision 77, 157–173 (2008)CrossRefGoogle Scholar
  8. 8.
    Kasper, A., Jäkel, R., Dillmann, R.: Using spatial relations of objects in real world scenes for scene structuring and scene understanding. In: Proc.15th Int. Conf. on Advanced Robotics, ICAR 2011, Tallinn, Estonia (2011)Google Scholar
  9. 9.
    Galindo, C., Saffiotti, A., Coradeschi, S., Buschka, P., Fernandez-Madrigal, J.A., Gonzalez, J.: Multi-hierarchical semantic maps for mobile robotics. In: Proc. 2005 IEEE/RSJ Int. Conf. onIntelligent Robots and Systems, IROS 2005, pp. 2278–2283 (2005)Google Scholar
  10. 10.
    Vasudevan, S., Gachter, S., Nguyen, V., Siegwart, R.: Cognitive maps for mobile robots-an object based approach. Robot. Auton. Syst. 55, 359–371 (2007)CrossRefGoogle Scholar
  11. 11.
    Lowe, D.G.: Distinctive Image Features from Scale-Invariant Keypoints. Int. Journal of Computer Vision 60, 91–110 (2004)CrossRefGoogle Scholar
  12. 12.
    Aydemir, A., Sjöö, K., Jensfelt, P.: Object search on a mobile robot using relational spatial information. In: Proc. 11th Int Conf. on Intelligent Autonomous Systems, IAS 2011 (2010)Google Scholar
  13. 13.

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Patricio Loncomilla
    • 1
  • Marcelo Saavedra
    • 1
  • Javier Ruiz-del-Solar
    • 1
  1. 1.Advanced Mining Technology Center & Dept. of Elect. Eng.Universidad de ChileChile

Personalised recommendations