Mightability: A Multi-state Visuo-spatial Reasoning for Human-Robot Interaction

  • Amit Kumar Pandey
  • Rachid Alami
Part of the Springer Tracts in Advanced Robotics book series (STAR, volume 79)


We, the Humans, are capable of estimating various abilities of ourselves and of the person we are interacting with. Visibility and reachability are among two such abilities. Studies in neuroscience and psychology suggest that from the age of 12-15 months children start to understand the occlusion of others line-of-sight and from the age of 3 years they start to develop the ability, termed as perceived reachability for self and for others. As such capabilities evolve in the children, they start showing intuitive and proactive behavior by perceiving various abilities of the human partner.

Inspired from such studies, which suggest that visuo-spatial perception plays an important role in Human-Human interaction, we propose to equip our robot to perceive various types of abilities of the agents in the workspace. The robot perceives such abilities not only from the current state of the agent but also by virtually putting an agent into various achievable states, such as turn left, stand up, etc. As the robot estimates what an agent might be able to ‘see’ and ‘reach’ if will be in a particular state, we term such analyses as Mightability Analyses. Currently the robot is equipped to perform such Mightability analyses at two levels: cells in the 3D grid and objects in the space, which we termed as Mightability Maps (MM) and Object Oriented Mightabilities (OOM) respectively.

We have shown the applications of Mightability analyses in performing various co-operative tasks like show and make an object accessible to the human as well as competitive tasks like hide and put away an object from the human. Such Mightability analyses equip the robot for higher-level learning and decisional capabilities as well as could facilitate the robot for better verbalize interaction and proactive behavior.


Humanoid Robot Object Orient Proactive Behavior Human Partner Human Robot Interaction 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Rochat, P.: Perceived reachability for self and for others by 3- to 5-year-old children and adults. Journal of Experimental Child Psychology 59(2), 317–333 (1995)CrossRefGoogle Scholar
  2. 2.
    Caron, A.J., Kiel, E.J., Dayton, M., Butler, S.C.: Comprehension of the referential intent of looking and pointing between 12 and 15 months. Journal of Cognition and Development 3(4), 445–464 (2002)Google Scholar
  3. 3.
    Dunphy-Lelii, S., Wellman, H.M.: Infants’ understanding of occlusion of others’ line-of-sight: Implications for an emerging theory of mind. European Journal of Developmental Psychology 1(1), 49–66 (2004)CrossRefGoogle Scholar
  4. 4.
    Deak, G.O., Flom, R.A., Pick, A.D.: Effects of gesture and target on 12-and18-month-olds’ joint visual attention to objects in front of or behind them. Developmental Psychology 36(4), 511–523 (2000)CrossRefGoogle Scholar
  5. 5.
    Moll, H., Tomasello, M.: 12-and18-month-old infants follow gaze to spaces behind barriers. Developmental Science 7(1), F1–F9 (2004)CrossRefGoogle Scholar
  6. 6.
    Zacharias, F., Borst, C., Hirzinger, G.: Capturing robot workspace structure: representing robot capabilities. In: Proceedings of the IEEE/RSJ IROS, San Diego, USA, pp. 3229–3236 (October 2007)Google Scholar
  7. 7.
    Zacharias, F., Borst, C., Hirzinger, G.: Online Generation of Reachable Grasps for Dexterous Manipulation Using a Representation of the Reachable Workspace. In: Proceedings of the International Conference on Advanced Robotics, Germany (2009)Google Scholar
  8. 8.
    Guilamo, L., Kuffner, J., Nishiwaki, K., Kagami, S.: Efficient prioritized inverse kinematic solutions for redundant manipulators. In: Proceedings of IEEE/RSJ IROS, pp. 1905–1910 (2005)Google Scholar
  9. 9.
    Guan, Y., Yokoi, K.: Reachable Space Generation of A Humanoid Robot Using The Monte Carlo Method. In: Proceedings of IEEE/RSJ IROS, pp. 1984–1989 (October 2006)Google Scholar
  10. 10.
    Breazeal, C., Berlin, M., Brooks, A., Gray, J., Thomaz, A.L.: Using perspective taking to learn from ambiguous demonstrations. In: Robotics and Autonomous Systems, pp. 385–393 (2006)Google Scholar
  11. 11.
    Johnson, M., Demiris, Y.: Perceptual Perspective Taking And Action Recognition. International Journal of Advanced Robotic Systems 2(4), 301–308 (2005)Google Scholar
  12. 12.
    Gregory Trafton, J., Cassimatis, N.L., Bugajska, M.D., Brock, D.P., Mintz, F., Schultz, A.C.: Enabling effective human-robot interaction using perspective-taking in robots. IEEE Transactions on Systems, Man, and Cybernetics, 460–470 (2005)Google Scholar
  13. 13.
    Marin-Urias, L.F., Sisbot, E.A., Pandey, A.K., Tadakuma, R., Alami, R.: Towards shared attention through geometric reasoning for human robot interaction. In: IEEE-RAS International Conference on Humanoid Robots, Paris, France, pp. 331–336 (2009)Google Scholar
  14. 14.
    Gardner, D.L., Mark, L.S., Ward, J.A., Edkins, H.: How do task characteristics affect the transitions between seated and standing reaches? Ecological Psychology 13, 245–274 (2001)CrossRefGoogle Scholar
  15. 15.
    Choi, H.J., Mark, L.S.: Scaling affordances for human reach actions. Human Movement Science 23, 785–806 (2004)CrossRefGoogle Scholar
  16. 16.
    Pandey, A.K., Alami, R.: Mightability Maps: A Perceptual Level Decisional Framework for Co-operative and Competitive Human-Robot Interaction. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Taiwan, pp. 5842–5848 (2010)Google Scholar
  17. 17.
    Carello, C., Grosofsky, A., Reichel, F.D., Solomon, H.Y., Turvey, M.T.: Visually Perceiving what is Reachable. Ecological Psychology 1(1), 27–54 (1989)CrossRefGoogle Scholar
  18. 18.
    Bootsma, R.J., Bakker, F.C., van Snippenberg, F.J., Tdlohreg, C.W.: The Effects of Anxiety on Perceiving the Reachability of Passing Objects. Ecological Psychology 4(1), 1–16 (1992)CrossRefGoogle Scholar
  19. 19.
    Simeon, T., Laumond, J.-P., Lamiraux, F.: Move3d: A generic platform for path planning. In: IEEE International Symposium on Assembly and Task Planning, Fukuoka, Japan, pp. 25–30 (2001)Google Scholar

Copyright information

© Springer-Verlag GmbH Berlin Heidelberg 2014

Authors and Affiliations

  • Amit Kumar Pandey
    • 1
    • 2
  • Rachid Alami
    • 1
    • 2
  1. 1.CNRS; LAASToulouseFrance
  2. 2.Université de Toulouse; UPS, INSA, INP, ISAE; LAASToulouseFrance

Personalised recommendations