Skip to main content
SpringerLink
Log in

Deep Semantic Abstractions of Everyday Human Activities

On Commonsense Representations of Human Interactions

  • Conference paper
  • First Online:
Book cover ROBOT 2017: Third Iberian Robotics Conference (ROBOT 2017)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 693))

Included in the following conference series:

Abstract

We propose a deep semantic characterisation of space and motion categorically from the viewpoint of grounding embodied human-object interactions. Our key focus is on an ontological model that would be adept to formalisation from the viewpoint of commonsense knowledge representation, relational learning, and qualitative reasoning about space and motion in cognitive robotics settings. We demonstrate key aspects of the space & motion ontology and its formalisation as a representational framework in the backdrop of select examples from a dataset of everyday activities. Furthermore, focussing on human-object interaction data obtained from RGBD sensors, we also illustrate how declarative (spatio-temporal) reasoning in the (constraint) logic programming family may be performed with the developed deep semantic abstractions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Multi-modal in this context refers to more than one aspect of space, e.g., topology, orientation, direction, distance, shape. Multi-domain denotes a mixed domain ontology involving points, line-segments, polygons, and regions of space, time, and space-time [23].

  2. 2.

    RGB-D Data (video, depth, body skeleton): We collect data using Microsoft Kinect v2 which provides RGB and depth data. The RGB stream has a resolution of 1920 \(\times \) 1080 pixel at 30 Hz and the depth sensor has a resolution of 512 \(\times \) 424 pixels at 30 Hz. Skeleton tracking can track up to 6 persons with 25 joints for each person. Further we use the point-cloud data to detect objects on the table using tabletop object segmentation.

  3. 3.

    ExpCog – http://www.commonsenserobotics.org

    openEASE – http://ease.informatik.uni-bremen.de/openease/.

References

  1. Aiello, M., Pratt-Hartmann, I.E., van Benthem, J.F.: Handbook of Spatial Logics. Springer, New York (2007)

    Book  MATH  Google Scholar 

  2. Bartels, G., Kresse, I., Beetz, M.: Constraint-based movement representation grounded in geometric features. In: Proceedings of the IEEE-RAS International Conference on Humanoid Robots, Atlanta, Georgia, USA, 15–17 October 2013

    Google Scholar 

  3. Bennett, B., Cohn, A.G., Torrini, P., Hazarika, S.M.: Describing rigid body motions in a qualitative theory of spatial regions. In: Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on on Innovative Applications of Artificial Intelligence, pp. 503–509 (2000)

    Google Scholar 

  4. Bennett, B., Cohn, A.G., Torrini, P., Hazarika, S.M.: A foundation for region-based qualitative geometry. In: Proceedings of the 14th European Conference on Artificial Intelligence, pp. 204–208 (2000)

    Google Scholar 

  5. Bhatt, M., Guesgen, H., Wölfl, S., Hazarika, S.: Qualitative spatial and temporal reasoning: Emerging applications, trends, and directions. Spat. Cogn. Comput. 11(1), 1–14 (2011)

    Google Scholar 

  6. Bhatt, M.: Reasoning about space, actions and change: A paradigm for applications of spatial reasoning. In: Qualitative Spatial Representation and Reasoning: Trends and Future Directions. IGI Global, USA (2012)

    Google Scholar 

  7. Bhatt, M.: Between sense and sensibility: Declarative narrativisation of mental models as a basis and benchmark for visuo-spatial cognition and computation focussed collaborative cognitive systems (2013). CoRR abs/1307.3040

  8. Bhatt, M., Dylla, F., Hois, J.: Spatio-terminological inference for the design of ambient environments. In: Proceedings of the 9th International Conference on Spatial Information Theory, COSIT 2009. Lecture Notes in Computer Science, Aber Wrac’h, France, 21–25 September 2009, vol. 5756, pp. 371–391. Springer (2009)

    Google Scholar 

  9. Bhatt, M., Kersting, K.: Semantic Interpretation of Multimodal Human Behaviour Data: Making Sense of Events, Activities. Processes, KI - Künstliche Intelligenz/Artificial Intelligence (2017)

    Google Scholar 

  10. Bhatt, M., Lee, J.H., Schultz, C.P.L.: CLP(QS): A declarative spatial reasoning framework. In: Proceedings of the 10th International Conference on Spatial Information Theory, COSIT 2011, Belfast, ME, USA, 12–16 September 2011, pp. 210–230 (2011)

    Google Scholar 

  11. Bhatt, M., Loke, S.: Modelling dynamic spatial systems in the situation calculus. Spat. Cogn. Comput. 8(1–2), 86–130 (2008). http://www.tandfonline.com/doi/abs/10.1080/13875860801926884

    Google Scholar 

  12. Bhatt, M., Schultz, C., Freksa, C.: The ‘Space’ in spatial assistance systems: conception, formalisation and computation. In: Tenbrink, T., Wiener, J., Claramunt, C. (eds.) Representing Space in Cognition: Interrelations of Behavior, Language, and Formal Models. Explorations in Language and Space. Oxford University Press (2013). ISBN 978-0-19-967991-1

    Google Scholar 

  13. Cohn, A.G., Renz, J.: Qualitative spatial reasoning. In: van Harmelen, F., Lifschitz, V., Porter, B. (eds.) Handbook of Knowledge Representation. Elsevier (2007)

    Google Scholar 

  14. Davis, E.: Pouring liquids: A study in commonsense physical reasoning. Artif. Intell. 172(12–13), 1540–1578 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  15. Davis, E.: How does a box work? a study in the qualitative dynamics of solid objects. Artif. Intell. 175(1), 299–345 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  16. Davis, E.: Qualitative spatial reasoning in interpreting text and narrative. Spat. Cogn. Comput. 13(4), 264–294 (2013)

    Google Scholar 

  17. Dubba, K.S.R., Bhatt, M., Dylla, F., Hogg, D.C., Cohn, A.G.: Interleaved inductive-abductive reasoning for learning complex event models. In: Muggleton, S., Tamaddoni-Nezhad, A., Lisi, F.A. (eds.) 21st International Conference on Inductive Logic Programming, ILP 2011, Revised Selected Papers. Lecture Notes in Computer Science, Windsor Great Park, UK, 31 July–3 August 2011, vol. 7207, pp. 113–129. Springer (2011)

    Google Scholar 

  18. Dubba, K.S.R., Cohn, A.G., Hogg, D.C., Bhatt, M., Dylla, F.: Learning relational event models from video. J. Artif. Intell. Res. (JAIR) 53, 41–90 (2015)

    MATH  MathSciNet  Google Scholar 

  19. Eppe, M., Bhatt, M.: Narrative based postdictive reasoning for cognitive robotics. In: 11th International Symposium on Logical Formalizations of Commonsense Reasoning, COMMONSENSE 2013 (2013)

    Google Scholar 

  20. Eschenbach, C., Schill, K.: Studying spatial cognition - a report on the DFG workshop on “The representation of motion”. KI 13(3), 57–58 (1999)

    Google Scholar 

  21. Guesgen, H.W.: Spatial reasoning based on Allen’s temporal logic. Technical Report TR-89-049, International Computer Science Institute Berkeley (1989)

    Google Scholar 

  22. Hayes, P.J.: Naive physics I: ontology for liquids. In: Hubbs, J.R., Moore, R.C. (eds.) Formal Theories of the Commonsense World. Ablex Publishing Corporation, Norwood (1985)

    Google Scholar 

  23. Hazarika, S.M.: Qualitative Spatial Change: Space-Time Histories and Continuity. Ph.D. thesis, The University of Leeds, School of Computing, supervisor - Anthony Cohn (2005)

    Google Scholar 

  24. Hernández, D., Clementini, E., Di Felice, P.: Qualitative distances. Springer (1995)

    Google Scholar 

  25. Moratz, R.: Representing relative direction as a binary relation of oriented points. In: ECAI, pp. 407–411 (2006)

    Google Scholar 

  26. Randell, D.A., Cui, Z., Cohn, A.G.: A spatial logic based on regions and connection. In: KR 1992, pp. 165–176 (1992)

    Google Scholar 

  27. Renz, J., Nebel, B.: Qualitative spatial reasoning using constraint calculi. In: Handbook of Spatial Logics, pp. 161–215 (2007)

    Google Scholar 

  28. Scivos, A., Nebel, B.: The finest of its class: the natural, point-based ternary calculus LR for qualitative spatial reasoning. In: Freksa, C., et al. International Conference Spatial Cognition on Spatial Cognition IV. Reasoning, Action, Interaction. Lecture Notes in Computer Science, Vol. 3343, pp. 283–303. Springer, Heidelberg (2004)

    Google Scholar 

  29. Spranger, M., Suchan, J., Bhatt, M.: Robust natural language processing - combining reasoning, cognitive semantics and construction grammar for spatial language. In: IJCAI 2016: 25th International Joint Conference on Artificial Intelligence. AAAI Press (2016)

    Google Scholar 

  30. Spranger, M., Suchan, J., Bhatt, M., Eppe, M.: Grounding dynamic spatial relations for embodied (robot) interaction. In: Proceedings of the PRICAI 2014: Trends in Artificial Intelligence, Gold Coast, QLD, Australia, 1–5 December 2014, pp. 958–971 (2014). http://dx.doi.org/10.1007/978-3-319-13560-1_83

  31. Suchan, J., Bhatt, M.: Commonsense scene semantics for cognitive robotics: towards grounding embodied visuo-locomotive interactions. In: ICCV 2017 Workshop: Vision in Practice on Autonomous Robots (ViPAR), International Conference on Computer Vision (ICCV) (2017)

    Google Scholar 

  32. Suchan, J., Bhatt, M., Santos, P.E.: Perceptual narratives of space and motion for semantic interpretation of visual data. In: Proceedings of the Computer Vision - ECCV 2014 Workshops - Zurich, Switzerland, 6–7 and 12 September 2014, Part II, pp. 339–354 (2014)

    Google Scholar 

  33. Tellex, S.: Natural language and spatial reasoning. Ph.D. thesis, Massachusetts Institute of Technology (2010)

    Google Scholar 

  34. Tyler, A., Evans, V.: The Semantics of English Prepositions: Spatial Scenes, Embodied Meaning and Cognition. Cambridge University Press, Cambridge (2003)

    Book  Google Scholar 

  35. Walega, P.A., Bhatt, M., Schultz, C.P.L.: ASPMT(QS): Non-monotonic spatial reasoning with answer set programming modulo theories. In: Proceedings of the 13th International Conference on Logic Programming and Nonmonotonic Reasoning, LPNMR 2015, Lexington, KY, USA, 27–30 September 2015, pp. 488–501 (2015)

    Google Scholar 

  36. Worgotter, F., Aksoy, E.E., Kruger, N., Piater, J., Ude, A., Tamosiunaite, M.: A simple ontology of manipulation actions based on hand-object relations (2012)

    Google Scholar 

Download references

Acknowledgements

We acknowledge funding by the Germany Research Foundation (DFG) via the Collaborative Research Center (CRC) EASE – Everyday Activity Science and Engineering (http://ease-crc.org). This paper builds on, and is a condensed version of, a workshop contribution [31] at the ICCV 2017 conference. We also acknowledge the support of Vijayanta Jain in preparation of part of the overall activity dataset; toward this, the help of Omar Moussa, Hubert Kloskoski, Thomas Hudkovic, Vyyom Kelkar as subjects is acknowledged.

Author information

Authors and Affiliations

  1. Spatial Reasoning, EASE CRC: Everyday Activity Science and Engineering, University of Bremen, Bremen, Germany

    Jakob Suchan & Mehul Bhatt

  2. Machine Perception and Interaction Laboratory, Centre for Applied Autonomous Sensor Systems (AASS), Örebro University, Örebro, Sweden

    Mehul Bhatt

Authors
  1. Jakob Suchan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mehul Bhatt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jakob Suchan .

Editor information

Editors and Affiliations

  1. Escuela Técnica Superior de Ingeniería, Universidad de Sevilla, Sevilla, Spain

    Anibal Ollero

  2. Institut de Robòtica I Informàtica Industrial (CSIC-UPC), Universitat Politècnica de Catalunya, Barcelona, Spain

    Alberto Sanfeliu

  3. Departamento de Informática e Ingeniería de Sistemas, Escuela de Ingeniería y Arquitectura, Instituto de Investigación en Ingeniería de Aragón, Zaragoza, Spain

    Luis Montano

  4. Institute of Electronics and Telematics Engineering of Aveiro (IEETA), Universidade de Aveiro, Aveiro, Portugal

    Nuno Lau

  5. IDMEC, Instituto Superior Técnico de Lisboa, Universidade de Lisboa, Lisbon, Portugal

    Carlos Cardeira

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Suchan, J., Bhatt, M. (2018). Deep Semantic Abstractions of Everyday Human Activities. In: Ollero, A., Sanfeliu, A., Montano, L., Lau, N., Cardeira, C. (eds) ROBOT 2017: Third Iberian Robotics Conference. ROBOT 2017. Advances in Intelligent Systems and Computing, vol 693. Springer, Cham. https://doi.org/10.1007/978-3-319-70833-1_39

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-70833-1_39

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-70832-4

  • Online ISBN: 978-3-319-70833-1

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation