Computer Vision for Medical Infant Motion Analysis: State of the Art and RGB-D Data Set

  • Nikolas HesseEmail author
  • Christoph Bodensteiner
  • Michael Arens
  • Ulrich G. Hofmann
  • Raphael Weinberger
  • A. Sebastian Schroeder
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11134)


Assessment of spontaneous movements of infants lets trained experts predict neurodevelopmental disorders like cerebral palsy at a very young age, allowing early intervention for affected infants. An automated motion analysis system requires to accurately capture body movements, ideally without markers or attached sensors to not affect the movements of infants. A vast majority of recent approaches for human pose estimation focuses on adults, leading to a degradation of accuracy if applied to infants. Hence, multiple systems for infant pose estimation have been developed. Due to the lack of publicly available benchmark data sets, a standardized evaluation, let alone a comparison of different approaches is impossible. We fill this gap by releasing the Moving INfants In RGB-D (MINI-RGBD) (Data set available for research purposes at data set, created using the recently introduced Skinned Multi-Infant Linear body model (SMIL). We map real infant movements to the SMIL model with realistic shapes and textures, and generate RGB and depth images with precise ground truth 2D and 3D joint positions. We evaluate our data set with state-of-the-art methods for 2D pose estimation in RGB images and for 3D pose estimation in depth images. Evaluation of 2D pose estimation results in a PCKh rate of 88.1% and 94.5% (depending on correctness threshold), and PCKh rates of 64.2%, respectively 90.4% for 3D pose estimation. We hope to foster research in medical infant motion analysis to get closer to an automated system for early detection of neurodevelopmental disorders.


Motion analysis Infants Pose estimation RGB-D Data set Cerebral palsy 


  1. 1.
    Adde, L., Helbostad, J.L., Jensenius, A.R., Taraldsen, G., Støen, R.: Using computer-based video analysis in the study of fidgety movements. Early Human Dev. 85(9), 541–547 (2009)CrossRefGoogle Scholar
  2. 2.
    Andriluka, M., Pishchulin, L., Gehler, P., Schiele, B.: 2D human pose estimation: new benchmark and state of the art analysis. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3686–3693 (2014)Google Scholar
  3. 3.
    Bogo, F., Black, M.J., Loper, M., Romero, J.: Detailed full-body reconstructions of moving people from monocular RGB-D sequences. In: 2015 IEEE International Conference on Computer Vision (ICCV) (2015)Google Scholar
  4. 4.
    Bogo, F., Romero, J., Loper, M., Black, M.J.: FAUST: dataset and evaluation for 3D mesh registration. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014)Google Scholar
  5. 5.
    Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2D pose estimation using part affinity fields. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1302–1310 (2017)Google Scholar
  6. 6.
    Cenci, A., Liciotti, D., Frontoni, E., Zingaretti, P., Carinelli, V.P.: Movements analysis of preterm infants by using depth sensor. In: International Conference on Internet of Things and Machine Learning (IML 2017) (2017)Google Scholar
  7. 7.
    Chen, H., Xue, M., Mei, Z., Bambang Oetomo, S., Chen, W.: A review of wearable sensor systems for monitoring body movements of neonates. Sensors 16(12), 2134 (2016)CrossRefGoogle Scholar
  8. 8.
    Chen, L., Wei, H., Ferryman, J.: A survey of human motion analysis using depth imagery. Pattern Recognit. Lett. 34(15), 1995–2006 (2013)CrossRefGoogle Scholar
  9. 9.
    Fan, M., Gravem, D., Cooper, D.M., Patterson, D.J.: Augmenting gesture recognition with erlang-cox models to identify neurological disorders in premature babies. In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pp. 411–420. ACM (2012)Google Scholar
  10. 10.
    Gravem, D., et al.: Assessment of infant movement with a compact wireless accelerometer system. J. Med. Devices 6(2), 021013 (2012)CrossRefGoogle Scholar
  11. 11.
    Hadders-Algra, M., Nieuwendijk, A.W., Maitijn, A., Eykern, L.A.: Assessment of general movements: towards a better understanding of a sensitive method to evaluate brain function in young infants. Dev. Med. Child Neurol. 39(2), 88–98 (1997)CrossRefGoogle Scholar
  12. 12.
    Haque, A., Peng, B., Luo, Z., Alahi, A., Yeung, S., Fei-Fei, L.: Towards viewpoint invariant 3D human pose estimation. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 160–177. Springer, Cham (2016). Scholar
  13. 13.
    Heinze, F., Hesels, K., Breitbach-Faller, N., Schmitz-Rode, T., Disselhorst-Klug, C.: Movement analysis by accelerometry of newborns and infants for the early detection of movement disorders due to infantile cerebral palsy. Med. Biol. Eng. Comput. 48(8), 765–772 (2010)CrossRefGoogle Scholar
  14. 14.
    Hesse, N., et al.: Learning an infant body model from RGB-D data for accurate full body motion analysis. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger G. (eds.) Medical Image Computing and Computer Assisted Intervention - MICCAI 2018, pp. 792–800. Springer, Cham (2018)CrossRefGoogle Scholar
  15. 15.
    Hesse, N., Schröder, A.S., Müller-Felber, W., Bodensteiner, C., Arens, M., Hofmann, U.G.: Body pose estimation in depth images for infant motion analysis. In: IEEE 39th Annual International Conference of the Engineering in Medicine and Biology Society (EMBC) (2017)Google Scholar
  16. 16.
    Hesse, N., Stachowiak, G., Breuer, T., Arens, M.: Estimating body pose of infants in depth images using random ferns. In: 2015 IEEE International Conference on Computer Vision Workshop (ICCVW) (2015)Google Scholar
  17. 17.
    Iarlori, S., Ferracuti, F., Giantomassi, A., Longhi, S.: RGBD camera monitoring system for Alzheimer’s disease assessment using recurrent neural networks with parametric bias action recognition. IFAC Proc. Vol. 47(3), 3863–3868 (2014)CrossRefGoogle Scholar
  18. 18.
    Jiang, C., Lane, C.J., Perkins, E., Schiesel, D., Smith, B.A.: Determining if wearable sensors affect infant leg movement frequency. Dev. Neurorehabil. 21, 1–4 (2017)Google Scholar
  19. 19.
    Kanemaru, N., et al.: Specific characteristics of spontaneous movements in preterm infants at term age are associated with developmental delays at age 3 years. Dev. Med. Child Neurol. 55(8), 713–721 (2013)Google Scholar
  20. 20.
    Karch, D., et al.: Kinematic assessment of stereotypy in spontaneous movements in infants. Gait Posture 36(2), 307–311 (2012)CrossRefGoogle Scholar
  21. 21.
    Karch, D., Kim, K.S., Wochner, K., Pietz, J., Dickhaus, H., Philippi, H.: Quantification of the segmental kinematics of spontaneous infant movements. J. Biomech. 41(13), 2860–2867 (2008)CrossRefGoogle Scholar
  22. 22.
    Kontschieder, P., et al.: Quantifying progression of multiple sclerosis via classification of depth videos. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8674, pp. 429–437. Springer, Cham (2014). Scholar
  23. 23.
    Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: a skinned multi-person linear model. ACM Trans. Graph. 34(6), 248:1–248:16 (2015)CrossRefGoogle Scholar
  24. 24.
    Loper, M.M., Black, M.J.: OpenDR: an approximate differentiable renderer. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 154–169. Springer, Cham (2014). Scholar
  25. 25.
    Machireddy, A., van Santen, J., Wilson, J.L., Myers, J., Hadders-Algra, M., Song, X.: A video/IMU hybrid system for movement estimation in infants. In: 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 730–733. IEEE (2017)Google Scholar
  26. 26.
    Marcroft, C., Khan, A., Embleton, N.D., Trenell, M., Plötz, T.: Movement recognition technology as a method of assessing spontaneous general movements in high risk infants. Front. Neurol. 5, 284 (2014).
  27. 27.
    Meinecke, L., Breitbach-Faller, N., Bartz, C., Damen, R., Rau, G., Disselhorst-Klug, C.: Movement analysis in the early detection of newborns at risk for developing spasticity due to infantile cerebral palsy. Hum. Mov. Sci. 25(2), 125–144 (2006)CrossRefGoogle Scholar
  28. 28.
    Morrison, C., Culmer, P., Mentis, H., Pincus, T.: Vision-based body tracking: turning Kinect into a clinical tool. Disabil. Rehabil.: Assist. Technol. 11(6), 516–520 (2016)Google Scholar
  29. 29.
    Olsen, M.D., Herskind, A., Nielsen, J.B., Paulsen, R.R.: Model-based motion tracking of infants. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) ECCV 2014. LNCS, vol. 8927, pp. 673–685. Springer, Cham (2015). Scholar
  30. 30.
    Olsen, M.D., Herskindt, A., Nielsen, J.B., Paulsen, R.R.: Body part tracking of infants. In: 22nd International Conference on Pattern Recognition (ICPR), pp. 2167–2172. IEEE (2014)Google Scholar
  31. 31.
    OpenPose Library: Accessed June 2018
  32. 32.
    Prechtl, H.F., Einspieler, C., Cioni, G., Bos, A.F., Ferrari, F., Sontheimer, D.: An early marker for neurological deficits after perinatal brain lesions. Lancet 349(9062), 1361–1363 (1997)CrossRefGoogle Scholar
  33. 33.
    Prechtl, H.: Qualitative changes of spontaneous movements in fetus and preterm infant are a marker of neurological dysfunction. Early Human Dev. 23(3), 151–158 (1990)CrossRefGoogle Scholar
  34. 34.
    Rahmati, H., Dragon, R., Aamo, O.M., Adde, L., Stavdahl, Ø., Van Gool, L.: Weakly supervised motion segmentation with particle matching. Comput. Vis. Image Underst. 140, 30–42 (2015)CrossRefGoogle Scholar
  35. 35.
    Rihar, A., Mihelj, M., Pašič, J., Kolar, J., Munih, M.: Infant trunk posture and arm movement assessment using pressure mattress, inertial and magnetic measurement units (IMUs). J. Neuroeng. Rehabil. 11(1), 133 (2014)CrossRefGoogle Scholar
  36. 36.
    Sciortino, G., Farinella, G.M., Battiato, S., Leo, M., Distante, C.: On the estimation of children’s poses. In: Battiato, S., Gallo, G., Schettini, R., Stanco, F. (eds.) ICIAP 2017. LNCS, vol. 10485, pp. 410–421. Springer, Cham (2017). Scholar
  37. 37.
    Serrano, M.M., Chen, Y.P., Howard, A., Vela, P.A.: Lower limb pose estimation for monitoring the kicking patterns of infants. In: 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 2157–2160. IEEE (2016)Google Scholar
  38. 38.
    Shivakumar, S.S., et al.: Stereo 3D tracking of infants in natural play conditions. In: International Conference on Rehabilitation Robotics (ICORR), pp. 841–846. IEEE (2017)Google Scholar
  39. 39.
    Simon, T., Joo, H., Matthews, I., Sheikh, Y.: Hand keypoint detection in single images using multiview bootstrapping. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4645–4653 (2017)Google Scholar
  40. 40.
    Singh, M., Patterson, D.J.: Involuntary gesture recognition for predicting cerebral palsy in high-risk infants. In: International Symposium on Wearable Computers (ISWC), pp. 1–8. IEEE (2010)Google Scholar
  41. 41.
    Spittle, A., Orton, J., Anderson, P.J., Boyd, R., Doyle, L.W.: Early developmental intervention programmes provided post hospital discharge to prevent motor and cognitive impairment in preterm infants. Cochrane Database Syst. Rev. 11 (2015).
  42. 42.
    Stahl, A., Schellewald, C., Stavdahl, Ø., Aamo, O.M., Adde, L., Kirkerød, H.: An optical flow-based method to predict infantile cerebral palsy. IEEE Trans. Neural Syst. Rehabil. Eng. 20(4), 605–614 (2012)CrossRefGoogle Scholar
  43. 43.
    Sun, B., Liu, X., Wu, X., Wang, H.: Human gait modeling and gait analysis based on Kinect. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 3173–3178. IEEE (2014)Google Scholar
  44. 44.
    Wei, S.E., Ramakrishna, V., Kanade, T., Sheikh, Y.: Convolutional pose machines. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4724–4732 (2016)Google Scholar
  45. 45.
    Zimmermann, C., Welschehold, T., Dornhege, C., Burgard, W., Brox, T.: 3D human pose estimation in RGBD images for robotic task learning. In: IEEE International Conference on Robotics and Automation (ICRA) (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Fraunhofer Institute of Optronics, System Technologies and Image Exploitation IOSBEttlingenGermany
  2. 2.University Medical Center Freiburg, Faculty of MedicineUniversity of FreiburgFreiburg im BreisgauGermany
  3. 3.Ludwig Maximilian University, Hauner Children’s HospitalMunichGermany

Personalised recommendations