Auto-labelling of Markers in Optical Motion Capture by Permutation Learning

  • Saeed GhorbaniEmail author
  • Ali Etemad
  • Nikolaus F. Troje
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11542)


Optical marker-based motion capture is a vital tool in applications such as motion and behavioural analysis, animation, and biomechanics. Labelling, that is, assigning optical markers to the pre-defined positions on the body, is a time consuming and labour intensive post-processing part of current motion capture pipelines. The problem can be considered as a ranking process in which markers shuffled by an unknown permutation matrix are sorted to recover the correct order. In this paper, we present a framework for automatic marker labelling which first estimates a permutation matrix for each individual frame using a differentiable permutation learning model and then utilizes temporal consistency to identify and correct remaining labelling errors. Experiments conducted on the test data show the effectiveness of our framework.


Labelling Motion capture Computer animation Deep learning 


  1. 1.
    Adams, R.P., Zemel, R.S.: Ranking via Sinkhorn propagation. ArXiv, pp. 1106–1925 (2011)Google Scholar
  2. 2.
    Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13, 281–305 (2012)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Birkhoff, G.: Three observations on linear algebra. Univ. Nac. Tacuman, Rev. Ser. A 5, 147–151 (1946)MathSciNetzbMATHGoogle Scholar
  4. 4.
    Etemad, S.A., Arya, A.: Expert-driven perceptual features for modeling style and affect in human motion. IEEE Trans. Hum.-Mach. Syst. 46(4), 534–545 (2016)CrossRefGoogle Scholar
  5. 5.
    Han, S., Liu, B., Wang, R., Ye, Y., Twigg, C.D., Kin, K.: Online optical marker-based hand tracking with deep labels. ACM Trans. Graph. 37(4), 166 (2018)CrossRefGoogle Scholar
  6. 6.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  7. 7.
    Herda, L., Fua, P., Plänkers, R., Boulic, R., Thalmann, D.: Using skeleton-based tracking to increase the reliability of optical motion capture. Hum. Mov. Sci. 20(3), 313–341 (2001). Scholar
  8. 8.
    Holden, D.: Robust solving of optical motion capture data by denoising. ACM Trans. Graph. 38(1), 1–12 (2018). Scholar
  9. 9.
    Holzreiter, S.: Autolabeling 3D tracks using neural networks. Clin. Biomech. 20(1), 1–8 (2005)CrossRefGoogle Scholar
  10. 10.
    Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.: Human3.6M: large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1325–1339 (2014)CrossRefGoogle Scholar
  11. 11.
    Loper, M., Mahmood, N., Black, M.J.: MoSh: motion and shape capture from sparse markers. ACM Trans. Graph. 33(6), 220 (2014)CrossRefGoogle Scholar
  12. 12.
    Maycock, J., Röhlig, T., Schröder, M., Botsch, M., Ritter, H.: Fully automatic optical motion tracking using an inverse kinematics approach. In: IEEE/RAS International Conference on Humanoid Robots, pp. 2–7 (2015).
  13. 13.
    Meyer, J., Kuderer, M., Muller, J., Burgard, W.: Online marker labeling for fully automatic skeleton tracking in optical motion capture. In: Proceedings of IEEE International Conference on Robotics and Automation, pp. 5652–5657 (2014).
  14. 14.
    Pons-Moll, G., Romero, J., Mahmood, N., Black, M.J.: Dyna: a model of dynamic human shape in motion. ACM Trans. Graph. (TOG) 34(4), 120 (2015)CrossRefGoogle Scholar
  15. 15.
    Rezatofighi, S.H., et al.: Deep perm-set net: learn to predict sets with unknown permutation and cardinality using deep neural networks. arXiv preprint arXiv:1805.00613 (2018)
  16. 16.
    Santa Cruz, R., Fernando, B., Cherian, A., Gould, S.: DeepPermNet: visual permutation learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3949–3957 (2017)Google Scholar
  17. 17.
    Schubert, T., Gkogkidis, A., Ball, T., Burgard, W.: Automatic initialization for skeleton tracking in optical motion capture. In: Proceedings of IEEE International Conference on Robotics and Automation, pp. 734–739. IEEE (2015)Google Scholar
  18. 18.
    Sigal, L., Balan, A.O., Black, M.J.: HumanEva: synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. Int. J. Comput. Vis. 87(1–2), 4 (2010)CrossRefGoogle Scholar
  19. 19.
    Sinkhorn, R.: A relationship between arbitrary positive matrices and doubly stochastic matrices. Ann. Math. Stat. 35(2), 876–879 (1964)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Troje, N.F.: Retrieving information from human movement patterns. In: Shipley, T.F., Zacks, J.M. (eds.) Understanding Events: How Humans See, Represent, and Act on Events. Oxford University, New York, vol. 1, pp. 308–334 (2008)CrossRefGoogle Scholar
  21. 21.
    Yu, Q., Li, Q., Deng, Z.: Online motion capture marker labeling for multiple interacting articulated targets. Comput. Graph. Forum 26(3), 477–483 (2007). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.York UniversityTorontoCanada
  2. 2.Queen’s UniversityKingstonCanada

Personalised recommendations