Recent work on anatomical face modeling focuses mainly on facial muscles and their activation. This paper considers a different aspect of anatomical face modeling: kinematic modeling of the jaw, i.e., the temporomandibular joint (TMJ). Previous work often relies on simple models of jaw kinematics, even though the actual physiological behavior of the TMJ is quite complex, allowing not only for mouth opening, but also for some amount of sideways (lateral) and front-to-back (protrusion) motions. Fortuitously, the TMJ is the only joint whose kinematics can be accurately measured with optical methods, because the bones of the lower and upper jaw are rigidly connected to the lower and upper teeth. We construct a person-specific jaw kinematic model by asking an actor to exercise the entire range of motion of the jaw while keeping the lips open so that the teeth are at least partially visible. This performance is recorded with three calibrated cameras. We obtain highly accurate 3D models of the teeth with a standard dental scanner and use these models to reconstruct the rigid body trajectories of the teeth from the videos (markerless tracking). The relative rigid transformations samples between the lower and upper teeth are mapped to the Lie algebra of rigid body motions in order to linearize the rotational motion. Our main contribution is to fit these samples with a three-dimensional nonlinear model parameterizing the entire range of motion of the TMJ. We show that standard principal component analysis (PCA) fails to capture the nonlinear trajectories of the moving mandible. However, we found these nonlinearities can be captured with a special modification of autoencoder neural networks known as nonlinear PCA. By mapping back to the Lie group of rigid transformations, we obtain a parametrization of the jaw kinematics which provides an intuitive interface allowing the animators to explore realistic jaw motions in a user-friendly way.
This is a preview of subscription content, log in to check access.
Buy single article
Instant unlimited access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Abadi, M., Barham, P., Chen, J., et al.: Tensorflow: a system for large-scale machine learning. In: Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, pp. 265–283 (2016)
Aichert, A., Wein, W., Ladikos, A., Reichl, T., Navab, N.: Image-based tracking of the teeth for orthodontic augmented reality. In: Proceedings of the 15th International Conference on Medical Image Computing and Computer-Assisted Intervention, Part II, pp. 601–608 (2012)
Alexa, M.: Linear combination of transformations. ACM Trans. Graph. 21(3), 380–387 (2002)
Bai, X., Wang, J., Simons, D., Sapiro, G.: Video snapcut: robust video object cutout using localized classifiers. ACM Trans. Graph. 28(3), 70:1–70:11 (2009)
Cong, M., Bao, M., E, J.L., Bhat, K.S., Fedkiw, R.: Fully automatic generation of anatomical face simulation models. In: Proceedings of the 14th ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 175–183 (2015)
Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)
De Zee, M., Dalstra, M., Cattaneo, P.M., Rasmussen, J., Svensson, P., Melsen, B.: Validation of a musculo-skeletal model of the mandible and its application to mandibular distraction osteogenesis. J. Biomech. 40(6), 1192–1201 (2007)
Gold, S., Rangarajan, A., Lu, C.P., Pappu, S., Mjolsness, E.: New algorithms for 2D and 3D point matching: pose estimation and correspondence. Pattern Recognit. 31(8), 1019–1031 (1998)
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)
Hannam, A.G., Stavness, I., Lloyd, J.E., Fels, S.: A dynamic model of jaw and hyoid biomechanics during chewing. J. Biomech. 41(5), 1069–1076 (2008)
Hannam, A.G., Stavness, I.K., Lloyd, J.E., Fels, S.S., Miller, A.J., Curtis, D.A.: A comparison of simulated jaw dynamics in models of segmental mandibular resection versus resection with alloplastic reconstruction. J. Prosthet. Dent. 104(3), 191–198 (2010)
Hastie, T., Stuetzle, W.: Principal curves. J. Am. Stat. Assoc. 84(406), 502–516 (1989)
Henderson, S.E., Desai, R., Tashman, S., Almarza, A.J.: Functional analysis of the rabbit temporomandibular joint using dynamic biplane imaging. J. Biomech. 47(6), 1360–1367 (2014)
Herda, L., Urtasun, R., Fua, P., Hanson, A.: An automatic method for determining quaternion field boundaries for ball-and-socket joint limits. In: Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition, pp. 95–100 (2002)
Ichim, A.E., Kadleček, P., Kavan, L., Pauly, M.: Phace: physics-based face modeling and animation. ACM Trans. Graph. 36(4), 153:1–153:14 (2017)
Kiefer, J.: Sequential minimax search for a maximum. Proc. Am. Math. Soc. 4(3), 502–506 (1953)
Klein, G., Murray, D.: Full-3D edge tracking with a particle filter. In: Proceedings of the British Machine Vision Conference, pp. 1119–1128 (2006)
Koolstra, J.H.: Dynamics of the human masticatory system. Crit. Rev. Oral Biol. Med. 13(4), 366–376 (2002)
Kozlov, Y., Bradley, D., Bächer, M., Thomaszewski, B., Beeler, T., Gross, M.: Enriching facial blendshape rigs with physical simulation. Comput. Graph. Forum 36(2), 75–84 (2017)
Kramer, M.: Nonlinear principal component analysis using autoassociative neural networks. AIChE J. 37(2), 233–243 (1991)
Lan, L., Cong, M., Fedkiw, R.: Lessons from the evolution of an anatomical facial muscle model. In: Proceedings of the ACM SIGGRAPH Digital Production Symposium, pp. 11:1–11:3 (2017)
Lepetit, V., Fua, P.: Monocular model-based 3D tracking of rigid objects: a survey. Found. Trends Comput. Graph. Vis. 1(1), 1–89 (2005)
Li, T., Bolkart, T., Black, M.J., Li, H., Romero, J.: Learning a model of facial shape and expression from 4D scans. ACM Trans. Graph. 36(6), 194 (2017)
Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)
Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proceedings of the 7th International Joint Conference on Artificial Intelligence, vol. 2, pp. 674–679 (1981)
Murray, R.M., Sastry, S.S., Zexiang, L.: A Mathematical Introduction to Robotic Manipulation. CRC Press, Boca Raton (1994)
Ramalingam, S., Bouaziz, S., Sturm, P., Brand, M.: SKYLINE2GPS: localization in urban canyons using omni-skylines. In: International Conference on Intelligent Robotics (IROS), pp. 3816–3823 (2010)
Rosenhahn, B., Perwass, C., Sommer, G.: Pose estimation of 3D free-form contours. Int. J. Comput. Vis. 62(3), 267–289 (2005)
Roweis, S.T., Saul, L.: Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500), 2323–2326 (2000)
Scholz, M., Fraunholz, M., Selbig, J.: Nonlinear principal component analysis: neural network models and applications. Lect. Notes Comput. Sci. Eng. 58, 44–67 (2008)
Scholz, M., Vigario, R.: Nonlinear PCA: a new hierarchical approach. In: Proceedings of European Symposium on Artificial Neural Networks, pp. 439–444 (2002)
Sifakis, E., Neverov, I., Fedkiw, R.: Automatic determination of facial muscle activations from sparse motion capture marker data. ACM Trans. Graph. 24(3), 417–425 (2005)
Sorkine-Hornung, O., Rabinovich, M.: Least-Squares Rigid Motion Using SVD. Technical Notes, ETHZ (2009)
Tenenbaum, J.B., de Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290(5500), 2319–2323 (2000)
Terzopoulos, D., Waters, K.: Physically-based facial modelling, analysis, and animation. Comput. Anim. Virtual Worlds 1(2), 73–80 (1990)
Villamil, M., Nedel, L., Freitas, C., Macq, B.: Simulation of the human TMJ behavior based on interdependent joints topology. Comput. Methods Progr. Biomed. 105(3), 217–232 (2012)
Wang, J., Suenaga, H., Hoshi, K., Yang, L., Kobayashi, E., Sakuma, I., Liao, H.: Augmented reality navigation with automatic marker-free image registration using 3-D image overlay for dental surgery. IEEE Trans. Biomed. Eng. 61(4), 1295–1304 (2014)
Wang, J., Suenaga, H., Yang, L., Kobayashi, E., Sakuma, I.: Video see-through augmented reality for oral and maxillofacial surgery. Int. J. Med. Robot. Comput. Assist. Surg. 13(2), e1754 (2017)
Wu, C., Bradley, D., Gross, M., Beeler, T.: An anatomically-constrained local deformation model for monocular face capture. ACM Trans. Graph. 35(4), 115:1–115:12 (2016)
Yoon, H.J., Baltali, E., Zhao, K.D., Rebellato, J., Kademani, D., An, K.N., Keller, E.E.: Kinematic study of the temporomandibular joint in normal subjects and patients following unilateral temporomandibular joint arthrotomy with metal fossa-eminence partial joint replacement. J. Oral Maxillofac. Surg. 65(8), 1569–1576 (2007)
Zhang, S., Gersdorff, N., Frahm, J.: Real-time magnetic resonance imaging of temporomandibular joint dynamics. Open Med. Imaging J. 5(1), 1–9 (2011)
Zoss, G., Bradley, D., Bérard, P., Beeler, T.: An empirical rig for jaw animation. ACM Trans. Graph. 37(4), 59:1–59:12 (2018)
This material is based upon work supported by the National Science Foundation under Grant Numbers IIS-1617172, IIS-1622360 and IIS-1764071. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. Wenwu Yang was partially funded by the NSF of China (U1609215 and 61472363). Daniel Sýkora was funded by the Fulbright Commission in the Czech Republic, the Technology Agency of the Czech Republic under research program TE01020415 (V3C—Visual Computing Competence Center), and the Grant Agency of the Czech Technical University in Prague (No. SGS17/215/OHK3/3T/18). We also gratefully acknowledge the support of Research Center for Informatics (No. CZ.02.1.01/0.0/0.0/16_019/0000765), Activision, Adobe, and Mitsubishi Electric Research Labs (MERL) as well as hardware donation from NVIDIA Corporation.
This study was funded by the National Science Foundation (IIS-1617172, IIS-1622360 and IIS-1764071), NSF of China (U1609215 and 61472363), the Fulbright Commission in the Czech Republic, the Technology Agency of the Czech Republic under research program TE01020415 (V3C-Visual Computing Competence Center), the Grant Agency of the Czech Technical University in Prague (No. SGS17/215/OHK3/3T/18), and Research Center for Informatics (No. CZ.02.1.01/0.0/0.0/16_19/0000765).
Conflict of interest
Daniel Sýkora has received research grants from the Fulbright Commission in the Czech Republic. Ladislav Kavan has received a hardware donation from NVIDIA Corporation. Wenwu Yang declares that he has no conflict of interest. Nathan Marshak declares that he has no conflict of interest. Srikumar Ramalingam declares that he has no conflict of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This work was done when Wenwu Yang was a visiting scholar at the University of Utah.
Electronic supplementary material
Below is the link to the electronic supplementary material.
About this article
Cite this article
Yang, W., Marshak, N., Sýkora, D. et al. Building anatomically realistic jaw kinematics model from data. Vis Comput 35, 1105–1118 (2019) doi:10.1007/s00371-019-01677-8
- Jaw kinematics
- Teeth motion