Humanoid Robotics: A Reference pp 1-36 | Cite as

# Sensor Calibration, Modeling, and Simulation

## Abstract

For humanoid robots, sensors are a critical aspect of the entire robot system because they are responsible for accurately perceiving the environment. Due to the cost of real experiments on humanoids, sensor modeling and simulation play important roles in robotics research. For applications in the real world, sensor calibration is the prerequisite for sensors to function properly, and this chapter begins with a review of general problems in sensor calibration, modeling, and simulation. An often used formulation of sensor calibration in robotics and computer vision is “AX = XB,” where *A*, *X*, and *B* are rigid-body transformations with *A* and *B* given from sensor measurements and *X* is the unknown rigid-body transformation to be calibrated. Many methods have been proposed to solve *X* given data streams of *A* and *B* under different scenarios. This chapter presents the most complete picture of the *AX* = *XB* solvers to date. First, a brief overview of the various important sensor calibration techniques is given and problems of interest are highlighted. Then, a detailed review of the various “AX = XB” algorithms is presented. The notations used in different algorithms are unified to show the interconnections between the selected methods in a straightforward way. Finally, the criterion for data selection and various error metrics are introduced, which are of critical importance for evaluating the performance of *AX* = *XB* solvers.

## Keywords

Sensor calibration Hand-eye calibration Humanoid robot Review## References

- 1.S. Thrun, W. Burgard, D. Fox,
*Probabilistic Robotics*(MIT press, Cambridge/Mass, 2005)zbMATHGoogle Scholar - 2.ARS. https://ars-project.readthedocs.org. Accessed 03 Dec 2017
- 3.N. Koenig, A. Howard, Design and use paradigms for gazebo, an open-source multi-robot simulator, in
*IEEE/RSJ International Conference on Intelligent Robots and Systems*, 2004, pp. 2149–2154Google Scholar - 4.C. Aguero, N. Koenig, I. Chen, H. Boyer, S. Peters, J. Hsu, B. Gerkey, S. Paepcke, J. Rivero, J. Manzo, E. Krotkov, G. Pratt, Inside the virtual robotics challenge: simulating real-time robotic disaster response. IEEE Trans. Autom. Sci. Eng.
**12**(2), 494–506 (2015)CrossRefGoogle Scholar - 5.G. Echeverria, S. Lemaignan, A. Degroote, S. Lacroix, M. Karg, P. Koch, C. Lesire, S. Stinckwich, Simulating complex robotic scenarios with morse, in
*SIMPAR*, 2012, pp. 197–208CrossRefGoogle Scholar - 6.Webots, http://www.cyberbotics.com. Commercial Mobile Robot Simulation Software
- 7.O. Michel, Webots: professional mobile robot simulation. Int. J. Adv. Robot. Syst.
**1**(1), 39–42 (2004)CrossRefGoogle Scholar - 8.E. Rohmer, S.P. Singh, M. Freese, V-rep: a versatile and scalable robot simulation framework, in
*2013 IEEE/RSJ International Conference on Intelligent Robots and Systems*(IEEE, 2013), pp. 1321–1326Google Scholar - 9.RoboDK, http://www.robodk.com. Accessed 03 Dec 2017
- 10.SimSpark, http://simspark.sourceforge.net/. Accessed 03 Dec 2017
- 11.CarMaker, https://ipg-automotive.com/products-services/simulation-software/carmaker/. Accessed 03 Dec 2017
- 12.PreScan, https://www.tassinternational.com/prescan. Accessed 03 Dec 2017
- 13.AdasWorks, https://adasworks.com. Accessed 03 Dec 2017
- 14.Google, https://www.google.com/selfdrivingcar. Accessed 03 Dec 2017
- 15.BMW, http://www.bmw.com. Accessed 03 Dec 2017
- 16.Tesla, https://www.teslamotors.com/presskit/autopilot. Accessed 03 Dec 2017
- 17.Uber, https://newsroom.uber.com/pittsburgh-self-driving-uber/. Accessed 03 Dec 2017
- 18.G. Champleboux, S. Lavallee, R. Szeliski, L. Brunie, From accurate range imaging sensor calibration to accurate model-based 3D object localization, in
*Computer Vision and Pattern Recognition*(IEEE, 1992), pp. 83–89Google Scholar - 19.Z. Zhang, R. Deriche, O. Faugeras, Q.-T. Luong, A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry. Artif. Intell.
**78**(1), 87–119 (1995)CrossRefGoogle Scholar - 20.R.I. Hartley, P. Sturm, Triangulation. Comput. Vis. Image Underst.
**68**(2), 146–157 (1997)CrossRefGoogle Scholar - 21.P.H. Torr, A. Zisserman, Mlesac: a new robust estimator with application to estimating image geometry. Comput. Vis. Image Underst.
**78**(1), 138–156 (2000)CrossRefGoogle Scholar - 22.F. Zhou, G. Zhang, Complete calibration of a structured light stripe vision sensor through planar target of unknown orientations. Image Vis. Comput.
**23**(1), 59–67 (2005)CrossRefGoogle Scholar - 23.V. Bychkovskiy, S. Megerian, D. Estrin, M. Potkonjak, A collaborative approach to in-place sensor calibration, in
*Proceedings of Second International Workshop Information Processing in Sensor Networks (IPSN 2003)*, Palo Alto, 22–23 April 2003 (Springer, 2003), pp. 301–316CrossRefGoogle Scholar - 24.J.M. Hollerbach, C.W. Wampler, The calibration index and taxonomy for robot kinematic calibration methods. Int. J. Robot. Res.
**15**(6), 573–591 (1996)CrossRefGoogle Scholar - 25.C.G. Atkeson, C.H. An, J.M. Hollerbach, Estimation of inertial parameters of manipulator loads and links. Int. J. Robot. Res.
**5**(3), 101–119 (1986)CrossRefGoogle Scholar - 26.J.J. Craig, P. Hsu, S.S. Sastry, Adaptive control of mechanical manipulators. Int. J. Robot. Res.
**6**(2), 16–28 (1987)CrossRefGoogle Scholar - 27.J.-J.E. Slotine, W. Li, On the adaptive control of robot manipulators. Int. J. Robot. Res.
**6**(3), 49–59 (1987)CrossRefGoogle Scholar - 28.R. Ortega, M.W. Spong, Adaptive motion control of rigid robots: a tutorial. Automatica
**25**(6), 877–888 (1989)MathSciNetCrossRefGoogle Scholar - 29.L.L. Whitcomb, A.A. Rizzi, D.E. Koditschek, Comparative experiments with a new adaptive controller for robot arms. IEEE Trans. Robot. Autom.
**9**(1), 59–70 (1993)CrossRefGoogle Scholar - 30.R. Tsai, Y. Lenz, A new technique for fully autonomous and efficient 3D robotics hand/eye calibration. IEEE Trans. Robot. Autom.
**5**(3), 345–358 (1989)CrossRefGoogle Scholar - 31.E. Mair, M. Fleps, M. Suppa, D. Burschka, Spatio-temporal initialization for IMU to camera registration, in
*IEEE International Conference on Robotics and Biomimetics*(IEEE, 2011), pp. 557–564Google Scholar - 32.J. Schmidt, F. Vogt, H. Niemann, Robust hand–eye calibration of an endoscopic surgery robot using dual quaternions, in
*Proceedings of the 25th DAGM Symposium on Pattern Recognition*, Magdeburg, Sept 2003 (Springer, 2003), pp. 548–556Google Scholar - 33.H. Malm, A. Heyden, A new approach to hand-eye calibration, in
*International Conference on Pattern Recognition*, vol. 1 (IEEE, 2000), pp. 525–529Google Scholar - 34.J. Heller, M. Havlena, T. Pajdla, A branch-and-bound algorithm for globally optimal hand-eye calibration, in
*IEEE Conference on Computer Vision and Pattern Recognition*(IEEE, 2012), pp. 1608–1615Google Scholar - 35.T. Ruland, T. Pajdla, L. Kruger, Globally optimal hand-eye calibration, in
*IEEE Conference on Computer Vision and Pattern Recognition*(IEEE, 2012), pp. 1035–1042Google Scholar - 36.H. Wu, W. Tizzano, T.T. Andersen, N.A. Andersen, O. Ravn, Hand-eye calibration and inverse kinematics of robot arm using neural network, in
*Robot Intelligence Technology and Applications 2*(Springer, Cham, 2014), pp. 581–591Google Scholar - 37.S.-J. Kim, M. Jeong, J. Lee, J. Lee, K. Kim, B. You, S. Oh, Robot head-eye calibration using the minimum variance method in
*IEEE International Conference on Robotics and Biomimetics (ROBIO)*(IEEE, 2010), pp. 1446–1451Google Scholar - 38.Y. Liu, Q. Wang, Y. Li, Calibration of a robot hand-eye system with a concentric circles target, in
*International Bhurban Conference on Applied Sciences and Technology*(IEEE, 2015), pp. 204–209Google Scholar - 39.Q. Ma, H. Li, G.S. Chirikjian, New probabilistic approaches to the ax = xb hand-eye calibration without correspondence, in
*2016 IEEE International Conference on Robotics and Automation*(IEEE, 2016), pp. 4365–4371Google Scholar - 40.H. Zhuang, Z.S. Roth, R. Sudhakar, Simultaneous robot/world and tool/flange calibration by solving homogeneous transformation equations of the form AX = YB. IEEE Trans. Robot. Autom.
**10**(4), 549–554 (1994)CrossRefGoogle Scholar - 41.F. Dornaika, R. Horaud, Simultaneous robot-world and hand-eye calibration. IEEE Trans. Robot. Autom.
**14**(4), 617–622 (1998)CrossRefGoogle Scholar - 42.R.L. Hirsh, G.N. DeSouza, A.C. Kak, An iterative approach to the hand-eye and base-world calibration problem, in
*IEEE International Conference on Robotics and Automation*, vol. 3 (IEEE, 2001), pp. 2171–2176Google Scholar - 43.A. Li, L. Wang, D. Wu, Simultaneous robot-world and hand-eye calibration using dual-quaternions and kronecker product. Int. J. Phys. Sci.
**5**(10), 1530–1536 (2010)Google Scholar - 44.F. Ernst, L. Richter, L. Matthäus, V. Martens, R. Bruder, A. Schlaefer, A. Schweikard, Non-orthogonal tool/flange and robot/world calibration. Int. J. Med. Rob. Comput. Assisted Surg.
**8**(4), 407–420 (2012)CrossRefGoogle Scholar - 45.J. Heller, D. Henrion, T. Pajdla, Hand-eye and robot-world calibration by global polynomial optimization, in
*IEEE International Conference on Robotics and Automation*(IEEE, 2014), pp. 3157–3164Google Scholar - 46.H. Li, Q. Ma, T. Wang, G.S. Chirikjian, Simultaneous hand-eye and robot-world calibration by solving the problem without correspondence. IEEE J. Robot. Autom. Lett.
**1**(1), 145–152 (2016)CrossRefGoogle Scholar - 47.J. Wang, L. Wu, M.Q.-H. Meng, H. Ren, Towards simultaneous coordinate calibrations for cooperative multiple robots, in
*IEEE/RSJ International Conference on Intelligent Robots and Systems*(IEEE, 2014), pp. 410–415Google Scholar - 48.Q. Ma, Z. Goh, G.S. Chirikjian, Probabilistic approaches to the AXB = YCZ calibration problem in multi-robot systems, in
*Proceedings of Robotics: Science and Systems*, 2016Google Scholar - 49.H. Chen, A screw motion approach to uniqueness analysis of head-eye geometry, in
*IEEE Conference on Computer Vision and Pattern Recognition Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’91)*, (IEEE, 1991) pp. 145–151Google Scholar - 50.F. Park, B. Martin, Robot sensor calibration: solving AX = XB on the Euclidean group. IEEE Trans. Robot. Autom.
**10**(5), 717–721 (1994)CrossRefGoogle Scholar - 51.Y. Shiu, S. Ahmad, Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form AX = XB. IEEE Trans. Robot. Autom.
**5**(1), 16–29 (1989)CrossRefGoogle Scholar - 52.J. Chou, M. Kamel, Finding the position and orientation of a sensor on a robot manipulator using quaternions. Int. J. Robot. Res.
**10**(3), 240–254 (1991)CrossRefGoogle Scholar - 53.Y. Shiu, S. Ahmad, Finding the mounting position of a sensor by solving a homogeneous transform equation of the form AX = XB, in
*IEEE International Conference on Robotics and Automation*, vol. 4 (IEEE, 1987), pp. 1666–1671Google Scholar - 54.Y. Dai, J. Trumpf, H. Li, N. Barnes, R. Hartley, Rotation averaging with application to camera-rig calibration, in
*Asian Conference on Computer Vision*(Springer, Berlin/Heidelberg, 2009), pp. 335–346Google Scholar - 55.M. Shah, R. Eastman, T. Hong, An overview of robot-sensor calibration methods for evaluation of perception systems, in
*Proceedings of the Workshop on Performance Metrics for Intelligent Systems*(ACM, 2012), pp. 15–20Google Scholar - 56.I. Fassi, G. Legnani, Hand to sensor calibration: a geometrical interpretation of the matrix equation AX = XB. J. Robot. Syst.
**22**(9), 497–506 (2005)CrossRefGoogle Scholar - 57.Z. Zhao, Y. Liu, Hand-eye calibration based on screw motions, in
*International Conference on Pattern Recognition*, vol. 3 (IEEE, 2006), pp. 1022–1026Google Scholar - 58.S. Gwak, J. Kim, F.C. Park, Numerical optimization on the Euclidean group with applications to camera calibration. IEEE Trans. Robot. Autom.
**19**(1), 65–74 (2003)CrossRefGoogle Scholar - 59.J.C. Chou, M. Kamel, Quaternions approach to solve the kinematic equation of rotation, AX = XB, of a sensor-mounted robotic manipulator, in
*IEEE International Conference on Robotics and Automation*(IEEE, 1988), pp. 656–662Google Scholar - 60.R. Horaud, F. Dornaika, Hand-eye calibration. Int. J. Robot. Res.
**14**(3), 195–210 (1995)CrossRefGoogle Scholar - 61.K. Daniilidis, E. Bayro-Corrochano, The dual quaternion approach to hand-eye calibration, in
*International Conference on Pattern Recognition*, vol. 1 (IEEE, 1996), pp. 318–322Google Scholar - 62.K. Daniilidis, Hand-eye calibration using dual quaternions. Int. J. Robot. Res.
**18**(3), 286–298 (1999)CrossRefGoogle Scholar - 63.N. Andreff, R. Horaud, B. Espiau, On-line hand-eye calibration, in
*International Conference on 3D Digital Imaging and Modeling*(IEEE, 1999), pp. 430–436Google Scholar - 64.N. Andreff, R. Horaud, B. Espiau, Robot hand-eye calibration using structure-from-motion. Int. J. Robot. Res.
**20**(3), 228–248 (2001)CrossRefGoogle Scholar - 65.Z. Zhao, Hand-eye calibration using convex optimization, in
*IEEE International Conference on Robotics and Automation*(IEEE, 2011), pp. 2947–2952Google Scholar - 66.Y. Seo, Y.-J. Choi, S.W. Lee, A branch-and-bound algorithm for globally optimal calibration of a camera-and-rotation-sensor system, in
*International Conference on Computer Vision*(IEEE, 2009), pp. 1173–1178Google Scholar - 67.J. Schmidt, F. Vogt, H. Niemann, Calibration–free hand–eye calibration: a structure–from–motion approach, in
*Joint Pattern Recognition Symposium*(Springer, Berlin/Heidelberg, 2005), pp. 67–74Google Scholar - 68.J. Angeles, G. Soucy, F.P. Ferrie, The online solution of the hand-eye problem. IEEE Trans. Robot. Autom.
**16**(6), 720–731 (2000)CrossRefGoogle Scholar - 69.M.K. Ackerman, A. Cheng, E. Boctor, G. Chirikjian, Online ultrasound sensor calibration using gradient descent on the Euclidean group, in
*IEEE International Conference on Robotics and Automation*(IEEE, 2014), pp. 4900–4905Google Scholar - 70.M.K. Ackerman, G.S. Chirikjian, A probabilistic solution to the AX = XB problem: sensor calibration without correspondence, in
*Geometric Science of Information*(Springer, Berlin, 2013), pp. 693–701zbMATHGoogle Scholar - 71.M.K. Ackerman, A. Cheng, G. Chirikjian, An information-theoretic approach to the correspondence-free AX = XB sensor calibration problem, in
*IEEE International Conference on Robotics and Automation*(IEEE, 2014), pp. 4893–4899Google Scholar - 72.B. Horn,
*Robot Vision*(MIT Press, Cambridge/Mass, 1986)Google Scholar - 73.V. Lepetit, F. Moreno-Noguer, P. Fua, EPnP: an accurate o(n) solution to the PnP problem. Int. J. Comput. Vis.
**81**(2), 155–166 (2009)CrossRefGoogle Scholar - 74.G.S. Chirikjian, A.B. Kyatkin,
*Engineering Applications of Noncommutative Harmonic Analysis: with Emphasis on Rotation and Motion Groups*(CRC Press, Boca Raton, 2001)zbMATHGoogle Scholar - 75.Y. Wang, G.S. Chirikjian, Nonparametric second-order theory of error propagation on motion groups. Int. J. Robot. Res.
**27**(11–12), 1258–1273 (2008)CrossRefGoogle Scholar - 76.M. Ackerman, A. Cheng, B. Shiffman, E. Boctor, G.S. Chirikjian, Sensor calibration with unknown correspondence: solving
*AX*=*XB*using Euclidean-group invariants, in*IEEE/RSJ International Conference on Intelligent Robots and Systems*(IEEE, 2013), pp. 1308–1313Google Scholar - 77.J.S.F. Vogt, H. Niemann, Vector quantization based data selection for hand-eye calibration, in
*Proceedings of the Vision, Modeling, and Visualization, Standford*, 16–18 Nov 2004 (IOS Press, 2004), p. 21Google Scholar - 78.J. Schmidt, H. Niemann, Data selection for hand-eye calibration: a vector quantization approach. Int. J. Robot. Res.
**27**(9), 1027–1053 (2008)CrossRefGoogle Scholar - 79.F. Shi, J. Wang, Y. Liu, An approach to improve online hand-eye calibration, in
*Pattern Recognition and Image Analysis*(Springer, Berlin/Heidelberg, 2005), pp. 647–655Google Scholar - 80.J. Zhang, F. Shi, Y. Liu, An adaptive selection of motion for online hand-eye calibration, in
*Advances in Artificial Intelligence*(Springer, Sydney, 2005), pp. 520–529zbMATHGoogle Scholar - 81.K.H. Strobl, G. Hirzinger, Optimal hand-eye calibration, in
*IEEE/RSJ International Conference on Intelligent Robots and Systems*(IEEE, 2006), pp. 4647–4653Google Scholar - 82.G.S. Chirikjian,
*Stochastic Models, Information Theory, and Lie Groups*, vol. 2 (Birkauser, Boston, 2011)Google Scholar