Multi-modal and Multi-spectral Registration for Natural Images

  • Xiaoyong Shen
  • Li Xu
  • Qi Zhang
  • Jiaya Jia
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8692)


Images now come in different forms – color, near-infrared, depth, etc. – due to the development of special and powerful cameras in computer vision and computational photography. Their cross-modal correspondence establishment is however left behind. We address this challenging dense matching problem considering structure variation possibly existing in these image sets and introduce new model and solution. Our main contribution includes designing the descriptor named robust selective normalized cross correlation (RSNCC) to establish dense pixel correspondence in input images and proposing its mathematical parameterization to make optimization tractable. A computationally robust framework including global and local matching phases is also established. We build a multi-modal dataset including natural images with labeled sparse correspondence. Our method will benefit image and vision applications that require accurate image alignment.


multi-modal multi-spectral dense matching variational model 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Agrawal, A.K., Raskar, R., Nayar, S.K., Li, Y.: Removing photography artifacts using gradient projection and flash-exposure sampling. ToG 24(3), 828–835 (2005)CrossRefGoogle Scholar
  2. 2.
    Andronache, A., von Siebenthal, M., Székely, G., Cattin, P.C.: Non-rigid registration of multi-modal images using both mutual information and cross-correlation. Medical Image Analysis 12(1), 3–15 (2008)CrossRefGoogle Scholar
  3. 3.
    Black, M.J., Anandan, P.: The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. CVIU 63(1), 75–104 (1996)Google Scholar
  4. 4.
    Brown, M., Susstrunk, S.: Multi-spectral sift for scene category recognition. In: CVPR, pp. 177–184 (2011)Google Scholar
  5. 5.
    Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J.(G.) (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  6. 6.
    Bruhn, A., Weickert, J.: Towards ultimate motion estimation: Combining highest accuracy with real-time performance. In: ICCV, pp. 749–755 (2005)Google Scholar
  7. 7.
    Chui, H., Rangarajan, A.: A new point matching algorithm for non-rigid registration. Computer Vision and Image Understanding 89(2), 114–141 (2003)CrossRefzbMATHGoogle Scholar
  8. 8.
    Fattal, R., Lischinski, D., Werman, M.: Gradient domain high dynamic range compression. ToG 21(3), 249–256 (2002)CrossRefGoogle Scholar
  9. 9.
    Firmenichy, D., Brown, M., Süsstrunk, S.: Multispectral interest points for rgb-nir image registration. In: ICIP, pp. 181–184 (2011)Google Scholar
  10. 10.
    Han, J., Pauwels, E.J., de Zeeuw, P.M.: Visible and infrared image registration in man-made environments employing hybrid visual features. Pattern Recognition Letters 34(1), 42–51 (2013)CrossRefGoogle Scholar
  11. 11.
    Heo, Y.S., Lee, K.M., Lee, S.U.: Robust stereo matching using adaptive normalized cross-correlation. PAMI 33(4), 807–822 (2011)CrossRefMathSciNetGoogle Scholar
  12. 12.
    Hermosillo, G., Chefd’Hotel, C., Faugeras, O.D.: Variational methods for multimodal image matching. IJCV 50(3), 329–343 (2002)CrossRefzbMATHGoogle Scholar
  13. 13.
    Horn, B.K.P., Schunck, B.G.: Determining optical flow. Artif. Intell. 17(1-3), 185–203 (1981)CrossRefGoogle Scholar
  14. 14.
    Hrkać, T., Kalafatić, Z., Krapac, J.: Infrared-visual image registration based on corners and hausdorff distance. In: Ersbøll, B.K., Pedersen, K.S. (eds.) SCIA 2007. LNCS, vol. 4522, pp. 383–392. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  15. 15.
    Irani, M., Anandan, P.: Robust multi-sensor image alignment. In: ICCV, pp. 959–966 (1998)Google Scholar
  16. 16.
    Jian, B., Vemuri, B.C.: Robust point set registration using gaussian mixture models. PAMI 33(8), 1633–1645 (2011)CrossRefGoogle Scholar
  17. 17.
    Kolár, R., Kubecka, L., Jan, J.: Registration and fusion of the autofluorescent and infrared retinal images. International Journal of Biomedical Imaging (2008)Google Scholar
  18. 18.
    Krishnan, D., Fergus, R.: Dark flash photography. ToG 28(3) (2009)Google Scholar
  19. 19.
    Liu, C., Yuen, J., Torralba, A., Sivic, J., Freeman, W.T.: Sift flow: Dense correspondence across different scenes. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part III. LNCS, vol. 5304, pp. 28–42. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  20. 20.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. IJCV 60(2), 91–110 (2004)CrossRefGoogle Scholar
  21. 21.
    Palos, G., Betrouni, N., Coulanges, M., Vermandel, M., Devlaminck, V., Rousseau, J.: Multimodal matching by maximisation of mutual information and optical flow technique. In: IEEE International Conference on Engineering in Medicine and Biology Society, pp. 1679–1682 (2004)Google Scholar
  22. 22.
    Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M.F., Hoppe, H., Toyama, K.: Digital photography with flash and no-flash image pairs. ToG 23(3), 664–672 (2004)CrossRefGoogle Scholar
  23. 23.
    Pluim, J.P.W., Maintz, J.B.A., Viergever, M.A.: Mutual information based registration of medical images: A survey. IEEE Transaction on Medical Imaging 22(8), 986–1004 (2003)CrossRefGoogle Scholar
  24. 24.
    Sen, P., Kalantari, N.K., Yaesoubi, M., Darabi, S., Goldman, D.B., Shechtman, E.: Robust patch-based hdr reconstruction of dynamic scenes. ToG 31(6), 203 (2012)CrossRefGoogle Scholar
  25. 25.
    Sun, D., Roth, S., Black, M.J.: Secrets of optical flow estimation and their principles. In: CVPR, pp. 2432–2439 (2010)Google Scholar
  26. 26.
    Sun, J., Kang, S.B., Xu, Z., Tang, X., Shum, H.Y.: Flash cut: Foreground extraction with flash and no-flash image pairs. In: CVPR (2007)Google Scholar
  27. 27.
    Sun, J., Zheng, N., Shum, H.Y.: Stereo matching using belief propagation. PAMI 25(7), 787–800 (2003)CrossRefGoogle Scholar
  28. 28.
    Szeliski, R.: Image alignment and stitching: A tutorial. Foundations and Trends in Computer Graphics Vision 2(1), 1–104 (2006)CrossRefGoogle Scholar
  29. 29.
    Tsin, Y., Kanade, T.: A correlation-based approach to robust point set registration. In: Pajdla, T., Matas, J.(G.) (eds.) ECCV 2004. LNCS, vol. 3023, pp. 558–569. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  30. 30.
    Wang, Y., Yang, J., Yin, W., Zhang, Y.: A new alternating minimization algorithm for total variation image reconstruction. SIAM Journal on Imaging Sciences 1(3), 248–272 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  31. 31.
    Wedel, A., Rabe, C., Vaudrey, T., Brox, T., Franke, U., Cremers, D.: Efficient dense scene flow from sparse or dense stereo data. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 739–751. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  32. 32.
    Werlberger, M., Pock, T., Bischof, H.: Motion estimation with non-local total variation regularization. In: CVPR, pp. 2464–2471 (2010)Google Scholar
  33. 33.
    Xiong, Z., Zhang, Y.: A critical review of image registration methods. International Journal of Image and Data Fusion 1(2), 137–158 (2010)CrossRefGoogle Scholar
  34. 34.
    Xu, L., Jia, J., Matsushita, Y.: Motion detail preserving optical flow estimation. PAMI 34(9), 1744–1757 (2012)CrossRefGoogle Scholar
  35. 35.
    Yan, Q., Shen, X., Xu, L., Zhuo, S., Zhang, X., Shen, L., Jia, J.: Cross-field joint image restoration via scale map. In: ICCV (2013)Google Scholar
  36. 36.
    Yang, J., Blum, R.S., Williams, J.P., Sun, Y., Xu, C.: Non-rigid image registration using geometric features and local salient region features. In: CVPR, pp. 825–832 (2006)Google Scholar
  37. 37.
    Yi, Z., Soatto, S.: Nonrigid registration combining global and local statistics. In: CVPR (2009)Google Scholar
  38. 38.
    Yuan, L., Sun, J., Quan, L., Shum, H.Y.: Image deblurring with blurred/noisy image pairs. ToG 26(3) (2007)Google Scholar
  39. 39.
    Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime TV-L1 optical flow. Pattern Recognition, 214–223 (2007)Google Scholar
  40. 40.
    Zhang, Z., Jiang, Y., Tsui, H.: Consistent multi-modal non-rigid registration based on a variational approach. Pattern Recognition Letters 27(7), 715–725 (2006)CrossRefGoogle Scholar
  41. 41.
    Zimmer, H., Bruhn, A., Weickert, J., Valgaerts, L., Salgado, A., Rosenhahn, B., Seidel, H.-P.: Complementary optic flow. In: Cremers, D., Boykov, Y., Blake, A., Schmidt, F.R. (eds.) EMMCVPR 2009. LNCS, vol. 5681, pp. 207–220. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  42. 42.
    Zitová, B., Flusser, J.: Image registration methods: a survey. Image and Vision Computing 21(11), 977–1000 (2003)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Xiaoyong Shen
    • 1
  • Li Xu
    • 2
  • Qi Zhang
    • 1
  • Jiaya Jia
    • 1
  1. 1.The Chinese University of Hong KongChina
  2. 2.Image & Visual Computing LabLenovo R&T, Project WebsiteHong KongChina

Personalised recommendations