An FPGA accelerator for PatchMatch multi-view stereo using OpenCL

  • Shunsuke Tatsumi
  • Masanori Hariyama
  • Koichi Ito
  • Takafumi Aoki
Original Research Paper


PatchMatch multi-view stereo (MVS) is one method generating depth maps from multi-view images and is expected to be used for various applications such as robot vision, 3D measurement, and 3D reconstruction. The major drawback of PatchMatch MVS is its large computational amount, and its acceleration is strongly desired. However, this acceleration is prevented by two problems. First, though PatchMatch MVS estimates depth maps by propagating estimation results among neighbor pixels, it is not suitable for GPU-based acceleration. Second, since the shape of a matching window used for stereo matching is changed dynamically, reading its pixels is inefficient in memory access. This paper proposes an FPGA accelerator exploiting on-chip FIFOs efficiently to solve the propagation problem. Moreover, reading pixels of a matching window is improved by a cover window which has the fixed shape and covers the matching window. The FPGA accelerator is designed using a design tool based on Open Computing Language (OpenCL). Although parameters of PatchMatch MVS depend on object images, these parameters can be changed easily by the OpenCL-based design. The experimental results demonstrate that the FPGA implementation achieves 3.4 and 2.2 times faster processing speeds than the CPU and GPU ones, respectively, and the power-delay product of the FPGA implementation is 3.2 and 5.7% of the CPU and GPU ones, respectively.


PatchMatch Multi-view stereo (MVS) 3D reconstruction OpenCL for FPGA Reconfigurable computing 


  1. 1.
    Bleyer, M., Rhemann, C., Rother, C.: PatchMatch stereo–stereo matching with slanted support windows. In: Proceedings of the British Machine Vision Conference (BMVC), pp. 14.1–14.11 (2011)Google Scholar
  2. 2.
    Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.: PatchMatch: a randomized correspondence algorithm for structural image editing. In: ACM Transactions on Graphics (Proc. SIGGRAPH), vol. 28(3) (2009)Google Scholar
  3. 3.
    Pradeep, V., Rhemann, C., Izadi, S., Zach, C., Bleyer, M., Bathiche, S.: MonoFusion: real-time 3D reconstruction of small scenes with a single web camera. In: IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 1–4 (2013)Google Scholar
  4. 4.
    Zollhöfer, M., Nießner, M., Izadi, S., Rhemann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., Stamminger, M.: Real-time non-rigid reconstruction using an RGB-D camera. ACM Trans. Graph. (TOG) 33(4), 156 (2014)CrossRefGoogle Scholar
  5. 5.
    Hiradate, M., Ito, K., Aoki, T., Watanabe, T., Unten, H.: An extension of PatchMatch stereo for 3D reconstruction from multi-view images. In: IAPR Asian Conference on Pattern Recognition (ACPR), pp. 61–65 (2015)Google Scholar
  6. 6.
    Khronos Group, The OpenCL Specification. Accessed 5 Mar 2017
  7. 7.
    Czajkowski, T.S., Neto, D., Kinsner, M., Aydonat, U., Wong, J., Denisenko, D., Yiannacouras, P., Freeman, J., Singh, D.P., Brown, S.D.: OpenCL for FPGAs: prototyping a compiler. In: Proceedings of the International Conference on Engineering of Reconfigurable Systems and Algorithms (ERSA), pp. 3–12 (2012)Google Scholar
  8. 8.
    Zheng, E., Dunn, E., Jojic, V., Frahm, J.: PatchMatch based joint view selection and depthmap estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1510–1517 (2014)Google Scholar
  9. 9.
    Bailer, C., Finckh, M., Lensch, H.P.: Scale robust multi view stereo. In: European Conference on Computer Vision (ECCV), pp. 398–411 (2012)Google Scholar
  10. 10.
    Galliani, S., Lasinger, K., Schindler, K.: Massively parallel multiview stereopsis by surface normal diffusion. In: IEEE International Conference on Computer Vision (ICCV), pp. 873–881 (2015)Google Scholar
  11. 11.
    Heise, P., Klose, S., Jensen, B., Knoll, A.: PM-Huber: PatchMatch with huber regularization for stereo matching. In: IEEE International Conference on Computer Vision (ICCV), pp. 2360–2367 (2013)Google Scholar
  12. 12.
    Besse, F., Rother, C., Fitzgibbon, A., Kautz, J.,: PMBP: PatchMatch belief propagation for correspondence field estimation. In: Proceedings of the British Machine Vision Conference (BMVC), pp. 132.1–132.11 (2012)Google Scholar
  13. 13.
    Bao, L., Yang, Q., Jin, H.: Fast edge-preserving patchmatch for large displacement optical flow. IEEE Trans. Image Process. 23(12), 4996–5006 (2014)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Rong, G., Tan, T.: Jump flooding in GPU with applications to voronoi diagram and distance transform. In: Proceedings of ACM Symposium on Interactive 3D Graphics and Games (i3D 2006), pp. 109–116 (2006)Google Scholar
  15. 15.
    Bao, Y., Tang, L., Schnable, P.S., Fernandez, M.G.S.: GPU-based parallelization of a sub-pixel highresolution stereo matching algorithm for highthroughput biomass sorghum phenotyping. In: Agricultural and Biosystems Engineering Conference Proceedings and Presentations, 452 (2015)Google Scholar
  16. 16.
    Goesele, M., Curless, B., Seitz, S.M.: Multi-view stereo revisited. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2402–2409 (2006)Google Scholar
  17. 17.
    Implementing FPGA Design with the OpenCL Standard. Accessed 5 Mar 2017
  18. 18.
    Intel FPGA SDK for OpenCL Best Practices Guide. Accessed 5 Mar 2017
  19. 19.
    Matsumoto, M., Nishimura, T.: Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator. ACM Trans. Model. Comput. Simul. (TOMACS) 8(1), 3–30 (1998)CrossRefMATHGoogle Scholar
  20. 20.
    Monte Carlo Black-Scholes Asian Option Pricing Design Example. Accessed 5 Mar 2017
  21. 21.
    Nallatech 395 with stratix V D8. Accessed 5 Mar 2017
  22. 22.
    Strecha, C.: Multi-view evaluation. Accessed 5 Mar 2017
  23. 23.
    Strecha, C., von Hansen, W., Gool, L.V., Fua, P., Thoennessen, U.: On benchmarking camera calibration and multi-view stereo for high resolution imagery. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–8 (2008)Google Scholar
  24. 24.
    Sakai, S., Ito, K., Aoki, T., Watanabe, T., Unten, H.: Phase-based window matching with geometric correction for multi-view stereo. IEICE Trans. Inf. Syst. E98-D(10), 1818–1828 (2015)CrossRefGoogle Scholar
  25. 25.
  26. 26.
    Fang, J., Varbanescu, A.L., Sips, H.: A comprehensive performance comparison of CUDA and OpenCL. In: International Conference on Parallel Processing (ICPP), pp. 216–225 (2011).
  27. 27.
    Karimi, K., Dickson, N.G., Hamze, F.: A performance comparison of CUDA and OpenCL. In: Proceedings of International Journal of High Performance Computing Applications (2011).
  28. 28.
    Hill, K., Craciun, S., George, A., Lam, H.: Comparative analysis of OpenCL vs. HDL with image-processing kernels on Stratix-V FPGA. In: IEEE 26th International Conference on Application-specific Systems Architectures and Processors (ASAP), pp. 189–193 (2015).

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  • Shunsuke Tatsumi
    • 1
  • Masanori Hariyama
    • 2
  • Koichi Ito
    • 1
  • Takafumi Aoki
    • 1
  1. 1.Graduate School of Information SciencesTohoku UniversitySendaiJapan
  2. 2.Graduate School of Information SciencesTohoku UniversitySendaiJapan

Personalised recommendations