A Parallel Fast Fourier Transform Algorithm for Large-Scale Signal Data Using Apache Spark in Cloud

  • Cheng Yang
  • Weidong Bao
  • Xiaomin ZhuEmail author
  • Ji Wang
  • Wenhua Xiao
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11336)


In the field of signal process, Fast Fourier Transform (FFT) is a widely used algorithm to transform signal data from time to frequency. Unfortunately, with the exponential growth of data, traditional methods cannot meet the demand of large-scale computation on these big data because of three main challenges of large-scale FFT, i.e., big data size, real-time data processing and high utilization of compute resources. To satisfy these requirements, an optimized FFT algorithm in Cloud is deadly needed. In this paper, we introduce a new method to conduct FFT in Cloud with the following contributions: first, we design a parallel FFT algorithm for large-scaled signal data in Cloud; second, we propose a MapReduce-based mechanism to distribute data to compute nodes using big data processing framework; third, an optimal method of distributing compute resources is implemented to accelerate the algorithm by avoiding redundant data exchange between compute nodes. The algorithm is designed in MapReduce computation framework which contains three steps: data preprocessing, local data transform and parallel data transform to integrate processing results. The parallel FFT is implemented in a 16-node Cloud to process real signal data The experimental results reveal an obvious improvement in the algorithm speed. Our parallel FFT is approximately five times faster than FFT in Matlab in when the data size reaches 10 GB.


Fast fourier transform Cloud computing Apache spark Parallel algorithm 



The authors would like to thank the anonymous referees for their helpful comments from which the preparation for this version of the paper has benefited. Thanks for Johann Sebastian Bach for his inspiring music accompanying the authors to complete the research. This work was supported in part by the National Natural Science Foundation of China under Grant 61572511 and Grant 91648204 and Grant 61872378, in part by the Scientific Research Project of National University of Defense Technology under Grant ZK16-03-57, in part by the China Postdoctoral Science Foundation under Grant 2016M602960 and Grant 2017T100796, in part by Science Fund for Distinguished Young Scholars in Hunan Province under Grant 2018JJ1032. Xiaomin Zhu is the corresponding author.


  1. 1.
    Armbrust, M., et al.: A view of cloud computing. Commun. ACM 53(4), 50–58 (2010)CrossRefGoogle Scholar
  2. 2.
    Armbrust, M., et al.: Above the clouds: A Berkeley View of Cloud Computing. Tech. rep., Technical ReportD UCB/EECS-2009-28, EECS Department, University of California, Berkeley (2009)Google Scholar
  3. 3.
    Baker, S.: Google and the wisdom of clouds. Business Week 14 (2007)Google Scholar
  4. 4.
    Bergland, G.D.: A parallel implementation of the fast fourier transform algorithm. IEEE Trans. Comput. 100(4), 366–370 (1972)CrossRefGoogle Scholar
  5. 5.
    Bosagh Zadeh, R., et al.: Matrix computations and optimization in apache spark. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 31–38. ACM (2016)Google Scholar
  6. 6.
    Cooley, J.W., Tukey, J.W.: An algorithm for the machine calculation of complex fourier series. Math. Comput. 19(90), 297–301 (1965)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)CrossRefGoogle Scholar
  8. 8.
    Frigo, M., Johnson, S.G.: FFTW: An adaptive software architecture for the FFT. In: Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, vol. 3, pp. 1381–1384. IEEE (1998)Google Scholar
  9. 9.
    Githens, J.: A fully parallel computer for radar data processing. In: IEEE Transactions on Aerospace and Electronic Systems, p. 736. No. 5 (1970)Google Scholar
  10. 10.
    Hassanieh, H., Adib, F., Katabi, D., Indyk, P.: Faster gps via the sparse fourier transform. In: International Conference on Mobile Computing and Networking, pp. 353–364 (2012)Google Scholar
  11. 11.
    Hassen, H., Khemakhem, M.: Arabic islamic manuscripts digitization based on hybrid K-NN/SVM approach and cloud computing technologies. In: Taibah University International Conference on Advances in Information Technology for the Holy Quran and Its Sciences (32519), pp. 366–371. IEEE (2013)Google Scholar
  12. 12.
    Kulkarni, P., Kumar, V., Verma, H.: Diagnostic acceptability of FFT-based ECG data compression. J. Med. Eng. Technol. 21(5), 185–189 (1997)CrossRefGoogle Scholar
  13. 13.
    Li, F., Xu, J., Zhouhong, J., Miao, W.: Aerial target detection via GPS satellite broadcast signal. J. Chin. Inert. Technol. 22(6), 788–793 (2014)Google Scholar
  14. 14.
    Marcum, J.: A statistical theory of target detection by pulsed radar. IRE Trans. Inf. Theory 6(2), 59–267 (1960)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Marple, L.: Computing the discrete-time “analytic” signal via FFT. IEEE Trans. Signal Process. 47(9), 2600–2603 (1999)CrossRefGoogle Scholar
  16. 16.
    Mullin, L.R., Small, S.G.: Four easy ways to a faster FFT. J. Math. Model. Algorithms 1(3), 193–214 (2002)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Palanisamy, B.: Purlieus: locality-aware resource allocation for MapReduce in a cloud. In: High Performance Computing, Networking, Storage and Analysis, pp. 1–11 (2011)Google Scholar
  18. 18.
    Prasad, N., Shameem, V., Desai, U., Merchant, S.: Improvement in target detection performance of pulse coded doppler radar based on multicarrier modulation with fast fourier transform (fft). IEE Proc. Radar, Sonar Navig. 151(1), 11–17 (2004)CrossRefGoogle Scholar
  19. 19.
    Preuss, R.: Very fast computation of the radix-2 discrete fourier transform. IEEE Trans. Acoustics, Speech, Signal Process. 30(4), 595–607 (1982)CrossRefGoogle Scholar
  20. 20.
    Reddy, B.S., Chatterji, B.N.: An FFT-based technique for translation, rotation, and scale-invariant image registration. IEEE Trans. Image Process. 5(8), 1266–1271 (1996)CrossRefGoogle Scholar
  21. 21.
    Spark, A.: Lightning-fast cluster computing (2016)Google Scholar
  22. 22.
    Tang, G., Peng, L., Baldwin, P.R., Mann, D.S., Jiang, W., Rees, I., Ludtke, S.J.: Eman2: an extensible image processing suite for electron microscopy. J. Struct. Biol. 157(1), 38–46 (2007)CrossRefGoogle Scholar
  23. 23.
    Ubeyli, E., Güler, I.: Spectral analysis of internal carotid arterial doppler signals using FFT, AR, MA, and ARMA methods. Comput. Biol. Med. 34(4), 293 (2004)CrossRefGoogle Scholar
  24. 24.
    Vincke, R., Landschoot, S.V., Cordemans, P., Peuteman, J., Steegmans, E., Boydens, J.: Algorithm parallelization using software design patterns, an embedded case study approach. In: Eighth International Conference on P2P, Parallel, Grid, Cloud and Internet Computing, pp. 470–473 (2013)Google Scholar
  25. 25.
    Wang, Y., Yang, R., Wo, T., Jiang, W., Hu, C.: Improving utilization through dynamic VM resource allocation in hybrid cloud environment. In: IEEE International Conference on Parallel and Distributed Systems, pp. 241–248 (2015)Google Scholar
  26. 26.
    Welch, P.: The use of fast fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Trans. Audio Electroacoust. 15(2), 70–73 (1967)CrossRefGoogle Scholar
  27. 27.
    Wold, E., Despain, A.: Pipeline and parallel-pipeline FFT processors for VLSI implementations. IEEE Trans. Comput. C–33(5), 414–426 (1984)CrossRefGoogle Scholar
  28. 28.
    Xu, L., Li, J., Stoica, P.: Target detection and parameter estimation for mimo radar systems. IEEE Trans. Aerosp. Electron. Syst. 44(3), 927–939 (2008)CrossRefGoogle Scholar
  29. 29.
    Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I.: Spark: cluster computing with working sets. HotCloud 10(10–10), 95 (2010)Google Scholar
  30. 30.
    Zaharia, M., et al.: Apache spark: a unified engine for big data processing. Commun. ACM 59(11), 56–65 (2016)CrossRefGoogle Scholar
  31. 31.
    Zhu, X., Mong Sim, K., Jiang, J., Wang, J., Chen, C.: Agent-based dynamic scheduling for earth-observing tasks on multiple airships in emergency. IEEE Syst. J. 10(2), 661–672 (2016)CrossRefGoogle Scholar
  32. 32.
    Zhu, X., Wang, J., Guo, H., Zhu, D., Yang, L.T., Liu, L.: Fault-tolerant scheduling for real-time scientific workflows with elastic resource provisioning in virtualized clouds. IEEE Trans. Parallel Distrib. Syst. 27(12), 3501–3517 (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Cheng Yang
    • 1
  • Weidong Bao
    • 1
  • Xiaomin Zhu
    • 1
    • 2
    Email author
  • Ji Wang
    • 1
  • Wenhua Xiao
    • 1
    • 3
  1. 1.National University of Defense TechnologyChangshaChina
  2. 2.State Key Laboratory of High Performance ComputingChangshaChina
  3. 3.Academy of Military SciencesBeijingChina

Personalised recommendations