Secure and efficient big data deduplication in fog computing

  • Jiajun Yan
  • Xiaoming WangEmail author
  • Qingqing Gan
  • Suyu Li
  • Daxin Huang


With the rapid development of the Internet of Things, the massive amount of big data generated by the Internet of Things terminals and the real-time processing requirements have brought enormous challenges. A two-tier computing model consisting solely of two entities, cloud and user, will not be sufficient to support processing large numbers of concurrent data requests. Therefore, fog computing was proposed. How to realize the secure and efficient deduplication of ciphertext in fog computing has become a new research topic. In this paper, we firstly present a new decentralized deduplication structure and then show how to apply it to construct a secure and efficient big data deduplication scheme in fog computing. The cloud server, in the proposed paper, can quickly determine which fog server needs to be traversed to search duplicate data, and instead of traversing all fog servers. This significantly improves the efficiency of big data deduplication in fog computing. Furthermore, the proposed scheme allows fog server to verify whether the user possesses the ownership of the data. Performance analysis and experimental results show the proposed scheme has less overheads than existing schemes.


Fog computing Secure deduplication Proof of ownership Efficiency 



This work was supported in part by the National Natural Science Foundation of China under Grant 61070164 and Grant 61272415, in part by the Natural Science Foundation of Guangdong Provience, China, under Grant S2012010008767, in part by the Science and Technology Planning Project of Guangdong Provience, China, under Grant 2013B010401015, and in part by the Zhuhai Top Discipline-Information Security.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.


  1. Boneh D, Gentry C, Lynn B et al (2003) Aggregate and verifiably encrypted signatures from bilinear maps. In: International conference on the theory and applications of cryptographic techniques. Springer, Berlin, Heidelberg, pp 416–432Google Scholar
  2. Cui H, Deng RH, Li Y et al (2017) Attribute-based storage supporting secure deduplication of encrypted data in cloud. IEEE Trans Big Data.
  3. Di Pietro R, Sorniotti A (2016) Proof of ownership for deduplication systems: a secure, scalable, and efficient solution. Comput Commun 82:71–82CrossRefGoogle Scholar
  4. Douceur JR, Adya A, Bolosky WJ et al (2002) Reclaiming space from duplicate files in a serverless distributed file system. In: Proceedings 22nd international conference on distributed computing systems. IEEE, pp 617–624Google Scholar
  5. Fu Y, Xiao N, Jiang H et al (2017) Application-aware big data deduplication in cloud environment. IEEE Trans Cloud Comput.
  6. Gou Z, Yamaguchi S, Gupta BB (2017) Analysis of various security issues and challenges in cloud computing environment: a survey. In: Identity theft: breakthroughs in research and practice. IGI Global, pp 221–247Google Scholar
  7. Gupta BB, Yamaguchi S, Agrawal DP (2018) Advances in security and privacy of multimedia big data in mobile and cloud computing. Multim Tools Appl 77(7):9203–9208CrossRefGoogle Scholar
  8. Halevi S, Harnik D, Pinkas B et al (2011) Proofs of ownership in remote storage systems. In: Proceedings of the 18th ACM conference on computer and communications security. ACM, pp 491–500Google Scholar
  9. Jiang T, Chen X, Wu Q et al (2017) Secure and efficient cloud data deduplication with randomized tag. IEEE Trans Inf Forensics Secur 12(3):532–543CrossRefGoogle Scholar
  10. Koo D, Hur J (2018) Privacy-preserving deduplication of encrypted data with dynamic ownership management in fog computing. Future Gener Comput Syst 78:739–752CrossRefGoogle Scholar
  11. Koo D, Shin Y, Yun J et al (2016) A hybrid deduplication for secure and efficient data outsourcing in fog Computing. In: 2016 IEEE international conference on cloud computing technology and science (CloudCom). IEEE, pp 285–293Google Scholar
  12. Kwon H, Hahn C, Kim D et al (2017) Secure deduplication for multimedia data with user revocation in cloud storage. Multim Tools Appl 76(4):5889–5903CrossRefGoogle Scholar
  13. Mishra S, Singh S, Ali ST (2018) MPoWS: merged proof of ownership and storage for block level deduplication in cloud storage. In: 2018 9th International conference on computing, communication and networking technologies (ICCCNT). IEEE, pp 1–7Google Scholar
  14. Ni J, Zhang K, Yu Y et al (2018) Providing task allocation and secure deduplication for mobile crowdsensing via fog computing[J]. IEEE Trans Dependable Secure Comput.
  15. Pooranian Z, Chen KC, Yu CM et al (2018) RARE: defeating side channels based on data-deduplication in cloud storage. In: IEEE INFOCOM 2018-IEEE conference on computer communications workshops (INFOCOM WKSHPS). IEEE, pp 444–449Google Scholar
  16. Shin Y, Koo D, Hur J et al (2017a) Secure proof of storage with deduplication for cloud storage systems. Multim Tools Appl 76(19):19363–19378CrossRefGoogle Scholar
  17. Shin Y, Koo D, Yun J et al (2017b) Decentralized server-aided encryption for secure deduplication in cloud storage. IEEE Trans Serv Comput.
  18. Stanek J, Kencl L (2018) Enhanced secure thresholded data deduplication scheme for cloud storage. IEEE Trans Dependable Secure Comput 15(4):694–707CrossRefGoogle Scholar
  19. Stergiou C, Psannis KE, Kim BG et al (2018a) Secure integration of IoT and cloud computing. Future Gener Comput Syst 78:964–975CrossRefGoogle Scholar
  20. Stergiou C, Psannis KE, Xifilidis T et al (2018) Security and privacy of big data for social networking services in cloud. In:IEEE INFOCOM 2018-IEEE conference on computer communications workshops (INFOCOM WKSHPS). IEEE, pp 438–443Google Scholar
  21. Xiong J, Zhang Y, Lin L et al (2017) ms-PoSW: a multi-server aided proof of shared ownership scheme for secure deduplication in cloud. Concurr Comput: Pract Exp.
  22. Yang X, Lu R, Choo KKR et al (2017) Achieving efficient and privacy-preserving cross-domain big data deduplication in cloud. IEEE Trans Big Data.
  23. Yaseen Q, Aldwairi M, Jararweh Y et al (2018) Collusion attacks mitigation in internet of things: a fog based model. Multim Tools Appl 77(14):18249–18268CrossRefGoogle Scholar
  24. Yu CM, Gochhayat SP, Conti M et al (2018) Privacy aware data deduplication for side channel in cloud storage. IEEE Trans Cloud Comput.
  25. Zhang Y, Xu C, Li H et al (2018) Healthdep: an efficient and secure deduplication scheme for cloud-assisted ehealth systems. IEEE Trans Ind Inf 14(9):4101–4112CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  • Jiajun Yan
    • 1
  • Xiaoming Wang
    • 1
    Email author
  • Qingqing Gan
    • 1
  • Suyu Li
    • 1
  • Daxin Huang
    • 1
  1. 1.The Department of Computer ScienceJinan UniversityGuangzhouChina

Personalised recommendations