Abstract
In recent years, deep learning has rapidly become one of the most successful approaches to machine learning. The essential idea of deep learning is to apply a multiple-layer structure to extract complex features from high-dimensional data and use those features to build models. However, deep learning models are susceptible to several types of attacks. For example, a centralized collection of photos, speech, and video clips from millions of individuals might meet with privacy risks when they are shared with others. Learning models can also disclose sensitive information. To integrate differential privacy to deep learning, we need to consider two challenges: high sensitivity and limited privacy budget. This chapter first presents the traditional Laplace method and illustrates the limitations of the method, and then present Private SGD Method, Deep Private Auto-Encoder Algorithm and Distributed Private SGD. Each of them is focusing on a particular deep learning algorithm and is dealing with those two challenges in different ways. Finally, this chapter shows several popular datasets that can be used in differentially private deep learning.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
References
M. Abadi, A. Chu, I. J. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24–28, 2016, pages 308–318, 2016.
Y. Bengio. Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2(1):1–127, 2009.
CIFAR-10 and C.-. datasets. www.cs.toronto.edu/kriz/cifar.html.
R. Collobert, K. Kavukcuoglu, and C. Farabet. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop, 2011.
J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. W. Senior, P. A. Tucker, K. Yang, and A. Y. Ng. Large scale distributed deep networks. In NIPS, pages 1232–1240, 2012.
C. Dwork and A. Roth. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9(3–4):211–407, Aug. 2014.
M. Fredrikson, S. Jha, and T. Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22Nd ACM SIGSAC Conference on Computer and Communications Security, CCS ’15, pages 1322–1333, New York, NY, USA, 2015. ACM.
Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436–444, 2015.
Y. Lécun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. Nips Workshop on Deep Learning & Unsupervised Feature Learning, 2012.
N. Phan, Y. Wang, X. Wu, and D. Dou. Differential privacy preservation for deep auto-encoders: an application of human behavior prediction. In AAAI, pages 1309–1316, 2016.
B. Recht, C. Ré, S. J. Wright, and F. Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In NIPS, pages 693–701, 2011.
R. Shokri and V. Shmatikov. Privacy-preserving deep learning. In SIGSAC, pages 1310–1321, 2015.
Torch7. A scientific computing framework for luajit (torch.ch).
J. Zhang, Z. Zhang, X. Xiao, Y. Yang, and M. Winslett. Functional mechanism: Regression analysis under differential privacy. Proc. VLDB Endow., 5(11):1364–1375, July 2012.
Z. Zhou and J. Feng. Deep forest: Towards an alternative to deep neural networks. CoRR, abs/1702.08835, 2017.
M. Zinkevich, M. Weimer, A. J. Smola, and L. Li. Parallelized stochastic gradient descent. In NIPS, pages 2595–2603. Curran Associates, Inc., 2010.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this chapter
Cite this chapter
Zhu, T., Li, G., Zhou, W., Yu, P.S. (2017). Differentially Private Deep Learning. In: Differential Privacy and Applications. Advances in Information Security, vol 69. Springer, Cham. https://doi.org/10.1007/978-3-319-62004-6_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-62004-6_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-62002-2
Online ISBN: 978-3-319-62004-6
eBook Packages: Computer ScienceComputer Science (R0)