Abstract
Deep learning recently becomes popular because it brings significant improvements on a wide variety of classification and recognition tasks. However, with the population and increasing usage of deep learning based models, not many people take into account the potential security risks which are likely to cause accidents in them. This paper mainly studies on the potential safety hazards in the obstacle recognition and processing system (ORPS) of the self-driving cars, which is constructed by deep learning architecture. We perform an attack that embeds a backdoor in the Mask R-CNN in ORPS by poisoning the dataset. The experiment result shows that it is possible to embed a backdoor in ORPS. We can see that the backdoored network can accurately recognize and trigger the backdoors in the poisoned dataset, which obviously change the size of bounding box and corresponding mask of those poisoned instances. But on the other hand, embedding a backdoor in the deep learning based model will only slightly affect the accuracy of detecting objects without backdoor triggers, which is imperceptible for users. Furthermore, in order to study the working mode of the backdoor and the possibility of detecting the backdoor in the network, we visualize the weights matrices in the backdoored network and try to modify them, but the results show that the existence of the backdoor in network is very cryptic, so it is difficult for users to detect and filter it. Eventually, we hope that our simple work can arouse people’s attention to the self-driving technology and even other deep learning based models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, X., Liu, C., Li, B., Lu, K., Song, D.: Targeted backdoor attacks on deep learning systems using data poisoning (2017)
Chollet, F.: Deep Learning with Python, 1st edn. Manning Publications Co., Greenwich (2017)
Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding (2016)
Deng, J., et al.: Imagenet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, pp. 248–255. IEEE (2009)
Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)
Evtimov, I., et al.: Robust physical-world attacks on machine learning models (2017)
Gardner, M.W., Dorling, S.: Artificial neural networks (the multilayer perceptron)-a review of applications in the atmospheric sciences. Atmos. Environ. 32(14–15), 2627–2636 (1998)
Girshick, R.: Fast r-cnn. arXiv preprint (2015). arXiv:1504.08083
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249–256 (2010)
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and Harnessing Adversarial Examples. ArXiv e-prints, December 2014
Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y.: Maxout networks. arXiv preprint (2013). arXiv:1302.4389
Gu, T., Dolan-Gavitt, B., Garg, S.: Badnets: identifying vulnerabilities in the machine learning model supply chain. CoRR abs/1708.06733 (2017). http://arxiv.org/abs/1708.06733
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. ArXiv e-prints, March 2017
He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8691, pp. 346–361. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10578-9_23
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)
Jiaxi, W., XiaoTong, L., Zhiqiang, L., Yi, T.: A security concern about deep learning models (2018)
Koh, P.W., Liang, P.: Understanding black-box predictions via influence functions (2017)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
Lecun, Y., et al.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), 541–551 (1989)
Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Liu, Y., et al.: Trojaning attack on neural networks. In: Network and Distributed System Security Symposium (2017)
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)
Moosavidezfooli, S.M., Fawzi, A., Fawzi, O., Frossard, P.: Universal adversarial perturbations, pp. 86–94 (2016)
Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML-2010), pp. 807–814 (2010)
Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)
Papernot, N., Mcdaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning, pp. 506–519 (2016)
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)
Saxena, P., Saxena, P., Saxena, P.: A uror: defending against poisoning attacks in collaborative deep learning systems. In: Conference on Computer Security Applications, pp. 508–519 (2016)
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., LeCun, Y.: Overfeat: integrated recognition, localization and detection using convolutional networks. arXiv preprint (2013). arXiv:1312.6229
Szegedy, C., Toshev, A., Erhan, D.: Deep neural networks for object detection. In: Advances in Neural Information Processing Systems, vol. 26, pp. 2553–2561 (2013)
Szegedy, C., et al.: Intriguing properties of neural networks (2013)
Timofte, R., Zimmermann, K., Gool, L.V.: Multi-view traffic sign detection, recognition, and 3d localisation. Mach. Vis. Appl. 25(3), 633–647 (2014)
Yang, C., Wu, Q., Li, H., Chen, Y.: Generative poisoning attack method against neural networks (2017)
Yang, F., Choi, W., Lin, Y.: Exploit all the layers: fast and accurate CNN object detector with scale dependent pooling and cascaded rejection classifiers. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2129–2137 (2016)
Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_53
Acknowledgement
This paper is partially supported by the National Natural Science Foundation of China grants 61772147, and the Key Basic Research of Guangdong Province Natural Science Fund Fostering Projects grants 2015A030308016.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Lin, X., Wu, J., Tang, Y. (2018). Generating Misleading Labels in Machine Learning Models. In: Vaidya, J., Li, J. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2018. Lecture Notes in Computer Science(), vol 11335. Springer, Cham. https://doi.org/10.1007/978-3-030-05054-2_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-05054-2_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-05053-5
Online ISBN: 978-3-030-05054-2
eBook Packages: Computer ScienceComputer Science (R0)