Generating Misleading Labels in Machine Learning Models

  • Xiaotong Lin
  • Jiaxi Wu
  • Yi TangEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11335)


Deep learning recently becomes popular because it brings significant improvements on a wide variety of classification and recognition tasks. However, with the population and increasing usage of deep learning based models, not many people take into account the potential security risks which are likely to cause accidents in them. This paper mainly studies on the potential safety hazards in the obstacle recognition and processing system (ORPS) of the self-driving cars, which is constructed by deep learning architecture. We perform an attack that embeds a backdoor in the Mask R-CNN in ORPS by poisoning the dataset. The experiment result shows that it is possible to embed a backdoor in ORPS. We can see that the backdoored network can accurately recognize and trigger the backdoors in the poisoned dataset, which obviously change the size of bounding box and corresponding mask of those poisoned instances. But on the other hand, embedding a backdoor in the deep learning based model will only slightly affect the accuracy of detecting objects without backdoor triggers, which is imperceptible for users. Furthermore, in order to study the working mode of the backdoor and the possibility of detecting the backdoor in the network, we visualize the weights matrices in the backdoored network and try to modify them, but the results show that the existence of the backdoor in network is very cryptic, so it is difficult for users to detect and filter it. Eventually, we hope that our simple work can arouse people’s attention to the self-driving technology and even other deep learning based models.


Misleading labels Deep learning Backdoor trigger 



This paper is partially supported by the National Natural Science Foundation of China grants 61772147, and the Key Basic Research of Guangdong Province Natural Science Fund Fostering Projects grants 2015A030308016.


  1. 1.
    Chen, X., Liu, C., Li, B., Lu, K., Song, D.: Targeted backdoor attacks on deep learning systems using data poisoning (2017)Google Scholar
  2. 2.
    Chollet, F.: Deep Learning with Python, 1st edn. Manning Publications Co., Greenwich (2017)Google Scholar
  3. 3.
    Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding (2016)Google Scholar
  4. 4.
    Deng, J., et al.: Imagenet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, pp. 248–255. IEEE (2009)Google Scholar
  5. 5.
    Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)CrossRefGoogle Scholar
  6. 6.
    Evtimov, I., et al.: Robust physical-world attacks on machine learning models (2017)Google Scholar
  7. 7.
    Gardner, M.W., Dorling, S.: Artificial neural networks (the multilayer perceptron)-a review of applications in the atmospheric sciences. Atmos. Environ. 32(14–15), 2627–2636 (1998)CrossRefGoogle Scholar
  8. 8.
    Girshick, R.: Fast r-cnn. arXiv preprint (2015). arXiv:1504.08083
  9. 9.
    Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)Google Scholar
  10. 10.
    Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249–256 (2010)Google Scholar
  11. 11.
    Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and Harnessing Adversarial Examples. ArXiv e-prints, December 2014Google Scholar
  12. 12.
    Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y.: Maxout networks. arXiv preprint (2013). arXiv:1302.4389
  13. 13.
    Gu, T., Dolan-Gavitt, B., Garg, S.: Badnets: identifying vulnerabilities in the machine learning model supply chain. CoRR abs/1708.06733 (2017).
  14. 14.
    He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. ArXiv e-prints, March 2017Google Scholar
  15. 15.
    He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8691, pp. 346–361. Springer, Cham (2014). Scholar
  16. 16.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  17. 17.
    Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Jiaxi, W., XiaoTong, L., Zhiqiang, L., Yi, T.: A security concern about deep learning models (2018)Google Scholar
  19. 19.
    Koh, P.W., Liang, P.: Understanding black-box predictions via influence functions (2017)Google Scholar
  20. 20.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  21. 21.
    Lecun, Y., et al.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), 541–551 (1989)CrossRefGoogle Scholar
  22. 22.
    Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  23. 23.
    Liu, Y., et al.: Trojaning attack on neural networks. In: Network and Distributed System Security Symposium (2017)Google Scholar
  24. 24.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)Google Scholar
  25. 25.
    Moosavidezfooli, S.M., Fawzi, A., Fawzi, O., Frossard, P.: Universal adversarial perturbations, pp. 86–94 (2016)Google Scholar
  26. 26.
    Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML-2010), pp. 807–814 (2010)Google Scholar
  27. 27.
    Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)CrossRefGoogle Scholar
  28. 28.
    Papernot, N., Mcdaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning, pp. 506–519 (2016)Google Scholar
  29. 29.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)Google Scholar
  30. 30.
    Saxena, P., Saxena, P., Saxena, P.: A uror: defending against poisoning attacks in collaborative deep learning systems. In: Conference on Computer Security Applications, pp. 508–519 (2016)Google Scholar
  31. 31.
    Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., LeCun, Y.: Overfeat: integrated recognition, localization and detection using convolutional networks. arXiv preprint (2013). arXiv:1312.6229
  32. 32.
    Szegedy, C., Toshev, A., Erhan, D.: Deep neural networks for object detection. In: Advances in Neural Information Processing Systems, vol. 26, pp. 2553–2561 (2013)Google Scholar
  33. 33.
    Szegedy, C., et al.: Intriguing properties of neural networks (2013)Google Scholar
  34. 34.
    Timofte, R., Zimmermann, K., Gool, L.V.: Multi-view traffic sign detection, recognition, and 3d localisation. Mach. Vis. Appl. 25(3), 633–647 (2014)CrossRefGoogle Scholar
  35. 35.
    Yang, C., Wu, Q., Li, H., Chen, Y.: Generative poisoning attack method against neural networks (2017)Google Scholar
  36. 36.
    Yang, F., Choi, W., Lin, Y.: Exploit all the layers: fast and accurate CNN object detector with scale dependent pooling and cascaded rejection classifiers. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2129–2137 (2016)Google Scholar
  37. 37.
    Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.School of Mathematics and Information ScienceGuangzhou UniversityGuangzhouChina

Personalised recommendations