Abstract
With the recent increase in the computational power of modern mobile devices, machine learning-based heavy tasks such as face detection and speech recognition are now integral parts of such devices. This requires frameworks to execute machine learning models (e.g., Deep Neural Networks) on mobile devices. Although there exist studies on the accuracy and performance of these frameworks, the quality of on-device deep learning frameworks, in terms of their robustness, has not been systematically studied yet. In this paper, we empirically compare two on-device deep learning frameworks with three adversarial attacks on three different model architectures. We also use both the quantized and unquantized variants for each architecture. The results show that, in general, neither of the deep learning frameworks is better than the other in terms of robustness, and there is not a significant difference between the PC and mobile frameworks either. However, in cases like Boundary attack, mobile version is more robust than PC. In addition, quantization improves robustness in all cases when moving from PC to mobile.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Caffe2—a new lightweight, modular, and scalable deep learning framework. https://caffe2.ai/
Home—pytorch. https://pytorch.org/mobile/home/
Imagenet. http://www.image-net.org/
Inception_v3—pytorch. https://pytorch.org/hub/pytorch_vision_inception_v3/
Introduction to quantization on pytorch—pytorch. https://pytorch.org/blog/introduction-to-quantization-on-pytorch/
Keras applications. https://keras.io/api/applications/
keras-applications/imagenet_utils.py at 1.0.8. https://github.com/keras-team/keras-applications/blob/1.0.8/keras_applications/imagenet_utils.py
Model optimization—tensorflow lite. https://www.tensorflow.org/lite/perfo-rmance/model_optimization
Tensorflow lite—ml for mobile and edge devices. https://www.tensorflow.org/lite
Tensorflow.js—machine learning for javascript developers. https://www.tensorflow.org/js
torchvision—pytorch 1.7.0 documentation. https://pytorch.org/docs/stable/torch-vision/index.html
torchvision.models—torchvision master documentation. https://pytorch.org/vision/stable/models.html
Wordnet—a lexical database for English. https://wordnet.princeton.edu/
Abadi, M., et al.: TensorFlow: a system for large-scale machine learning. In: Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, pp. 265–283. OSDI 2016, USENIX Association, USA (2016)
Brendel, W., Rauber, J., Bethge, M.: Decision-based adversarial attacks: reliable attacks against black-box machine learning models (2018)
Carlini, N., Wagner, D.: Towards evaluating the robustness of neural networks. In: 2017 IEEE Symposium on Security and Privacy (SP), pp. 39–57 (2017). https://doi.org/10.1109/SP.2017.49
Chen, T., et al.: Mxnet: a flexible and efficient machine learning library for heterogeneous distributed systems (2015)
Fawzi, A., Moosavi-Dezfooli, S.M., Frossard, P.: Robustness of classifiers: from adversarial to random noise. In: Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 1632–1640. NIPS 2016, Curran Associates Inc., Red Hook, NY, USA (2016)
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, 7–9 May 2015, Conference Track Proceedings, San Diego, CA, USA (2015). http://arxiv.org/abs/1412.6572
Guo, Q., et al.: An empirical study towards characterizing deep learning development and deployment across different frameworks and platforms. In: 34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019, 11–15 Nov 2019, pp. 810–822. IEEE, San Diego, CA, USA (2019). https://doi.org/10.1109/ASE.2019.00080
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, 27–30 June 2016, pp. 770–778. IEEE Computer Society, Las Vegas, NV, USA (2016). https://doi.org/10.1109/CVPR.2016.90
Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks (2018)
Huang, Y., Hu, H., Chen, C.: Robustness of on-device models: adversarial attack to deep learning models on android apps (2021)
Ignatov, A., et al.: AI benchmark: running deep neural networks on android smartphones. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11133, pp. 288–314. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11021-5_19
Jacob, B., et al.: Quantization and training of neural networks for efficient integer-arithmetic-only inference (2017)
Kurakin, A., Goodfellow, I., Bengio, S.: Adversarial examples in the physical world (2017)
Luo, C., He, X., Zhan, J., Wang, L., Gao, W., Dai, J.: Comparison and benchmarking of AI models and frameworks on mobile devices (2020)
Moosavi-Dezfooli, S.M., Fawzi, A., Frossard, P.: DeepFool: a simple and accurate method to fool deep neural networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2574–2582 (2016). https://doi.org/10.1109/CVPR.2016.282
Paszke, A., et al.: Automatic differentiation in pytorch (2017)
Rauber, J., Zimmermann, R., Bethge, M., Brendel, W.: Foolbox native: fast adversarial attacks to benchmark the robustness of machine learning models in pytorch, tensorflow, and jax. J. Open Source Softw. 5(53), 2607 (2020). https://doi.org/10.21105/joss.02607
Sandler, M., Howard, A.G., Zhu, M., Zhmoginov, A., Chen, L.: MobileNetV2: inverted residuals and linear bottlenecks. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, 18–22 June 2018, pp. 4510–4520. IEEE Computer Society, Salt Lake City, UT, USA (2018). https://doi.org/10.1109/CVPR.2018.00474
Seide, F., Agarwal, A.: CNTK: microsoft’s open-source deep-learning toolkit. In: Krishnapuram, B., Shah, M., Smola, A.J., Aggarwal, C.C., Shen, D., Rastogi, R. (eds.) Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 13–17 Aug 2016, p. 2135. ACM, San Francisco, CA, USA (2016). https://doi.org/10.1145/2939672.2945397
Su, J., Vargas, D.V., Sakurai, K.: One pixel attack for fooling deep neural networks. IEEE Trans. Evol. Comput. 23(5), 828–841 (2019). https://doi.org/10.1109/TEVC.2019.2890858
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, 27–30 June 2016, pp. 2818–2826. IEEE Computer Society, Las Vegas, NV, USA (2016). https://doi.org/10.1109/CVPR.2016.308
Wu, H., Judd, P., Zhang, X., Isaev, M., Micikevicius, P.: Integer quantization for deep learning inference: principles and empirical evaluation (2020)
Acknowledgement
This work was enabled in part by support from WestGrid (www.westgrid.ca) and Compute Canada (www.computecanada.ca) and the Natural Sciences and Engineering Research Council of Canada [RGPIN/04552-2020].
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 IFIP International Federation for Information Processing
About this paper
Cite this paper
Eslami Abyane, A., Hemmati, H. (2022). Robustness Analysis of Deep Learning Frameworks on Mobile Platforms. In: Clark, D., Menendez, H., Cavalli, A.R. (eds) Testing Software and Systems. ICTSS 2021. Lecture Notes in Computer Science, vol 13045. Springer, Cham. https://doi.org/10.1007/978-3-031-04673-5_13
Download citation
DOI: https://doi.org/10.1007/978-3-031-04673-5_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-04672-8
Online ISBN: 978-3-031-04673-5
eBook Packages: Computer ScienceComputer Science (R0)
