Skip to main content
SpringerLink
Log in

HCI Based In-Cabin Monitoring System for Irregular Situations with Occupants Facial Anonymization

  • Conference paper
  • First Online:
Intelligent Human Computer Interaction (IHCI 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12616))

Included in the following conference series:

Abstract

Absence of human in-charge in fully autonomous vehicles (FAVs) impose multi-pronged in-cabin monitoring in real-time to monitor any irregular situation. Moreover, occupant’s visual information (such as video, images) transmission from in-cabin of vehicle to the data center and control-room is required for various in-cabin monitoring tasks. However, this information transfer may cause a substantial threat to the occupants. Though applying some patch on face of the occupants protects personal information however, simultaneously it deteriorates the important facial information which is very crucial in monitoring various tasks such as emotion detection. Therefore, a human-computer-interaction (HCI) is required to manage this trade-off between facial information and facial anonymization without any important information loss. Also, this HCI based in-cabin monitoring system should be implicit to retain the users trust in the intelligent transportation system (ITS). In this paper, we have proposed generative adversarial network (GAN) based in-cabin monitoring approach for emotion detection in FAV. In this method, we proposed to generate an artificial (virtual) face having real-facial expressions to provide facial anonymity to the occupants while monitoring FAV cabin. Therefore, this method provides anonymous facial information which is essentially required in detection and monitoring tasks to avoid irregular situations. We have used both publicly available and our own in-cabin dataset to assess our proposed approach. Our experiments show satisfactory results.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Koopman, P., Wagner, M.: Autonomous vehicle safety: An interdisciplinary challenge. IEEE Intell. Transp. Syst. Mag. 9(1), 90–96 (2017)

    Article  Google Scholar 

  2. Yurtsever, E., Lambert, J., Carballo, A., Takeda, K.: A survey of autonomous driving: common practices and emerging technologies. IEEE Access 8, 58443–58469 (2020)

    Article  Google Scholar 

  3. Skrickij, V., Šabanovič, E., Žuraulis, V.: Autonomous road vehicles: recent issues and expectations. IET Intel. Transport Syst. 14(6), 471–479 (2020)

    Article  Google Scholar 

  4. Marcondes, F., Durães, D., Gonçalves, F., Fonseca, J., Machado, J., Novais, P.: In-vehicle violence detection in carpooling: a brief survey towards a general surveillance system. In: Dong, Y. (ed.) DCAI 2020. AISC, vol. 1237, pp. 211–220. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-53036-5_23

    Chapter  Google Scholar 

  5. Khan, M.Q., Lee, S.: A comprehensive survey of driving monitoring and assistance systems. Sensors 19(11), 2574 (2019)

    Article  Google Scholar 

  6. Bell, J.L., Taylor, M.A., Chen, G.X., Kirk, R.D., Leatherman, E.R.: Evaluation of an in-vehicle monitoring system (IVMS) to reduce risky driving behaviors in commercial drivers: comparison of in-cab warning lights and supervisory coaching with videos of driving behavior. J. Saf. Res. 60, 125–136 (2017)

    Article  Google Scholar 

  7. Szawarski, H., Le, J., Rao, M.K.: Monitoring a vehicle cabin. U.S. Patent 10,252,688, issued April 9 (2019)

    Google Scholar 

  8. Song, X.: Safety and clean vehicle monitoring system. U.S. Patent 10,196,070, issued February 5 (2019)

    Google Scholar 

  9. Fridman, L., et al.: MIT advanced vehicle technology study: large scale naturalistic driving study of driver behavior and interaction with automation. IEEE Access 7, 102021–102038 (2019)

    Article  Google Scholar 

  10. Bosch, E., et al.: Emotional GaRage: a workshop on in-car emotion recognition and regulation. In: Adjunct Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 44–49 (2018)

    Google Scholar 

  11. Qin, Z., Weng, J., Cui, Y., Ren, K.: Privacy-preserving image processing in the cloud. IEEE Cloud Comput. 5(2), 48–57 (2018)

    Article  Google Scholar 

  12. Xia, Z., Zhu, Y., Sun, X., Qin, Z., Ren, K.: Towards privacy-preserving content-based image retrieval in cloud computing. IEEE Trans. Cloud Comput. 6(1), 276–286 (2015)

    Article  Google Scholar 

  13. Taeihagh, A., Lim, H.S.M.: Governing autonomous vehicles: emerging responses for safety, liability, privacy, cybersecurity, and industry risks. Transp. Rev. 39(1), 103–128 (2019)

    Article  Google Scholar 

  14. Glancy, D.J.: Privacy in autonomous vehicles. Santa Clara L. Rev. 52, 1171 (2012)

    Google Scholar 

  15. Collingwood, L.: Privacy implications and liability issues of autonomous vehicles. Inf. Commun. Technol. Law 26(1), 32–45 (2017)

    Article  Google Scholar 

  16. Lim, H.S.M., Taeihagh, A.: Autonomous vehicles for smart and sustainable cities: an in-depth exploration of privacy and cybersecurity implications. Energies 11(5), 1062 (2017)

    Article  Google Scholar 

  17. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples (2014)

    Google Scholar 

  18. Braunegg, A., et al.: APRICOT: A Dataset of Physical Adversarial Attacks on Object Detection. arXiv preprint arXiv:1912.08166 (2019)

  19. Fox, E., Lester, V., Russo, R., Bowles, R.J., Pichler, A., Dutton, K.: Facial expressions of emotion: are angry faces detected more efficiently? Cogn. Emot. 14(1), 61–92 (2000)

    Article  Google Scholar 

  20. Blanz, V., Scherbaum, K., Vetter, T., Seidel, H.P.: Exchanging faces in images. In: Computer Graphics Forum, vol. 23, no. 3, pp. 669–676. Blackwell Publishing, Inc., Oxford and Boston (2004)

    Google Scholar 

  21. Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., Nayar, S.K.: Face swapping: automatically replacing faces in photographs. In: ACM SIGGRAPH Papers, pp. 1–8 (2008)

    Google Scholar 

  22. Lin, Y., Wang, S., Lin, Q., Tang, F.: Face swapping under large pose variations: a 3D model based approach. In: IEEE International Conference on Multimedia and Expo, pp. 333–338. IEEE (2012)

    Google Scholar 

  23. Zhang, Y., Zheng, L., Thing, V.L.: Automated face swapping and its detection. In: IEEE 2nd International Conference on Signal and Image Processing (ICSIP), pp. 15–19. IEEE (2017)

    Google Scholar 

  24. Nirkin, Y., Masi, I., Tuan, A.T., Hassner, T., Medioni, G.: On face segmentation, face swapping, and face perception. In: 13th IEEE International Conference on Automatic Face & Gesture Recognition, pp. 98–105. IEEE (2018)

    Google Scholar 

  25. Natsume, R., Yatagawa, T., Morishima, S.: Rsgan: face swapping and editing using face and hair representation in latent spaces. arXiv preprint arXiv:1804.03447 (2018)

  26. Korshunov, P., Marcel, S.: Deepfakes: a new threat to face recognition? Assessment and detection. arXiv preprint arXiv:1812.08685 (2018)

  27. Natsume, R., Yatagawa, T., Morishima, S.: Fsnet: an identity-aware generative model for image-based face swapping. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11366, pp. 117–132. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20876-9_8

    Chapter  Google Scholar 

  28. Bailer, W.: Face swapping for solving collateral privacy issues in multimedia analytics. In: Kompatsiaris, I., Huet, B., Mezaris, V., Gurrin, C., Cheng, W.-H., Vrochidis, S. (eds.) MMM 2019. LNCS, vol. 11295, pp. 169–177. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05710-7_14

    Chapter  Google Scholar 

  29. Nirkin, Y., Keller, Y., Hassner, T.: FSGAN: subject agnostic face swapping and reenactment. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 7184–7193 (2019)

    Google Scholar 

  30. Naruniec, J., Helminger, L., Schroers, C., Weber, R.M.: High‐resolution neural face swapping for visual effects. In: Computer Graphics Forum, vol. 3, no. 4, pp. 173–184 (2020)

    Google Scholar 

  31. Mishra, A., Kim, J., Kim, D., Cha, J., Kim, S.: An intelligent in-cabin monitoring system in fully autonomous vehicles. In: 17th International SoC Conference (ISOCC 2020), South Korea (2020)

    Google Scholar 

  32. Kim, S., Shrestha, R.: Automotive Cyber Security: Introduction, Challenges, and Standardization. Springer Singapore, Singapore (2020). https://doi.org/10.1007/978-981-15-8053-6

    Book  Google Scholar 

  33. Shrestha, R., Kim, S.: Integration of IoT with blockchain and homomorphic encryption: challenging issues and opportunities. In: Advances in Computers, vol. 115, pp. 293–331. Elsevier (2019)

    Google Scholar 

Download references

Acknowledgment

This research was supported by Korea Research Fellowship program funded by the Ministry of Science and ICT through the National Research Foundation of Korea (NRF-2019H1D3A1A01071115).

Author information

Authors and Affiliations

  1. Yonsei Institute of Convergence Technology, Yonsei University, Incheon, South Korea

    Ashutosh Mishra, Jaekwang Cha & Shiho Kim

Authors
  1. Ashutosh Mishra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jaekwang Cha
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shiho Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ashutosh Mishra .

Editor information

Editors and Affiliations

  1. Woosong University, Daejeon, Korea (Republic of)

    Madhusudan Singh

  2. Dongseo University, Busan, Korea (Republic of)

    Dae-Ki Kang

  3. Keimyung University, Daegu, Korea (Republic of)

    Jong-Ha Lee

  4. Indian Institute of Information Technology, Allahabad, India

    Uma Shanker Tiwary

  5. Hankuk University of Foreign Studies, Yongin, Korea (Republic of)

    Dhananjay Singh

  6. Pukyong National University, Busan, Korea (Republic of)

    Wan-Young Chung

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mishra, A., Cha, J., Kim, S. (2021). HCI Based In-Cabin Monitoring System for Irregular Situations with Occupants Facial Anonymization. In: Singh, M., Kang, DK., Lee, JH., Tiwary, U.S., Singh, D., Chung, WY. (eds) Intelligent Human Computer Interaction. IHCI 2020. Lecture Notes in Computer Science(), vol 12616. Springer, Cham. https://doi.org/10.1007/978-3-030-68452-5_40

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-68452-5_40

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-68451-8

  • Online ISBN: 978-3-030-68452-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation