Mixed Reality-Based Platform for Smart Cockpit Design and User Study for Self-driving Vehicles

  • Xiaohua SunEmail author
  • Shiyu Wu
  • Shengchen Zhang
  • Hanlin Wang
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 903)


The change of the role people plays in autonomous driving leads to different use scenario and user behavior, which should be considered in the design of smart cockpit to enable better experience. However, the smart cockpit design and user study under autonomous driving conditions are difficult to achieve by existing approaches due to high cost and the limitation current methods pose on designer imagination. Mixed reality incorporates the ability of virtual reality to simulate immersive scenarios and events, and combines both virtual and real worlds, thus enables further natural user interaction and intuitive experience. We present in this paper a platform for designers to build virtual cockpit and validate their design based on mixed reality. In innovation and design phase, the platform provides basic modularized components which can be assembled to various usable space and HMI. It helps designers to explore the possibility of cockpit space of self-driving vehicles unlimitedly and build it conveniently according to different scenarios and functions. In user validation phase, designers can do user study for designed cockpit in virtual potential use scenarios, integrating a series of driving condition and events. With the support of mixed reality, the platform can provide more natural experience of some components (seats, screens, etc.) to better validate and optimize the design.


Self-driving vehicles Smart cockpit Modular components Design validation User experience 


  1. 1.
    Sotelo, M., Rodriguez, F., Magdalena, L., Bergasa, L., Boquete, L.: A color vision-based lane tracking system for self-driving on unmarked roads. Self-driving Robots. 16, 95–116 (2004)Google Scholar
  2. 2.
    Levinson, J., et al.: Towards fully self-driving: Systems and algorithms. In: Intelligent Vehicles Symposium (IV), pp. 163–168. IEEE (2011)Google Scholar
  3. 3.
    Martens, M.H., Wilschut, E.S., Pauwelussen, J.: Semi-autonomous driving: do drivers still respond to unexpected events?. In: 15th World Congress on Intelligent Transport Systems and ITS America’s 2008 Annual MeetingITS AmericaERTICOITS JapanTransCore (2008)Google Scholar
  4. 4.
    Reimer, B., et al.: Behavioral impact of drivers’ roles in automated driving. In: Proceedings of the 8th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 217–224. ACM (2016)Google Scholar
  5. 5.
    Walch, M., Lange, K., Baumann, M., Weber, M.: Autonomous driving: investigating the feasibility of car-driver handover assistance. In: Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 11–18. ACM (2015)Google Scholar
  6. 6.
    Schieben, A., Temme, G., Köster, F., Flemisch, F.: How to interact with a highly automated vehicle. Generic interaction design schemes and test results of a usability assessment, pp. 251–267. DLR (2011)Google Scholar
  7. 7.
    Park, H.: In-vehicle AR-HUD system to provide driving-safety information. ETRI J. 35, 1038–1047 (2013)CrossRefGoogle Scholar
  8. 8.
    BMW unveiled its BMW i Inside Future sculpture at CES (2017).
  9. 9.
    Sun, X., Li, T.: Survey of studies supporting the use of in-air gesture in car HMI. UXPA, China (2014)Google Scholar
  10. 10.
    Schmidt, R., Weisser, H., Schulenberg, P., Goellinger, H.: Autonomous driving on vehicle test tracks: overview, implementation and results. In: Proceedings of the IEEE Intelligent Vehicles Symposium 2000, IV 2000, pp. 152–155. IEEE (2002)Google Scholar
  11. 11.
    Rebelo, F., Noriega, P., Duarte, E., Soares, M.: Using virtual reality to assess user experience. Hum. Factors 54(6), 964–982 (2012)CrossRefGoogle Scholar
  12. 12.
    Desai, S., Blackler, A., Popovic, V.: Intuitive interaction in a mixed reality system. In: 2016 Design Research Society 50th Anniversary Conference. Brighton (2016)Google Scholar
  13. 13.
    Pacheco, C., Duarte, E., Rebelo, F., Teles, J.: Using virtual reality in the design of indoor environments: selection and evaluation of wall colors by a group of elderly. In: Kaber, D.B., Boy, G. (eds.) Advances in Cognitive Ergonomics. Advances in Human Factors and Ergonomics Series, pp. 784–792. CRC Press, Miami (2010)CrossRefGoogle Scholar
  14. 14.
    Purschke, F., Schulze, M., Zimmermann, P.: Virtual reality - new methods for improving and accelerating the development process in vehicle styling and design. In: Computer Graphics International, p. 789. IEEE Computer Society (1998)Google Scholar
  15. 15.
    Noon, C., Zhang, R., Winer, E., Oliver, J., Gilmore, B., Duncan, J.: A system for rapid creation and assessment of conceptual large vehicle designs using immersive virtual reality. Comput. Ind. 63(5), 500–512 (2012)CrossRefGoogle Scholar
  16. 16.
    Meyrueis, V., Paljic, A., Leroy, L., Fuchs, P.: A template approach for coupling virtual reality and CAD in an immersive car interior design scenario. Int. J. Prod. Dev. 18(5), 395–410 (2017)CrossRefGoogle Scholar
  17. 17.
    Sportillo, D., Paljic, A., Boukhris, M., Fuchs, P., Ojeda, L., Roussarie, V.: An immersive virtual reality system for semi-autonomous driving simulation: a comparison between realistic and 6-DoF controller-based interaction. In: The International Conference on Computer and Automation Engineering, pp. 6–10 (2017)Google Scholar
  18. 18.
    Bajana, J., Francia, D., Liverani, A., Krajčovič, M.: Mobile tracking system and optical tracking integration for mobile mixed reality. Int. J. Comput. Appl. Technol. 53, 13 (2016)CrossRefGoogle Scholar
  19. 19.
    LaViola, J., Kruijff, E., McMahan, R., Bowman, D., Poupyrev, I.: 3D User Interfaces. Addison-Wesley, Boston (2017)Google Scholar
  20. 20.
  21. 21.
    Dery-Pinna, A.M., Fierstone, J., Picard, E.: Component model and programming: a first step to manage human computer interaction adaptation. In: International Conference on Mobile Human-Computer Interaction, pp. 456–460. Springer, Heidelberg (2003)Google Scholar
  22. 22.
    Koo, J., Kwac, J., Ju, W., Steinert, M., Leifer, L., Nass, C.: Why did my car just do that? Explaining semi-autonomous driving actions to improve driver understanding, trust, and performance. Int. J. Interact. Des. Manuf. (IJIDeM) 9, 269–275 (2014)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Xiaohua Sun
    • 1
    Email author
  • Shiyu Wu
    • 1
  • Shengchen Zhang
    • 1
  • Hanlin Wang
    • 2
  1. 1.College of Design and InnovationTongji UniversityShanghaiChina
  2. 2.Philips Design CenterShanghaiChina

Personalised recommendations