Abstract
Most submissions to the 2018 Sussex-Huawei Locomotion-Transportation (SHL) recognition challenge strongly overestimated the performance of their algorithms in relation to their performance achieved on the challenge evaluation data. Similarly, recent studies on smartphone based trip data collection promise accurate and detailed recognition of various modes of transportation, but it appears that in field tests the available techniques cannot live up to the expectations. In this chapter we experimentally demonstrate potential sources of upward scoring bias in the evaluation of travel activity classifiers. Our results show that (1) performance measures such as accuracy and the average class-wise F1 score are sensitive to class prevalence which can vary strongly across sub-populations, (2) cross-validation with random train/test splits or large number of folds can easily introduce dependencies between training and test data and are therefore not suitable to reveal overfitting, and (3) splitting the data into disjoint subsets for training and test does not always allow to discover model overfitting caused by lack of variation in the data.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bohte W, Maat K, Quak W (2008) A method for deriving trip destinations and modes for GPS-based travel surveys. Res. Urbanism Ser. 1(1):127–143
Ciliberto M, Morales FJO, Gjoreski H, Roggen D, Mekki S, Valentin S (2017) High reliability android application for multidevice multimodal mobile data acquisition and annotation. In: Proceedings of the 15th ACM conference on embedded network sensor systems, p. 62. ACM
Ghosh A, Riccardi G (2014) Recognizing human activities from smartphone sensor signals. In: Proceedings of the 22nd ACM international conference on Multimedia, pp 865–868. ACM (2014)
Gjoreski H, Ciliberto M, Wang L, Morales FJO, Mekki S, Valentin S, Roggen D (2018) The university of sussex-huawei locomotion and transportation dataset for multimodal analytics with mobile devices. IEEE Access. https://doi.org/10.1109/ACCESS.2018.2858933
Harding C, Srikukenthiran S, Zhang Z, Habib KN, Miller E (2017) On the user experience and performance of smartphone apps as personalized travel survey instruments: results from an experiment in Toronto. In: The paper presented at the 11th international conference on transport survey methods, Quebec
Hemminki S, Nurmi P, Tarkoma S (2013) Accelerometer-based transportation mode detection on smartphones. In: Proceedings of the 11th ACM Conference on Embedded Networked Sensor Systems, p 13. ACM (2013)
Liang X, Wang G (2017) A convolutional neural network for transportation mode detection based on smartphone platform. In: 2017 IEEE 14th international conference on mobile Ad Hoc and sensor systems (MASS), pp 338–342. IEEE
Manzoni V, Maniloff D, Kloeckl K, Ratti C (2010) Transportation mode identification and real-time CO\(\_2\) emission estimation using smartphones. Massachusetts Institute of Technology, SENSEable City Lab
Mizell D (2003)Using gravity to estimate accelerometer orientation. In: Null, p 252. IEEE
Morgan N, Bourlard HA (1995) Neural networks for statistical recognition of continuous speech. Proceedings of the IEEE 83(5):742–772
Ronao CA, Cho SB (2017) Recognizing human activities from smartphone sensors using hierarchical continuous hidden Markov models. Int J Distribut Sensor Netw 13(1):1550147716683687
Siirtola P, Röning J (2012) Recognizing human activities user-independently on smartphones based on accelerometer data. IJIMAI 1(5):38–45
Stenneth L, Wolfson O, Yu PS, Xu B (2011) Transportation mode detection using mobile phones and gis information. In: Proceedings of the 19th ACM SIGSPATIAL international conference on advances in geographic information systems, pp 54–63. ACM
Stopher P, FitzGerald C, Xu M (2007) Assessing the accuracy of the Sydney household travel survey with GPS. Transportation 34(6):723–741
Wang L, Gjoreskia H, Murao K, Okita T, Roggen D (2018) Summary of the sussex-huawei locomotion-transportation recognition challenge. In: Proceedings of the 2018 ACM international joint conference and 2018 international symposium on pervasive and ubiquitous computing and wearable computers, pp 1521–1530. ACM
Widhalm P, Leodolter M, Brändle N (2018) Top in the lab, flop in the field?: evaluation of a sensor-based travel activity classifier with the SHL dataset. In: Proceedings of the 2018 ACM international joint conference and 2018 international symposium on pervasive and ubiquitous computing and wearable computers, pp 1479–1487. ACM (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Widhalm, P., Leodolter, M., Brändle, N. (2019). Into the Wild—Avoiding Pitfalls in the Evaluation of Travel Activity Classifiers. In: Kawaguchi, N., Nishio, N., Roggen, D., Inoue, S., Pirttikangas, S., Van Laerhoven, K. (eds) Human Activity Sensing. Springer Series in Adaptive Environments. Springer, Cham. https://doi.org/10.1007/978-3-030-13001-5_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-13001-5_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-13000-8
Online ISBN: 978-3-030-13001-5
eBook Packages: Computer ScienceComputer Science (R0)