Multimodal anomaly detection for assistive robots
Detecting when something unusual has happened could help assistive robots operate more safely and effectively around people. However, the variability associated with people and objects in human environments can make anomaly detection difficult. We previously introduced an algorithm that uses a hidden Markov model (HMM) with a log-likelihood detection threshold that varies based on execution progress. We now present an improved version of our previous algorithm (HMM-D) and introduce a new algorithm based on Gaussian process regression (HMM-GP). We also present a new and more thorough evaluation of 8 anomaly detection algorithms with force, sound, and kinematic signals collected from a robot closing microwave doors, latching a toolbox, scooping yogurt, and feeding yogurt to able-bodied participants. Overall, HMM-GP had the highest performance in terms of area under the curve for these real-world tasks, and multiple modalities improved performance with some anomalies being better detected with particular modalities. With synthetic anomalies, HMM-D exhibited shorter detection delays and outperformed HMM-GP with high-magnitude anomalies. In general, higher-magnitude synthetic anomalies tended to be detected more rapidly.
KeywordsMultimodality Anomaly detection Assistive manipulation Execution monitoring
We thank Youkeun Kim, Zackory Erickson, Ariel Kapusta, Chansu Kim, and Jane Chisholm for their assistance throughout this project. This work was supported in part by NSF Awards IIS-1150157, EFRI-1137229, and NIDILRR Grant 90RE5016-01-00 via RERC TechSAge. Dr. Kemp is a cofounder, a board member, an equity holder, and the CTO of Hello Robot, Inc., which is developing products related to this research. This research could affect his personal financial status. The terms of this arrangement have been reviewed and approved by Georgia Tech in accordance with its conflict of interest policies.
- Ando, S., Thanomphongphan, T., Hoshino, D., Seki, Y., & Suzuki, E. (2011). ACE: Anomaly clustering ensemble for multi-perspective anomaly detection in robot behaviors. In Proceedings of the international conference on data mining (pp. 1–12). SIAM.Google Scholar
- Argall, B. D. (2016). Modular and adaptive wheelchair automation. In M. A. Hsieh, O. Khatib, & V. Kumar (Eds.), Experimental robotics (pp. 835–848). Springer.Google Scholar
- Bittencourt, A. C., Saarinen, K., Sander-Tavallaey, S., Gunnarsson, S., & Norrlöf, M. (2014). A data-driven approach to diagnostics of repetitive processes in the distribution domain-applications to gearbox diagnostics in industrial robots and rotating machines. Mechatronics, 24(8), 1032–1041.CrossRefGoogle Scholar
- Blank, S., Pfister, T., & Berns, K. (2011). Sensor failure detection capabilities in low-level fusion: a comparison between fuzzy voting and kalman filtering. In IEEE international conference on robotics and automation (ICRA) (pp 4974–4979). IEEE.Google Scholar
- Brambilla, D., Capisani, L. M., Ferrara, A., & Pisu, P. (2008). Fault detection for robot manipulators via second-order sliding modes. IEEE Transactions on Industrial Electronics, 55(11), 3954–3963. https://doi.org/10.1109/TIE.2008.2005932. ISSN 0278-0046.
- Chen, T. L., Ciocarlie, M., Cousins, S., Grice, P., Hawkins, K., Hsiao, et al. (2013). Robots for humanity: Using assistive robots to empower people with disabilities. IEEE Robotics & Automation Magazine, 20(1), 30–39.Google Scholar
- Chu, V., McMahon, I., Riano, L., McDonald, C. G., He, Q., Perez-Tejada, J. M., et al. (2013). Using robotic exploratory procedures to learn the meaning of haptic adjectives. In IEEE international conference on robotics and automation (ICRA) (pp. 3048–3055). IEEE.Google Scholar
- Ciocarlie, M., Hsiao, K., Leeper, A., & Gossow, D. (2012). Mobile manipulation through an assistive home robot. In IEEE/RSJ international conference on intelligent robots and systems. IEEE.Google Scholar
- Copilusi, C., Kaur, M., & Ceccarelli, M. (2015a). Lab experiences with LARM clutched arm for assisting disabled people (pp. 603–611). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-09411-3_64. ISBN: 978-3-319-09411-3.
- Copilusi, C., Kaur, M., & Ceccarelli, M., (2015b). Lab experiences with larm clutched arm for assisting disabled people. In New trends in mechanism and machine science (pp. 603–611). Springer.Google Scholar
- Cressie, N. A. C. (1993). Statistics for spatial data. New York, CH: Wiley series in probability and mathematical statistics, Wiley. ISBN: 978-0-471-00255-0. http://opac.inria.fr/record=b1085822.
- Dames, P. M., Schwager, M., & Rus, D. (2016). Active magnetic anomaly detection using multiple micro aerial vehicles. IEEE Robotics and Automation Letters, 1(1), 153–160. https://doi.org/10.1109/LRA.2015.2511444. ISSN: 2377-3766.
- Di Lello, E., Klotzbucher, M., De Laet, T., & Bruyninckx, H. (2013). Bayesian time-series models for continuous fault detection and recognition in industrial robotic tasks. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 5827–5833). IEEE.Google Scholar
- Eclipse Automation. (2016). Meet obi, a robot that helps disabled individuals eat unassisted. https://meetobi.com/. Accessed 15 July 2017.
- Fagogenis, G., Carolis, V. D., & Lane, D. M. (2016). Online fault detection and model adaptation for underwater vehicles in the case of thruster failures. In IEEE international conference on robotics and automation. IEEE.Google Scholar
- Fiala, M. (2005). Artag, a fiducial marker system using digital techniques. In IEEE computer society conference on computer vision and pattern recognition (Vol. 2, pp. 590–596). IEEE.Google Scholar
- Fujii, H., Yamashita, A., & Asama, H., (2016). Defect detection with estimation of material condition using ensemble learning for hammering test. In IEEE international conference on robotics and automation. IEEE.Google Scholar
- Gehring, J., Miao, Y., Metze, F., & Waibel, A. (2013). Extracting deep bottleneck features using stacked auto-encoders. In Proceedings of IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 3377–3381). IEEE.Google Scholar
- Graf, B., Reiser, U., Hägele, M., Mauz, K., & Klein, P. (2009). Robotic home assistant care-o-bot® 3-product vision and innovation platform. In IEEE workshop on advanced robotics and its social impacts (ARSO) (pp. 139–144). IEEE.Google Scholar
- Haidu, A., Kohlsdorf, D., & Beetz, M., (2015) Learning action failure models from interactive physics-based simulations. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 5370–5375). IEEE.Google Scholar
- Hawkins, K. P., Grice, P. M., Chen, T. L., King, C.-H., & Kemp, C. C. (2014) Assistive mobile manipulation for self-care tasks around the head. In IEEE symposium on computational intelligence in robotic rehabilitation and assistive technologies (pp. 16–25). IEEE.Google Scholar
- Hung, Y.-X., Chiang, C.-Y., Hsu, S.J., & Chan, C.-T., (2010). Abnormality detection for improving elders daily life independent. In International conference on smart homes and health telematics (pp. 186–194). Springer.Google Scholar
- Jiménez Villarreal, J., & Ljungblad, S. (2011) Experience centred design for a robotic eating aid. In Proceedings of the 6th international conference on human–robot interaction (pp. 155–156). ACM.Google Scholar
- Kappler, D., Pastor, P., Kalakrishnan, M., Wuthrich, M., & Schaal, S. (2015). Data-driven online decision making for autonomous manipulation. In Proceedings of robotics: Science and systems.Google Scholar
- Kazemi, V., & Sullivan, J. (2014) One millisecond face alignment with an ensemble of regression trees. In IEEE conference on computer vision and pattern recognition (pp. 1867–1874). https://doi.org/10.1109/CVPR.2014.241.
- Ketabdar, H., Vepa, J, Bengio, S., & Bourlard, H. (2006) Using more informative posterior probabilities for speech recognition. In IEEE international conference on acoustics speech and signal processing proceedings (Vol. 1, pp. 1). IEEE.Google Scholar
- Khan, S. S., Karg, M. E., Hoey, J., & Kulic, D. (2012). Towards the detection of unusual temporal events during activities using HMMs. In Proceedings of the ACM conference on ubiquitous computing (pp. 1075–1084). ACM.Google Scholar
- Kim, D. J., Wang, Z., Paperno, N., & Behal, A. (2014). System design and implementation of UCF-MANUS—an intelligent assistive robotic manipulator. IEEE/ASME transactions on mechatronics, 19(1), 225–237. https://doi.org/10.1109/TMECH.2012.2226597. ISSN: 1083-4435.
- King, C. H., Chen, T. L., Jain, A., & Kemp, C. C. (2010). Towards an assistive robot that autonomously performs bed baths for patient hygiene. In IEEE/RSJ international conference on intelligent robots and systems (pp. 319–324). https://doi.org/10.1109/IROS.2010.5649101.
- King, D. E. (2009). Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 10, 1755–1758.Google Scholar
- Lee, H., Hwang, B., & Cho, S. (2002). Analysis of novelty detection properties of autoassociative mlp. Journal of Korean Institute of Industrial Engineers, 28(2), 147–161.Google Scholar
- Leidner, D., Dietrich, A., Beetz, M., & Albu-Schäffer, A. (2016). Knowledge-enabled parameterization of whole-body control strategies for compliant service robots. Autonomous Robots, 40(3), 519–536. https://doi.org/10.1007/s10514-015-9523-3. ISSN: 1573-7527.
- Lepora, N. F., Pearson, M. J., Mitchinson, B., Evans, M., Fox, C., Pipe, A., Gurney, K., & Prescott, T. J. (2010) Naive bayes novelty detection for a moving robot with whiskers. In IEEE international conference on robotics and biomimetics (ROBIO) (pp. 131–136). IEEE.Google Scholar
- Lühr, S., Venkatesh, S., West, G., & Bui, H. H. (2004). Explicit state duration HMM for abnormality detection in sequences of human activity. In Pacific rim international conference on artificial intelligence (pp. 983–984). Springer.Google Scholar
- Malhotra, P., Vig, L., Shroff, G., & Agarwal, P., (2015). Long short term memory networks for anomaly detection in time series. In 23rd European symposium on artificial neural networks, computational intelligence and machine learning (p. 89).Google Scholar
- Marcolino, F., & Wang, J. (2013). Detecting anomalies in humanoid joint trajectories. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 2594–2599). IEEE.Google Scholar
- Mathieu, B., Essid, S., Fillon, T., Prado, J., & Richard, G. (2010). Yaafe, an easy to use and efficient audio feature extraction software. In Proceedings of the 11th international conference on music information retrieval (ISMIR (pp. 441–446).Google Scholar
- Mealtime Partners. (2017). Specializing in assistive dining and drinking equipment. http://www.mealtimepartners.com/. Accessed July 15, 2017.
- Mendoza, J. P., Veloso, M., & Simmons, R. (2014). Focused optimization for online detection of anomalous regions. In IEEE international conference on robotics and automation (ICRA) (pp. 3358–3363). IEEE.Google Scholar
- Morris, B. T., & Trivedi, M. M. (2008). Learning and classification of trajectories in dynamic scenes: A general framework for live video analysis. In IEEE fifth international conference on advanced video and signal based surveillance (AVSS) (pp. 154–161). IEEE.Google Scholar
- Nguyen, H., Anderson, C., Trevor, A., Jain, A., Xu, Z., & Kemp, C. C. (2008). El-e: An assistive robot that fetches objects from flat surfaces. In Robotic helpers, international conference on human–robot interaction.Google Scholar
- Niekum, S., Osentoski, S., Atkeson, C. G., & Barto, A. G., (2014). Learning articulation changepoint models from demonstration. In Robotics science and systems (RSS) workshop on learning plans with context from human signals.Google Scholar
- Ogorodnikova, O. (2008). Methodology of safety for a human robot interaction designing stage. In Conference on human system interactions (pp. 452–457). IEEE.Google Scholar
- Papageorgiou, X. S., Tzafestas, C. S., Maragos, P., Pavlakos, G., Chalvatzaki, G., Moustris, G., et al. (2014). Advances in intelligent mobility assistance robot integrating multimodal sensory processing. In International conference on universal access in human–computer interaction (pp. 692–703). Springer.Google Scholar
- Park, D., Erickson, Z., Bhattacharjee, T., & Kemp, C. C. (2016a). Multimodal execution monitoring for anomaly detection during robot manipulation. In IEEE international conference on robotics and automation. IEEE.Google Scholar
- Park, D., Kim, Y. K., Erickson, Z., & Kemp, C. C. (2016b). Towards assistive feeding with a general-purpose mobile manipulator. In IEEE international conference on robotics and automation-workshop on human–robot interfaces for enhanced physical interactions.Google Scholar
- Pastor, P., Kalakrishnan, M., Chitta, E., Theodorou, S., & Schaal, S. (2011). Skill learning and task outcome prediction for manipulation. In IEEE international conference on robotics and automation (ICRA) (pp. 3828–3834). IEEE.Google Scholar
- Patterson Medical. Meal buddy, (2017). http://pattersonmedical.com/. Accessed: July 15, 2017.
- Rabiner, L. R. (1989). A tutorial on hidden markov models and selected applications in speech recognition. In Proceedings of the IEEE (pp. 257–286).Google Scholar
- Rasmussen, C. E. (2004). Gaussian processes in machine learning. In Advanced lectures on machine learning (pp. 63–71). Springer.Google Scholar
- Rasmussen, C. E., Kuss, M., et al. (2003). Gaussian processes in reinforcement learning. In Advances in neural information processing systems (Vol. 4, p. 1).Google Scholar
- Rodriguez, A., Bourne, D., Mason, M., Rossano, G. F., & Wang, J., (2010). Failure detection in assembly: Force signature analysis. In IEEE conference on automation science and engineering (CASE) (pp. 210–215). IEEE.Google Scholar
- Rodriguez, A., Mason, M. T., Srinivasa, S., Bernstein, M., & Zirbel, A. (2011). Abort and retry in grasping. In IEEE international conference on intelligent robots and systems (IROS).Google Scholar
- Sakaguchi, T., Yokoi, K., Ujiie, T., Tsunoo, S., & Wada, K. (2009). Design of common environmental information for door-closing tasks with various robots. International Journal of Robotics and Automation, 24(3), 203.Google Scholar
- Schliep, A., Rungsarityotin, W., & Georgi, B. (2004). General hidden markov model library. http://www.ghmm.org/.
- Schrer, S., Killmann, I., Frank, B., Vlker, M., Fiederer, L., Ball, T., & Burgard, W. (2015). An autonomous robotic assistant for drinking. In IEEE international conference on robotics and automation (ICRA) (pp. 6482–6487). https://doi.org/10.1109/ICRA.2015.7140110.
- Silvrio, J., Rozo, L., Calinon, S., & Caldwell, D. G. (2015). Learning bimanual end-effector poses from demonstrations using task-parameterized dynamical systems. In 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 464–470). https://doi.org/10.1109/IROS.2015.7353413.
- Snelson, E., & Ghahramani, Z. (2006). Sparse gaussian processes using pseudo-inputs. Advances in Neural Information Processing Systems, 18, 1257.Google Scholar
- Sölch, M., Bayer, J., Ludersdorfer, M., & van der Smagt, P. (2016). Variational inference for on-line anomaly detection in high-dimensional time series. arXiv preprint arXiv:1602.07109.
- Suarez, A., Heredia, G., & Ollero, A., (2016). Cooperative sensor fault recovery in multi-UAV systems. In IEEE international conference on robotics and automation. IEEE.Google Scholar
- Suetani, H., Ideta, A. M., & Morimoto, J. (2011). Nonlinear structure of escape-times to falls for a passive dynamic walker on an irregular slope: Anomaly detection using multi-class support vector machine and latent state extraction by canonical correlation analysis. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 2715–2722). IEEE.Google Scholar
- Sukhoy, V., Georgiev, V., Wegter, T., Sweidan, R., & Stoytchev, A., (2012). Learning to slide a magnetic card through a card reader. In IEEE international conference on robotics and automation (ICRA) (pp. 2398–2404). IEEE.Google Scholar
- Takahashi, Y., & Suzukawa, S. (2006). Easy human interface for severely handicapped persons and application for eating assist robot. In IEEE international conference on mechatronics (pp. 225–229). https://doi.org/10.1109/ICMECH.2006.252529.
- Vasic, M., & Billard, A. (2013). Safety issues in human–robot interactions. In IEEE international conference on robotics and automation (ICRA) (pp. 197–204). IEEE.Google Scholar
- Wakita, Y., Yoon, W. K., & Yamanobe, W. K., (2012). User evaluation to apply the robotic arm rapuda for an upper-limb disabilities patient’s daily life. In IEEE international conference on robotics and biomimetics (ROBIO) (pp. 1482–1487). https://doi.org/10.1109/ROBIO.2012.6491178.
- Warrender, C., Forrest, S., & Pearlmutter, B. (1999). Detecting intrusions using system calls: Alternative data models. In Proceedings of the IEEE symposium on security and privacy (pp. 133–145). IEEE.Google Scholar
- Williams, G., Baxter, R., He, H., Hawkins, S., & Gu, L. (2002). A comparative study of RNN for outlier detection in data mining. In Proceedings of IEEE international conference on data mining, ICDM (pp. 709–712). IEEE.Google Scholar
- Yamazaki, K., Ueda, R., Nozawa, S., Mori, Y., Maki, T., Hatao, N., et al. (2009). A demonstrative research for daily assistive robots on tasks of cleaning and tidying up rooms. In First international symposium on quality of life technology (June 2009).Google Scholar
- Yu, D., & Seltzer, M L., (2011). Improved bottleneck features using pretrained deep neural networks. In INTERSPEECH (Vol. 237, p. 240).Google Scholar