High Performance Heterogeneous Data Storage System for High Frequency Sensor Data in a Landslide Laboratory

Abstract

Wireless sensor networks can be deployed in landslide prone areas to monitor various geological and weather properties to detect a possible landslide and provide early warning for evacuation. Landslide lab is used to mirror geological conditions of target deployment site for the purposes of testing and perfecting the deployment locations of sensors and associated data models for accurate predictions under various simulated conditions. The landslide lab simulates the full cycle of landslide occurrence on a compact time window of few hours. Hence it’s critical to capture sensor data at very high frequency, ranging from 100 samples to 1000 samples per second to understand all the minute changes to the geological properties leading to landslide occurrence. A typical test scenario using 16 sensors, sampling at 1000 samples per second, running for 8 h would generate 460 million data points. A very finely tuned RDBMS system would take 25 h to store this data at the rate of 5000 records/s at peak i/o rate and even NoSQL data store would not be able to efficiently retrieve this huge volumes of data. We prove that our system outperforms the NoSQL data store by an order of magnitude and the RDBMS data store by two orders of magnitudes in both storage and retrieval of the high frequency sensor data. In this paper we present the architecture, features and performance metrics of our high performance, scalable, and distributed heterogeneous system for data capturing, processing, and retrieval of high frequency sensor data.