VEDAR: Accountable Behavioural Change Detection

Abstract

With exponential increase in the availability of telemetry/streaming/real time data, understanding contextual behavior changes is a vital functionality in order to deliver unrivalled customer experience and build high performance and high availability systems. Real time behavior change detection finds a use case in number of domains such as social networks, network traffic monitoring, ad exchange metrics, etc. In streaming data, behavior change is an implausible observation that does not fit in with the distribution of rest of the data. A timely and precise revelation of such behavior changes can give substantial information about the system in critical situations which can be a driving factor for vital decisions. Detecting behavior changes in streaming fashion is a difficult task as the system needs to process high speed real time data and continuously learn from data along with detecting anomalies in a single pass of data. This paper illustrates a novel algorithm called Accountable Behavior Change Detection (VEDAR) which can detect and elucidate the behavior changes in real time and operates in a fashion similar to human perception. The algorithm is bench-marked by comparing its performance on open source anomaly data sets against industry standard algorithms like Numenta HTM and Twitter AdVec (SH-ESD). The proposed algorithm outperforms above mentioned algorithms for behaviour change detection, efficacy is given in Sect. 5.