1 Introduction

Over 65s are a growing segment of the UK population forecast to rise by another 2 million before 2021. In this age group falls are the most common cause of death (Tian et al. 2013) in the UK, and are an ongoing problem in the rest of the developed world, with an average of 35,848 fall-related deaths occurring annually between 2010 and 2012 (Turner et al. 2015). One study performed in Torbay found 28% of falls proved to either be fatal or become fatal within 12 months, highlighting that research into preventative measures may be a more promising approach than rehabilitation. Falls account for over 4 million bed days in the UK (NHS Improvement 2017). Identifying preventative measures to be taken against falls could reduce morbidity, while also reducing costs and load on health services. On top of the direct physical health concerns, falls have a lasting psychological effect which can drastically reduce a person’s confidence in their independent abilities, leading to an increase in sedentary behaviour and depression (Arfken et al. 1994). Methodologies for anticipating falls are essential as the benefits of prevention far outweigh those of rehabilitation, by improving quality of life and fostering long-lasting independence.

Automatically identifying the potential risk of falling for people would help prevent falls before they occur. Traditionally, most accurate health monitoring has occurred in laboratory or hospital conditions. Several assessments, such as the 400-Meter Walk Test and the One-Leg Standing Test, exist as standards for identifying deterioration of mobility and morbidity (Vestergaard et al. 2009; Ayena et al. 2016). However, laboratory testing is costly and not undertaken routinely. It is an intrusive method of monitoring, which can present challenges when people suffer from mobility issues or other aversions to hospitals. Furthermore, these are one-off performances which may not be representative of a subject’s regular behaviour. With more elderly people struggling to maintain independence and live in their own homes, an automated home-based solution to ease the workload on manned health services and provide regular insight into risk of falling would be an attractive alternative approach. In this paper sensors that can provide the required level of in-home monitoring are investigated and considered, along with data analysis approaches that can identify specific activities or behaviours that may be considered as precursors to falls. The aim is to identify potential components of a system that uses real time monitoring in an assisted-living or smart home environment to identify whether a resident has an increased risk of falling. Early preventive intervention could then replace the need for expensive rehabilitation.

As pervasive computing technologies have become more accessible, continuous monitoring in the home is now possible using off-the-shelf sensor equipment. Perhaps most synonymous with IoT is the smart home, a concept in which a house is fitted with many sensors and devices to deliver a networked home environment. The underlying technologies in simple in-home sensors have remained the same over the last decade with infrared (IR), magnetic switches, temperature sensors, and other passive binary sensor technologies largely retaining their core designs. However, the technologies required to allow even simple sensors to communicate over a network have become smaller and cheaper, making them easier to integrate into a smart home environment. More complex technologies are now also available, such as WiFi, Bluetooth Low-Energy (BLE), Radar and RFID (Adib and Katabi 2013; Faragher and Harle 2014; Hillyard et al. 2015; Bocca et al. 2014), and are typically capable of producing continuous timestream data. Different devices are usually selected for specific tasks. For example, while risk of falling can be determined using either wearable or passive devices (Sucerquia et al. 2017; Phillips et al. 2017), wearables are particularly popular for activity recognition with their focus on user movement and physiological status (Scanaill et al. 2006). Whereas, passive devices have been used to map behavioural information by monitoring sensor changes in relation to room transitions, activities and other habitual events (Ogawa et al. 2002).

Fig. 1
figure 1

Ontology of behavioural modelling for fall prediction

Full size image

The accuracy of lab-based testing is difficult to achieve using off-the-shelf in-home sensor equipment. However continuous monitoring in a smart home provides an opportunity for the resident’s activities to be tracked over time (Juarez et al. 2015), and hence changes in behaviours can be identified. Behavioural Modelling (as seen in Fig. 1) can be considered as either Activities of Daily Living (voluntary) or Physiological expressions (involuntary). Activities of Daily Living (ADLs) are everyday tasks performed in daily life, and the idea is to identify them using in-home passive sensors as regular sequences of similar sensor activations. Identification of ADLs, their frequency and change over time should aid recognition of risky behaviour (Wilson and Atkeson 2005). Likewise, changes in physiological expressions (such as gait, heart rate and blood pressure), while involuntary, may also precede or identify changes in behaviours indicative of change in risk (Albu-Shamah and Zhan 2012).

Fall detection, as opposed to predicting risk of fall, has received considerable attention recently and been achieved effectively using passive sensors (Stone and Skubic 2014). However, risk determination is a more complex problem that offers the benefit of supporting intervention before a fall occurs. The timescale in which fall prediction is performed is key to identifying potential interventions that reduce the negative impact of the fall. Prediction of an imminent fall offers limited opportunity for effective intervention; whereas prediction of a fall weeks before it occurs, supported by continuous in-home monitoring, could allow for supportive recommendations or interventions to prevent future falls. The focus of this paper is to explore monitoring and data analysis techniques that support longer term fall risk determination. The current approaches and future research paths for Behavioural Modelling are highlighted. Then potential Sensor Implementations for capturing behaviour and physiological expressions are explored. Finally the integration of sensors and data analysis for Fall Risk Determination are discussed.

2 Behavioural modelling

Activity recognition is the study of identifying actions from the observed behaviour of an agent. Behavioural monitoring aims to identify sequences of events in human behaviour, supporting assumptions and observations of recurring patterns and behavioural trends. Visualisation provides a quick way to make observations, however by modelling behavioural data, more information can be extracted from events. Assembling profiles of regular activities can support identification of trends in performance during important events, such as ADLs, with changes in behavioural performance possibly attributed to medical conditions and other anomalies.

2.1 Human activity recognition

In pervasive computing, activity recognition is a field of research aimed at identifying the actions and events of humans in real-life settings. For automated healthcare, accurate activity recognition is paramount as it a foundation for the identification of behaviour and decision-making ability in subjects. Activity recognition can be performed with basic binary sensors. Actions and events can be interpreted by monitoring sensor changes as a user moves through their home and uses certain areas and or appliances. These events can be assigned relative importance, and be used to construct a logical sequence of state changes. For instance, if doors at either side of a room were triggered in sequence, it can be assumed that a user has crossed from one side of the room to the other (Ogawa et al. 2002). These assumptions are valid when abstracting simple events, such as getting out of a chair, moving from one room to the other or turning on an appliance. More complex activities however become far more difficult to identify, even when making use of wearable or vision-based sensors.

Some of the challenges in identifying complex activities are a result of dimensional limits in data capture, such as:

  • Recognising concurrent and interleaved activities, in which users can perform multiple actions simultaneously with some overlapping others at varying intervals. Some actions may also interrupt others.

  • Ambiguity of interpretation, where single actions can have multiple meanings under relevant perception.

  • Multiple residents, requiring parallel activities to be recognised for each user.

Research in activity recognition can be summarised as separate fields of pattern discovery and recognition: recognition by which a researcher will create a high-level conceptual model to identify features of an activity as they are expressed through sensor outputs (Kim et al. 2010), alongside pattern discovery where low level sensor data is analysed without assumptions or existing models to find patterns. Pattern discovery identifies previously unknown models for activities, which can then be re-purposed for other recognition projects. Most activity recognition is performed using wearable or vision-based sensors on users (Ann and Theng 2014). Two models: the Hidden Markov Model (HMM) (a generative probabilistic model), and the Conditional Random Field (CRF) (both generative and discriminative) are among the most widely-used techniques in activity recognition.

Markov models in general recognise simple activities accurately and struggle in more complex scenarios, however HMMs can be used to identify a model using outputs from unknown activities. In the context of activity recognition, hidden states are represented by activities, and the observable output is the sensor data. Construction of the model is performed around determining a hidden state sequence complementary to the observable output sequence. The design should also consider ensuring the model parameters are accurately and consistently observable in the output data sequences (Kim et al. 2010). Accuracy of the final model can also be supplemented by introducing some known aspects of the model during the training phase, with some researchers even combining smaller HMMs into large models. Nevertheless, in cases of multiple concurrent or interleaved activities, HMMs can still struggle to produce an accurate result (Cohen 2001).

CRFs are an effective alternative to HMMs in situations where specific observations, which HMMs may not identify, are required. CRFs find the dependence of hidden variables on observed variables. While HMMs find the joint probability distribution to between these two values, CRFs identify a potential function based on conditional probability (Noury and Hadidi 2012). They have been found to be more flexible than HMMs for this reason. CRFs perform better in real life situations since many activities can be performed out of order, concurrently and in an interwoven fashion. In conditions where arbitrary relationships exist among observable sequences, a CRF allows more flexibility than other methods. While HMMs also depend on the assumption of independence between observations, CRFs can infer probabilities based on previous and possible observations (Cohen 2001).

Recent approaches to activity recognition are making effective use of deep learning techniques such as Convolutional Neural Networks (CNN) applied to timeseries data from wearables (Zeng et al. 2014). Recurrent Neural Nets (RNN) employing Long Short-Term Memory (LSTM), offering the ability to model sequential data, have also been applied for human activity recognotion with data from wearables (Ordóñez and Roggen 2016). Similarly, fall detection employing deep learning approaches have also been shown to be effective with both supervised and unsupervised datasets (Núñez-Marcos et al. 2017; Fan et al. 2017). Hence, while deep learning approaches are becoming very popular due to the general increases in processing power available, they tend to be particularly suitable for tasks in which extensive labeled data is available for training. As a result, deep learning research so far focuses mainly on activity recognition and fall detection. Labelled data for activity recognition in fall prediction scenarios can be time consuming to produce, however, there are likely to be applications for deep learning as larger realistic fall datasets become available.

Many activity modelling methods exist, with most focussed on activity-specific algorithmic approaches. Alternatively, a statistical link between behaviour and language has been noted (Aipperspach et al. 2010). Due to their shared evolutionary traits, known as the principle of “Language-as-action” , originally proposed by Vygotsky (1978), they also share similarly applicable statistical features. Aipperspach used existing datasets from the Aware Home (Georgia Institute of Technology 2001) and House_n (Massachusetts Institute of Technology 2005) installations. They found that sensor data output from these datasets showed similar statistical features to text, both following a Zipf distribution (Powers 1998). This approach to behavioural modelling is novel in its assumption that behaviour, like language, is “a result of a rational deliberation process”. While many established methods of activity recognition are prevalent, Aipperspach identified a radical approach which proved successful in their study.

2.1.1 Activities of daily living

ADLs are used to denote events during daily independent life which would be considered vital to a person’s ability to live independently, or not. Typical ADLs may include being able to dress oneself, get out of bed, manage medications, feed oneself and other similar activities. Some scales of ADL function in common use include the Katz ADL index (Wallace 2008), and the Bristol ADL scale (Bucks et al. 1996).

Katz et al. (1963) assumed that there is an “ordered regression [in skills] as part of the natural process of ageing”. Much of Katz’s work has recently been criticised, however his work was widely influential at the time of publishing, and many of the concepts introduced are still relevant in the field. Their assumption of predictable decline refers to the longer-term degradation of elderly ability, and influenced future works by indicating that particular ADLs could be more indicative of morbidity than others. Specific ADLs becoming more difficult for a person could serve as an indication of a decline in overall activity performance. While the definite sequence of procedural loss Katz pioneered is unlikely, important ADLs becoming difficult for a resident can be indicative of degenerative medical conditions.

Accuracy in behavioural modelling, and predicting physiological and mental states, improves when activities are assigned relative importance (Wilson and Atkeson 2005). This means that specific activities may be more important in behavioural modelling than others, which can be indicated in the behavioural model. Some activities have a shared weight between all individuals, while others vary in importance per person due to medical conditions, habits, and routines. Weighting activities due to their importance is key to identifying a person’s ability to live independently, reflecting their behaviour and potential actions. Weighting activities using Katz’s concepts can benefit a model’s capability to accurately predict ADL performance loss and full loss of ADLs.

Predictable behaviour in the form of ADLs may be captured as a series of sensor outputs, and so changes in this interpreted behaviour could be indicative of increased fall risk. In sensor data analytics and activity recognition, ADLs can be represented as the abstracted knowledge from sensor data with context, which can be modelled to identify patterns of behaviour.

2.1.2 Anomalous behaviour

Anomalous behaviour can be identified by first establishing regular behavioural patterns, mapping ADLs and other day to day activities to a model. In most in-home monitoring of the elderly, anomalous behaviour can be identified as risky, in that the accepted routine of events is in some way disturbed. This can be from missing a step, taking a significantly longer time to perform an ADL, a change in the pattern of an ADL, or even losing important ADLs completely. It is important to monitor variances from behavioural patterns, as this can indicate the user is at risk of a fall or other dangerous event, or that the established patterns may be inaccurate.

Aberrant behaviour was defined by Juarez et al. (2015) as a high number of movements between two locations, exceeding a relative threshold. While many studies will simplify distinguishable anomalous behaviours under one label, these behaviours were categorised to create a more effective model. Fall, fatigue, and general aberrant behaviour scenarios were independently identified using distinct expressions. Being able to distinguish fatigue from other degenerative conditions in behaviour can prove the difference between preventing a fall and having to detect it, as it can be indicative of dementia under specific contextual conditions. In this study, fatigue is detected when the intensity of all or some observed behaviour decreases over a short period of time, usually when the subject is moving between rooms of the house. Falls are evident when a long period of inactivity is detected from the resident, with no motion detected. In cases where a fall occurs, the trailing motion between two sensors being activated simultaneously also indicates a fall. This can occur differently in practice and may also be displayed as a long period of inactivity with no motion before a fast activation of more than one sensor indicating movement, referred to as a two-stage fall scenario.

Data visualisation can be important in identifying anomalies and deviations. For example, Ogawa et al. (2002) monitored elderly residents in the home, and produced histograms for each sensor being used, rather than pooling the data together. This allowed the data to be effectively visualised, while also providing an abstract representation. Their study mainly covered the monitoring of user behaviour for health reasons, rather than producing a model or for any predictions. In one example, they use a histogram to visualise the cooker use by one subject over the course of a year. With this information, they were able to identify the regular timings at which the cooker was used. Through visualising data in this scale, observable anomalous performance in ADLs is identified.

Some residents may leave and re-enter the same room multiple times within the space of a few minutes, which can be attributed to mental condition, displaying symptoms similar to that of dementia (Juarez et al. 2015). Suryadevara and Mukhopadhyay (2012) designed an in-home monitoring system to predict future behaviours of residents. In doing so, they classified actual behaviour as either normal, or irregular based on a number of key factors. Primarily, irregular behaviour could be characterised where the duration was outside an allowable duration range. Time-based anomaly detection appears fairly simple, however the effective modelling and identification of irregular behaviour supports accurate prediction for the time frames of normal events, as well as irregular ones. Measuring the length of events, and the time interval between events is the an effective way of approximating a user’s actions as they move around the house, allowing profiles of behaviour to be constructed and compared.

2.2 Physiological conditions preceding a fall

Physiological monitoring offers an alternative insight into user risk which is not identifiable through behaviour alone. Physiological expressions tend to require less context in interpretation when compared to behavioural data. For instance, if a resident has just used the oven, it can be assumed they will sit down shortly. However, instead they may use the fridge and leave the room. Both events could be equally likely. Instead, a resident entering a room with a high heart rate is likely to sit down shortly. This is a generalisation, but in essence physiological factors can offer additional context to activities with respect to prediction. Typically, heart rate, galvanic skin response, and parameters of movement, such as cadence and gait, have been found to be the best indicators of morbidity and fall risk (Vestergaard et al. 2009; Stone and Skubic 2014; De Santos Sierra et al. 2011).

Laboratory testing is the best practice for measuring physiological factors and medical conditions with clinical accuracy, with follow-up appointments improving accuracy and the available models. Long term studies performed in a home environment over several years provide useful information on morbidity, especially elderly patients for whom many underlying conditions may affect lifespan (Vestergaard et al. 2009). A relationship was found to exist between the performance in a 400-meter walk test and the subsequent mortality in older adults in Vestergaard’s paper. They found that time taken to complete the walk, and the difference between subsequent lap times, were both significant indicators of mortality. These indications should also be supplemented with associated physiological factors, to improve accuracy and provide context. This test is usually performed in laboratory or hospital conditions, in which a physician would also be able to consider the patient’s condition and other metrics from the test. This includes but is not limited to, whether or not a break was taken, variation in lap time, and existing health conditions. Therefore, long term morbidity generalisations may be observable by monitoring the deterioration of time taken to walk regular distances.

Mental status can also indicate fatigue, instability and other fall-related conditions. De Santos Sierra et al. (2011) attempted to use physiological features to estimate the mental state of a subject with as little intrusive measurement as possible, to prevent any inconsistencies. They selected heart rate, and galvanic skin response to identify stress levels in subjects. This was performed in real-time, and provided accurate insight into a subject’s mental state with little invasive measurement.

Gait and other parameters of movement can indicate fall risk individually, but supplementing this information with physiological data can also assist in determining the risk of falls. In a study performed with heart rate monitors and foot pressure sensors, a relationship between maximum heart rate and the spread of pressure exerted by the feet was theorised. Studies have shown that two conditions related to extreme heart rates were linked to fall risk, Bradycardia and Tachycardia for slow and fast heart rates respectively (Albu-Shamah and Zhan 2012). By monitoring heart rate while also measuring the spread and intensity of foot pressure, it was noted that a relationship must exist between the pressure applied on the forefoot area of either foot and escalating heart rate. A range of behaviours can be predicted by observing these changes. For instance, pressure on the ball of either or both feet while experiencing a 15–20 bpm increase in heart rate can indicate the user is about to stand. By combining data on heart rate and foot pressure, a sharp change in heart rate as foot pressure increased could be linked to Bradycardia or Tachycardia, indicating increased fall risk.

3 Sensor implementations

The ability to interpret a subject’s behaviour relies on accurate monitoring with appropriate sensors. Most common sensors can be categorised as either passive or active. Passive sensors are embedded in one area and capture data as they are invoked, remaining mostly unintrusive. Simple passive sensors are effective in behavioural modelling when many are used, due to their predictive potential when analysing sequences of sensor activations (Wilson and Atkeson 2005). They struggle to support accurate behavioural modelling when few sensors are used, as the granularity of the data one sensor can collect is low. Temporal location can be approximated by interpreting sequences of simple sensor activations as movement, and accuracy can improve with the number of sensors used (Tapia et al. 2004). More complex passive sensor technologies, such as WiFi and BLE, can be used to perform more accurate location tracking with fewer sensors (Scanaill et al. 2006). In contrast, active sensors measure data directly from the user by using more intrusive configurations such as wearable or vision-based sensor technologies. Fewer active sensors are required than passive in similar environments because active sensors can capture more complex information with higher granularity. However, sensors using these technologies can be more expensive than simple sensors. A subject may have more objections to active sensors for many reasons, such as discomfort and privacy concerns. Wearable and vision-based sensors are useful for monitoring physiological expressions such as heart rate and movement, however monitoring a subject using these sensors can prove more intrusive or require fixed conditions.

3.1 Passive in-house sensors

Passive in-house sensors refer to those which are physically installed inside smart home installations with no intervention or consideration expected of the subjects. These sensors are typically designed to monitor the results of user interactions around the home, such as identifying the opening and closing of doors, presence or movement in a room, and the usage of individual appliances. While wearable sensors are often used to perform approximate activity recognition, passive in-house sensors can be utilised for full behavioural monitoring (Ogawa et al. 2002). Binary passive in-house sensors allow us to interpret behaviour and actions as sequences of sensor activations. They can also be used to indicate approximate location by interpreting temporal context of sensor activations, with varying accuracy. A binary sensor can utilise many sensor technologies, however it only outputs a boolean value to indicate activation. Passive implementations of sensor technologies include reed switch sensors, IR motion detectors, and the more recent advances in WiFi and BLE positioning methods. A combination of passive sensor technologies can be used to develop a smart home environment. Figure 1 shows a sensor configuration making use of motion and location sensors, such as IR and WiFi, pressure switches, and other passive sensor options. Sensor outputs are stored as a time series with the parallel tracks shown in Sensor Data. Activities can then be extracted from sequences in the data, which can then be compounded to form a behavioural profile.

Simple sensors used in studies performing in-home monitoring range from magnetic switches, to IR movement sensors, and many others. Switches can be used to indicate the opening and closing of a door, drawer, cupboard, or to indicate general activity near its location. Ogawa et al. (2002) made use of IR sensors, as well as magnetic switches, watt meters, and interrupt sensors to monitor behaviour. By monitoring resident behaviour over two years with these sensors, sensor usage was graphed to allow assumptions to be drawn on regular behaviours such as worship or gardening. This study used only 16 sensors in one house and monitored behaviour of the resident for visualisation. Other studies in this field have made use of more sensors in a more dense configuration. This means the dataset will contain more sequences of activations for each action performed. One of MIT’s House_n studies made use of 77 and 84 sensors in two houses with 6 and 7 rooms respectively (Tapia et al. 2004). Tapia’s work made use of binary and multi-class Naive Bayes classifiers to model sensor data, achieving satisfactory activity recognition. This study was performed under the principle that the accuracy of activities and behaviours interpreted from the dataset will rely on 3 factors: the number of sensors used in the configuration, the context provided in respect to the area represented by the sensor, and the density of the dataset. While little can be inferred from a single switch being used, a dense configuration allows behaviour to be mapped and modelled around sequential series of sensor activations. Aipperspach et al. (2010) performed a similar study making use of open datasets from the Georgia Tech Aware Home project and a MIT House_n study. A language-as-action approach to modelling sensor data produced unsatisfactory results with the Georgia Tech dataset, as only 16 sensors were used in the environment. Their results improved when making use of MIT’s data, as the increased density in the sensor configuration allowed for natural sequences to be observed and modelled from the dataset. Notably, Aipperspach found that the structure and design of the studies and datasets used could potentially scale to support 1000 tracked sensors while maintaining its 100Hz polling rate, providing further validation of Tapia’s relationship between the quality of inferred information, and the number of sensors used.

Lab-based assessments are the current standard for gait measurement. In attempting to obtain similar results to a lab-based gait velocity test, Rana et al. (2016) marked walkways between rooms with IR motion sensors at either end. These sensors are activated when movement or an obstruction is detected. They found that monitoring the “transition times”, the time spanning activations of sequential sensors, when moving between rooms in the house allowed them to relate the validated gait velocity of a subject and their transition times between rooms in the house. While this study initially required the “clinical gait velocity” which was accurately measured in a lab-based environment, it provided a cheap solution to a previously invasive test procedure. Gait velocity is indicative of morbidity, fall risk, and other health conditions, and so performing this test over a long term without regular clinical follow up exams can promote earlier identification of performance degradation.

Simple sensors can be used in tandem to perform basic activity recognition, however a potentially more effective solution to activity monitoring and modelling is accurate positional tracking. This would provide for accurate positioning throughout a tracked space so that location would no longer be approximate, as it is when using simple sensors. With precise positional tracking, areas of a room can become indicative of separate activity function. Positional tracking can be handled with a range of technologies, with WiFi and BLE among the most popular. These two technologies are common in smartphones, wireless routers, iBeacons and many other similar devices. Fingerprinting, one method of positioning, is usually performed with a smartphone or transceiver by tracking a device’s signal identity information, such as a MAC address. The fingerprint is then combined with the Received Signal Strength (RSS) to produce a traceable vector. In most cases, WiFi and BLE configurations require fewer sensors than would be required to achieve the same granularity using simple sensor technology. Making use of a smartphone or other tracking beacon is conventional, however this can be considered invasive, especially where subjects may not carry a smartphone at all times. Device-free localisation (DFL) methods were explored by Hillyard et al. (2015), achieving tracking on a resident with just 0.25 m average error in a 6m x 5m room. The “in-motion” design of the twinned receivers stems from the suspension of a platform on which one receiver is placed. A newer DFL technology was designed for localising moving objects by using WiFi signals (Adib and Katabi 2013). This technology has been further developed into “Emerald” (Massachusetts Institute of Technology 2015), which is designed to detect falls in the elderly wirelessly throughout a home using just one base station. These studies demonstrate the capabilities of WiFi and BLE positioning, which could have potential to be used for modelling movement and behaviour.

While combining the capabilities of both technologies produces the best results, there are significant differences between them which may make one easier to implement than the other. Primarily, size and power requirements for each technology differ greatly. BLE requires significantly less power than WiFi, and far smaller sensor modules in comparison to a WiFi router. Typically, BLE modules are battery powered for instance while WiFi routers are usually stationary and require a wall outlet for power. Due to its lower power usage, BLE also has a comparatively low range on the 2.4 GHz spectrum compared to that of WiFi at 2.4 GHz/5 GHz (Baniukevic et al. 2013). Also, when producing an unintrusive system, BLE RSS signals, for fingerprinting, can be retrieved from a large spectrum of devices, whereas Apple iOS devices currently do not expose WiFi RSS signals (Zhuang et al. 2016). Ensuring a positional monitoring system works passively with the largest range of devices that residents already own could reduce intrusiveness.

3.2 Vision-based sensors

Vision presents a computationally difficult challenge in monitoring, as even low-resolution video requires significant preprocessing due to the high concentration of information. Many existing video monitoring solutions cannot make use of all of the available information in real time, so specific aspects are usually selected. While computer vision systems are capable of identifying objects within a video, they are currently unable to process all aspects of video footage in real time (Stone and Skubic 2013). Depth sensor solutions, a subsection of computer vision, can be likened to WiFi and BLE positional monitoring systems. However, depth-based sensors can capture more detailed data, with additional contextual cues which cannot be captured with subject positioning alone. These sensors are the subject of privacy concerns making it difficult in some cases to source test subjects for use with in-home installations. Residents display aversion to having constant visual surveillance despite most sensors only performing IR-based motion capture, rather than recording high resolution video.

Two main motion capture solutions exist in the majority of studies performed within the last decade: Vicon’s motion capture solutions such as ViconMX (Jiang et al. 2011); and Microsoft’s Kinect sensors (Stone and Skubic 2013). Vicon’s motion capture solutions are highly praised for their accuracy. For example, Ismail et al. (2017) detailed a lab-based monitoring solution with a Vicon motion sensor to validate their in-home Kinect sensor monitoring data. Vicon’s sensor data is considered to be accurate enough in monitoring joint movements with medically confirmed accuracy, and is widely used in lab-based monitoring assessments (Zhang et al. 2007). Kinect’s main selling point is its comparatively low price point, compared to Vicon’s enterprise pricing. The Kinect sensor is an affordable motion capture solution, however some aspects of captured data suffer in accuracy when compared to that of a Vicon solution. For instance, it is noted that when observing knee flexion and extension data on a model where the knee was fully extended, Kinect would often identify the knee as being hyper extended instead. Pfister et al. (2014) found that Kinect’s measured peak flexion angles were consistently lower than that of Vicon’s solution. While Kinect produces data with lower accuracy than a Vicon motion capture solution, Kinect’s basic accuracy still allows it to be used for rudimentary monitoring at a considerably lower price point than any of Vicon’s options. A newer version of the Kinect sensor was released in 2013, capable of tracking up to 25 individual joints including thumbs, as well as monitoring user facial expression and heart rate (Demerjian 2013). This provides new prospects for studies aiming to combine physiological and gait-based data to determine the risk of falling. Microsoft’s most recent iteration of the Kinect sensor was discontinued in 2017 (Wilson 2017).

Motion approximation is currently the main application of stereo camera implementations. Two camera sensors are used to apply digital skeletons to a moving subject to approximate aspects of movement such as posture, walking speed, and gait (Alazrai et al. 2015). These sensors use low resolution RGB images and depth maps, captured at 30 frames per second, to identify the 3D coordinates of many skeletal joints on a person. Phillips et al. (2017) used Kinect to identify gait parameters and potential falls. Their results showed that a 5.1cm/s decrease in gait speed in the home was associated with a high probability of a fall occurring within the next 3 weeks.

Gait has been identified as a key indicator for falls prediction, and determining fall risk. It has been defined as “the external performance of the human body structure, function, exercise conditioning system... when humans walk” (Jiang et al. 2011). When monitoring gait, a person’s stability and symmetry of gait indicate consistency of safe movement. While no unified model of human movement currently exists, some elements of mobility have been modelled. Zhang et al. (2007) proposed a Fourteen-Linkage model for Walking (FLW) using a kinematic approach to gait measurement. In production of the model, a Vicon motion capture system and wearable 3-axis accelerometer were used in a configuration allowing for 30 separate points to be monitored around the body. Many researchers have mapped similar data onto a 2D stick figure model separated into a variable number of segments. This study used a 14 segment model with 12 joints between adjacent segments. Joints in the model were monitored to identify position, segment angle, joint angle, as well as their velocity/acceleration, and angular velocity/acceleration. Stability of footprints and the walking cycle can then be modelled. The model provides an opportunity to measure gait with clinical accuracy, outside of a laboratory environment.

3.3 Wearable sensors

Wearable sensors can accurately monitor movement, cadence, heart rate, and other features due to direct access to a user’s physiological expressions. More recently popularised by fitness trackers, wearable devices have been used for many years now as a relatively reliable source of movement and vital data. However their use requires additional considerations such as battery life, size, and user comfort. As wearable sensors are usually mobile when used in an in-home environment, power usage is a concern for these devices. Accelerometers and gyroscopes are relatively low-cost sensors, however the systems required to process the data they produce typically require constant CPU time for monitoring and analysing data. In wearable devices, power requirements can vastly reduce a sensor’s effectiveness. Giving a small device a large battery can make it bulky, ensuring devices which are already intrusive become more so. Conversely, a smaller battery will ensure the device requires frequent charging which will require the user to recharge it often (Alazrai et al. 2015). Size is also a concern when considering user acceptance. Some elderly subjects dislike wearable devices due to size or weight, finding them intrusive and cumbersome. While wearable sensors can produce impressive lab-based measurements, they are not a long term solution for non-intrusive studies.

Wearable devices offer the opportunity to monitor several physiological features, however many potential subjects object to them. Users may object to wearing the sensors for privacy reasons, as continual monitoring of personal health/vital information can be considered invasive. In contrast to other sensor configurations, wearable sensors typically require the user to mount the sensor on the correct point, such as the wrist or chest. While most users would prefer to simply wear a wrist mounted sensor, more accurate heart rate measurements can be taken through straps on a user’s chest. Wrist mounted heart rate sensors work via reflection photoplethysmography. Heart rate is measured by measuring the fluctuations in light level responses to coloured light. However researchers argue about this method’s accuracy in comparison to EKG monitoring since its measurements are based on pulse timing (Lee et al. 2013).

Placement of a sensor usually also indicates the underlying technology used. Sensors can be embedded in clothing to improve potential uptake in wearable technology, making them more tenable. For instance, pressure sensors have been embedded in shoes using sensitive insoles, to identify foot pressure and locations of peak force (Albu-Shamah and Zhan 2012). Banuleasa et al. (2016) showed that embedded EKG sensors in clothing can provide valuable information on the user’s respiration. By weaving conductive yarn through clothing, a functional antenna can be formed which changes shape with the regular flexion and compression associated with respiration. The changes in the antenna’s shape affected the signal on the receiving end, allowing breathing to be effectively monitored. This approach allows for effective monitoring of physiological indicators with a minimally intrusive sensor design, which can give clear indication of physical capability.

Fig. 2
figure 2

A passive monitoring system for time-series-based behavioural interpretation

Full size image

Gyroscopes and Accelerometers provide direct, corresponding movement indication with clear rotation, pan, and tilt axes. Many studies make use of a combination of these sensors (Albu-Shamah and Zhan 2012; Scanaill et al. 2006). For instance, in a study performed by Yang et al. (2013), a two-axis inertial accelerometer with gyroscope was used to detect falls using a threshold technique to determine the direction and force of a fall. This sensor was mounted on the subject’s head, which allowed it to identify forward, backward, and side-facing falls in real time with constant reporting via BLE. Studies such as this are not uncommon, with fall identification being widely studied in the field (Jiang et al. 2011; Sucerquia et al. 2017). Krenzel et al. (2012) shows a method for detecting the occurrence of falls via accelerometer and gyroscopic measurements. Instabilities in a user’s movement can be identified, indicating the proximity of an upcoming fall. The graphs produced from sensor data show the erratic differences in amplitude between data collected from that of a person walking in a stable fashion, to that of a person walking with instability. Accelerometer data can be used for fall prediction and detection, however it is also useful in activity recognition. Figure 2 shows a wearable solution used in the selfBACK project (Sani et al. 2016), making use of accelerometer data for activity recognition. Regular features are expressed and patterns in the data are identified. The intensity and regularity of activities can then be measured and modelled to produce an profile of a user’s behaviour for day-to-day profile comparison (Fig. 3).

Fig. 3
figure 3

A wearable system for constant activity profile generation, using motion measured with an accelerometer and gyroscope

Full size image

3.4 Summary

In this section, several sensor modalities have been identified. From the works referenced, it is clear that each sensor displays different capabilities. Table 1 shows these capabilities as they relate to the behavioural expressions listed in Fig. 1.

Table 1 Listing of sensor modalities and their capabilities
Full size table

4 Fall risk determination

The task of predicting falls differs depending on the timespan being considered. The aim may vary from detecting an imminent fall in the next few seconds to identifying increased risk of a fall over the next few months. Studies measuring physiological expressions and movement tend to be effective in detection and prediction, while studies assessing underlying conditions related to fall risk and extended gait parameters are more suited to long term modelling and determining the risk of falling. As falls are abnormal, training data can be difficult to acquire. Khan and Hoey (2017) highlighted this lack of data and suggested methodologies for considering the problem from a data availability perspective.

4.1 Fall detection

Fall detection has been extensively studied with many accurate methods using passive, wearable and vision-based sensors. Motion capture devices offer unique pose detection and monitoring tools which can identify sudden and intense relaxation, tension and loss of control (Stone and Skubic 2014). A long term study performed over 9 years made use of a dataset consisting of 445 simulated falls, and 9 authentic, naturally occurring falls, monitored using a Kinect depth camera device. Primarily, issues occurred in detection when the monitored subject was further from the camera, reducing the functional resolution of the subject. Camera-based solutions also suffer false positives stemming from occlusion, where the stream is interrupted by objects or other bodies obscuring the device’s view of the subject.

4.2 Fall prediction timescales

Fall prediction aims to identify an imminent fall based on a continuous stream of input. This may offer the opportunity for timely intervention to reduce the negative consequences of a fall. Little research has been done on identifying potential interventions given the short time frame involved. Yang et al. (2013) proposed an airbag-based intervention to be deployed in tandem with their accelerometer-based falls prediction solution. Their study made use of an IMU-package mounted on a subject’s chest, which contained two triple-axis accelerometers and gyroscopes. This unit was coupled with a neural network trained on a set of scripted falls. Maintaining a high accuracy in its predictions, Yang demonstrated a system which was capable of predicting falls “one-walking-step-ahead” of incidence. The proposed air bag solution would be deployed over a subject’s head, trunk and hip in a manner similar to that of automotive protection, preventing impact and damage where possible.

Methods of determining the risk of falls tend to identify a longer period of time over which a fall is likely to occur. When forecasting increased risk of falling several days, weeks, or months in advance, interventions can be considered more intrusive since the time frame of a fall cannot be precisely indicated. During a 5 year study by Phillips et al. (2017) where motion capture information was taken using an embedded Kinect device in a home environment, a model was derived which estimated that a 5.1 cm/s decrease in gait velocity was associated with an increased risk of a fall within the next 3 weeks. Larger datasets are best suited to longer term prediction as underlying fall risk characteristics can more effectively be derived. However while additional features characteristic of fall risk were expected to become visible during the study, gait velocity remained the primary fall risk indicator between short and long term fall prediction. Cuaya et al. (2013) made use of multiple gait aspects rather than being limited solely to gait velocity. Gait parameters were selected by a domain expert in one method and by a feature selection algorithm in another. While the experts selected length characteristics such as left/right step length, left/right stride length and base of support step, the network selected time and angle based characteristics. The resultant neural networks based on these models displayed similar, and in some cases, equal accuracy in predicting falls. Competing work between domain experts and algorithms for characteristic detection could contribute to further research in neural networking analysis of gait.

4.3 Patterns in movement

Parameters of gait can be monitored to identify patterns in movement as well as variances which may indicate deterioration or other conditions. Gait velocity is widely used to indicate fall risk in elderly patients (Verghese 2009; Montero-Odasso and et al. 2005). Using machine learning techniques, statistical links between gait and a range of conditions, spanning from fall risk to dementia and death, can be modelled with increasing accuracy (Yang et al. 2013; Ismail et al. 2017). Fritz and Lusardi (2009) went as far as to label walking speed as the “sixth vital sign” and noted its potential in predicting the length of hospital stays, medical costs and general health status. Gait velocity is the most widely observed parameter of movement. Balance, cadence, accelerometer-based motion patterns and postural sway can also be observed, however most models which successfully predict falls appear to give importance to gait velocity by design. Behaviour-based patterns in movement have also been recently explored, such as path tortuosity (Kearns et al. 2012). As less intrusive sensor equipment for continual positional monitoring becomes available, it is expected that more research will be performed in this area (Adib and Katabi 2013).

Measuring the speed of walking, or gait velocity, of a subject has been shown to be effective in identifying many potential conditions, such as dementia (Beauchet et al. 2008). Decreased walking speed was found to be related to reduced performance in the Trail Making test A and B (Quach et al. 2011). This test assesses the performance of subjects in object avoidance situations, and links a decrease in walking speed as subjects are required to further engage their attention. Furthermore, a key link was found between the gait velocity of a subject while maintaining conversation which can be used to monitor progression from Mild Cognitive Impairment (MCI) to Dementia (Montero-Odasso and Sarquis-Adamson 2017). In predicting falls, Quach focussed on the relationship between gait speed and falls which was found to follow a U curve with slower walkers being more likely to fall indoors, and more likely to fall in general. Faster walkers were more likely to fall outdoors. The gait velocity of elderly subjects living in a community was monitored, along with other physiological indicators. They found that only decreases in a subject’s gait speed greater than 0.15m/s, over an 18 month period, were predictive of falls.

Another lab-based assessment, the One-Leg Standing Test (OLST), aims to assess the subject’s balance by measuring the length of time a subject can support themselves on one foot (Michikawa et al. 2009). Unlike other assessments previously mentioned, OLST has no link to mortality, however it can be linked to relative frailty and fall risk in elderly patients. Some researchers argue that current research relating fall risk to OLST times can be inconsistent. However Ayena et al. (2016) designed a methodology for performing the OLST in a home environment which accurately evaluated fall risk in a range of patients, including healthy, elderly, and those with physical impairments and conditions.

Gait assessment in a laboratory will produce the most accurate indication of a person’s optimal gait. In contrast, in-home monitoring identifies natural gait over a longer period of time. This provides a more realistic/regular approach to mobility measurement, as a person is more likely to walk with a natural gait because they are in their home setting (Jiang et al. 2011). There is currently no way to precisely determine if a person’s exhibited gait is normal, however by passively taking measurements in a home environment over a long period of time relatively accurate results can be achieved. Mounting a device on a subject allows motion, acceleration, and intensity to be modelled to detect falls with relative ease. Most of the publicly available data on falls detection has been performed with younger subjects due to the dangers associated with simulating falls in elderly subjects, even in a controlled environment. Sucerquia et al. (2017) explored the lack of fall data for elderly subjects and was able to collect wearable measurements from an elderly woman who was experienced in martial arts and capable of performing safe falls. Their methodology had a device containing two accelerometers and a gyroscope mounted on subject’s waist, which provided high distinction among activities. Throughout their testing, it was determined that simulated fall data was easier to obtain than authentic falls in a home environment but may incur some bias on the overall dataset. It is also noted that fall prediction in elderly subjects proved more difficult to model than in younger subjects, which suggests the model and dataset may not be directly transferable.

Most stride measurement-based assessments are performed in hospital or laboratory scenarios which is costly and invasive towards subjects. Ideally a passive monitoring solution capable of identifying stride length and gait passively in a home environment could be implemented, ensuring a resident’s regular movement metrics are effectively measured. Phillips et al. (2017) performed a study to identify whether a similar embedded system could be implemented. They found that constant monitoring with an embedded Kinect sensor was not only able to accurately measure these movement parameters, but was able to adjust these measurements over time. In a similar work performed using a Vicon stereo camera, Ismail et al. (2017) found a relationship between postural sway and the likelihood of an upcoming fall. Due to privacy concerns around vision-based sensors, IR motion sensors may present an effective alternative. It was demonstrated that off-the-shelf IR motion detection devices could be used to monitor the transfer time between sequential sensors and estimate the gait velocity of a person (Rana et al. 2016). In practice, one study produced comparable results to that of a laboratory assessment and could potentially be used to correlate any previous measurements.

Verghese (2009) used a GAITRite pressure sensitive walkway to monitor quantitative markers of gait, including velocity, cadence, stride length, double support, stride length variability, and swing time variability. A statistical link between decreased gait speed and increased fall risk was noted. However, other gait markers also proved to be indicative of fall risk. Stride length variability provided the best fit in final models, even after other physiological factors had been considered. Gait velocity remains a strong indicator and tends to be the easiest to measure with the least specialist equipment.

Cuaya et al. (2013) compared two dynamic Bayesian network structures, one constructed from gait features determined by experts in the domain, while the other used parameters selected using a feature selection algorithm. In cases where predictions were made on the indication of an imminent fall, both networks performed with equal accuracy. However, in cases where predictions were made on whether a fall would occur within the next 6 months, the expert network showed a 5.56% improvement in accuracy. One conclusion to draw from this is that the dataset did not exhibit the same features as the experts had previously observed outside of the study. Since the computational model was designed to observe features expressed in the dataset, it was at a disadvantage, however it still produced strong results.

4.4 Applying behavioural models

Monitoring gait and other motion parameters provides excellent insight into the movement of a subject, their subsequent falls risk, and even morbidity. The non-discrete sensors required for measuring these expressions can prove more financially and computationally expensive than simple sensor implementations. Many studies have linked behaviour and activity recognition with binary sensor outputs using a variety of machine learning techniques, such as Dynamic Bayesian Networks, Case Based Reasoning (CBR), and Hidden Markov Models (Tapia et al. 2004; Zhou et al. 2011; Suryadevara and Mukhopadhyay 2012). Observations from these studies have been used to make assessments on subject behaviour, stability of motion and observable conditions.

There has been only limited research on the applications of behavioural modelling and activity recognition for predicting falls, and identifying risk of falling, using simple sensors. Parameters of gait, movement, and heart rate can strongly indicate fall risk, however these factors can currently only be measured accurately using active sensors in laboratory environments (Vestergaard et al. 2009). A less intrusive approach would be monitoring residents in a home environment with passive sensors, allowing for authentic behaviour to be observed and modelled with accuracy. In these conditions, gait velocity and activity performance can be assumed natural and most indicative of regular behaviour, and fall risk should be inferable.

The overall goal in performing falls prediction studies is to minimise risk of falls and prevent known falls where possible. Intervention is important in this field, as residents at risk of falling may be incapable of helping themselves or preventing falls. This means that risk must be effectively communicated to appropriate contacts so interventions can be made where required.

Interventions require unique and relevant care for each instance of fall risk. Some popular machine learning methods are effective at deriving observations in datasets, but their workings are often difficult to interpret. For instance, reading the weights and activation functions in a neural network are likely to be difficult to abstract into a rule set. In CBR the reasoning employed in its decision making is clear and well-demonstrated (Zhou et al. 2011). This functionality allows CBR-based monitoring systems to provide relevant explanation by example, along with expected interventions which have been tailored for each resident, ensuring carers and residents are far more likely to receive relevant care in their home environment with minimal intrusion.

5 Conclusions

This review identifies and assesses the existing approaches to the use of in-house sensors for monitoring residents’ on-going health. From the literature that has been considered, it was noted that behavioural modelling can be performed by identifying sequences of outputs in sensor data, and by highlighting changes through the weighting of relatively important abstracted activities. Advances in data visualisation and machine learning techniques have improved identification of risky and unusual behaviour, which can be identified and monitored to correlate with diagnoses of degenerative medical conditions.

The studies in this review demonstrate that behaviour can be successfully modelled using passive in-home sensors, while wearable and vision-based sensors are shown to be effective in capturing motion in a moving subject with clinical accuracy. The recent advances in positional tracking using WiFi and BLE signals could allow for more granular information to be made available for activity monitoring. There is limited research and very few studies that explore long-term monitoring and modelling of behaviour using primarily simple in-home sensors, with the aim of identifying the risk of falls in residents. The general lack of focus on detecting the risk of falls is particularly disappointing when the potential to provide safe and timely intervention is considered. With focussed research, a product could potentially be brought to market to monitor risk of falling in a home environment, and contact care providers in cases of increased risk.