1 Introduction

One in three adults aged 65 and older fall each year [1]. Older adults are hospitalized for fall-related injuries five times more often than they are for injuries from other causes [2]. With the population aging, the number of falls and the related costs will increase. Falls are more frequent in patients with advanced Parkinson’s disease (PD) and they occur at even earlier stages and more frequently in patients with atypical parkinsonian disorders [3–5]. About half of PD patients who fall will require medical care for fall-related injuries, and many never recover to their pre-fall motor and independence baseline [4, 6].

Current medical practice for diagnosing falls relies on expensive hospital admissions to determine if cardiologic, blood pressure, balance, gait, or seizure disturbances caused a fall. Patients are connected for short periods to bulky, single-function devices that can provide only limited diagnostic information as this information is confined to the hospital setting after a fall has occurred. Currently, advanced technologies may allow using inexpensive and wearable multisensor devices on outpatients to determine the causes of their near-falls and falls as well as collect other critical diagnostic information in a daily life setting.

Only recently has technology evolved to allow scientists to continuously record multiple data streams from the body in everyday life in a comfortable and unobtrusive way. This technology has been used for real-world applications such as stress monitoring [7], as well as gait and vital sign assessment and fall detection (see review [8]). The Wearable Multimodal Monitoring System (WMMS) improves upon previous technology by integrating and synchronizing multiple data streams in real-time while also recording valuable patient feedback via a smartphone. The system is an extension of on-going work to build a Multi-Aspect Real-world Integrated Neuroimaging (MARIN) system to study stress [9, 10]. The WMMS takes advantage of the original MARIN system architecture by utilizing some of the same devices, and also includes new devices and mobile applications specifically engineered for the study of falls in movement disorder patients. Our multimodal monitoring system is aimed to diagnose the causes of falls and near-falls so appropriate treatments can be undertaken to prevent subsequent occurrences.

2 The Wearable Multimodal Monitoring System (WMMS)

The Wearable Multimodal Monitoring System (WMMS) is a highly integrated system designed for real-life long-term monitoring of patients susceptible to falls. It uses several state-of-the-art microelectronics and communication technologies in a mobile, wireless data collection and computing platform with multiple, wearable biosensors that simultaneously and continuously monitor cardiovascular, autonomic, motor, and neurological activity in the daily life environment. We chose these modalities as they can capture the most common causes of intrinsic falls unrelated to accidents. In addition, the system requests and receives critical patient feedback about symptoms and other outcome measures. It is envisioned that the data collected by this system will be suitable for the creation of algorithms that can go beyond diagnosis, to the prediction of falls. These algorithms could then be implemented in next-generation systems to alert the patient when conditions and behaviors exist that increase their risk of falling.

2.1 Components

Peripheral Monitoring Devices. The WMMS contains five commercially available devices (Fig. 1). (1) The Zephyr Bioharness 3 is a chest band that is capable of monitoring heart rate, EKG/R-R intervals, respiration rate, posture, and 3-axis accelerations. The data will allow us to determine whether abnormal heart rhythms or heart ischemia cause patient symptoms (lightheadedness or syncope) and/or falls. (2) The Empatica E3 is a small wrist-worn device containing a 3-axis accelerometer and optical temperature sensor, as well as an electrodermal activity sensor and a photopletismography sensor, which measure physiological arousal that will be used to determine autonomic disturbances. (3) The MINDO 4-channel wireless EEG Headset is a 4-channel wireless EEG monitoring system equipped with dry electrodes to monitor syncope-related decrease or seizure-related increase in brain activity. The headset can provide up to 256 samples per second from each EEG channel. (4) The BodyDyn 10-DOF Wireless Body Motion and Posture Monitor is the prototype of a wearable body motion and posture monitoring system that will be used to determine if gait disturbances are the possible cause of a fall or near-fall. It will be also used to determine whether motor disturbances such as tremors, dyskinesias, dystonia and freezing are possible causes of falls and near-falls. Each device can provide up to 100 samples per second of 3D linear accelerations data points, 3D angular acceleration data points, 3D magnetic flux, and barometric pressure. These small, unobtrusive sensors will be affixed to clothing or other devices at locations including the wrists, chest, waist, back, and ankles. While the four devices outlined above stream data wirelessly to the smartphone in real-time, (5) the HealthStats BPro ABPM Watch records data locally and those data are then added to the other datastreams post hoc. The BPro is an ambulatory blood pressure monitoring (ABPM) system in the form of a wristwatch used to measure blood pressure (BP) and heart rate (HR) and to determine whether drops in BP or HR can cause falls and near-falls (orthostatic hypotension). It uses modified applanation tonometry to measure the pressure pulses detected at the radial artery in the wrist every 5 to 15 min.

Fig. 1.
figure 1

The Wearable Multimodal Monitoring System (WMMS)

Full size image

Handheld Electronic Device. An Android smartphone serves both as the data hub for the sensors and the graphic user interface for the patient. The WMMS uses several Android-based applications. The main widget provides an event monitoring panel which allows the patient to log notes for salient events and answer related questions. Additionally, the WMMS includes three interactive applications designed to assess symptom severity and a suite of inventories that gauges non-motor functions (i.e., mood).

Event Monitoring Panel. The event monitoring panel is a widget that is available to the patient when the phone is turned on. It includes six different buttons that the patient can select to log salient events (right center of Fig. 1): falls and near-falls, medication, loss of consciousness, meals, dizziness, and tremors or dyskinesias. When selecting a button, the patient is directed to answer questions about the event. Each button follows a pathway of questions designed by clinicians to capture important associated information in a uniform way. This information will be helpful in the development of fall prediction algorithms, because it provides subjective information that adds context to the continuous physiological data streams.

Applications. In order to collect data about symptoms throughout the day, we have developed three interactive apps that can be easily accessed by the patient. The first app is our mobile version of a force transducer based tapping task. The parameters replicate those of a study of Huntington’s Disease patients where outcome measures (variability of tapping intervals and tap frequency) were found to be sensitive enough to distinguish between carriers (pre-manifest) and age-matched healthy controls [11]. This task has also been performed with PD patients with success [12]. The task is performed via a touch screen and does not utilize a force transducer; however, we expect that this mobile setup will still be sensitive enough to quantify motor impairment. We will compare this new app with the Movement Disorder Society-United Parkinson Disease Rating Scale (MDS-UPDRS) tapping score. The second app developed for the WMMS is a Baseline Measurements App that guides the user visually and verbally through a series of movements taken from clinical motor scales. It then creates an output file that time stamps each movement start and end to allow for easy data analysis of the time synced multimodal data streams. The third app is an Everyday Activity App which follows the same design as the Baseline Measurement App, but it asks the user to perform everyday activities such as walking around a room and typing on a keyboard. Both of these movement apps will allow us to explore how the time-synced multimodality data tracks motor function in the clinic and at home during clinically relevant and everyday actions.

Inventory Suite. The Inventory Suite contains questionnaires validated in the literature for the study of non-motor symptoms such as mood. These inventories have been translated to an easy to use mobile format. The suite will be deployed at least once a day, but some shorter inventories will be repeated throughout the day. It contains the Stress Visual Analog Scale, Fatigue Visual Analog Scale, Self-Assessment Manikin, Pittsburgh Sleep Diary, Beck Depression Inventory, and Spielberger State Anxiety Inventory. The data collected from these questionnaires will be used to relate different perceived states with physiological data and will provide potentially important predictive information for falls.

3 Pilot Study Results

To show feasibility and to evaluate the WMMS we collected preliminary data from six healthy young controls (36 ± 9 yrs.) at the U.S. Army Research Laboratory (ARL). Participants spent three hours interacting with the software developed for the WMMS and four hours of their normal work day wearing the system. We also collected preliminary data from three patients and one age-matched healthy control participant (65.2 ± 7.6 yrs.) recruited at the UCSD Movement Disorder Center. These participants interacted with the software and received clinical evaluations over a three hour period under close supervision. Participants at both sites signed informed consent before entering into the study.

3.1 Comfort Ratings/Ease of Use

To determine ease of use we employed the well-established Visual Analog Scale (VAS) [13, 14] to assess comfort of the whole system and of individual sensors separately. The Android handheld device calculated a number from 0-100 corresponding to the location of the cursor as placed on the line by the participant. The three patients and one age-matched control at UCSD rated the WMMS at 90 ± 7 % of the perceived comfort visual analog scale (PC-VAS), and this rating did not after three hours of wear (Fig. 2). Healthy young adults at ARL rated the system initially as less comfortable (75 ± 12 % of the PC-VAS; compared to patients), but these ratings also did not change over seven hours of wear. These preliminary results indicate that the system’s high comfort level was maintained even while being worn over extended durations. All pilot users were asked if the system was comfortable upon first placement, and all users replied that the system was comfortable. Therefore, we believe that healthy adults and patients have different thresholds for comfort that is in turn reflected in different initial PC-VAS ratings. All participants also reported that the WMMS was esthetically acceptable, unobtrusive, and easy to use.

Fig. 2.
figure 2

Comfort Ratings (from the PC-VAS) at intervals during the pilot study

Full size image

3.2 Quality of WMMS Data Compared to Medical Instruments

One of the goals of the pilot study held in the clinic was to compare the quality of data collected from the WMMS to data collected from gold-standard medical instruments.

Heart Rate and Rhythm. There were no substantial differences between the EKG readings (heart rate, rhythm, and waves) from the Bioharness and the EKG machine (Illustrated in Fig. 3 for two patients).

Fig. 3.
figure 3

Comparison of simultaneously recorded EKG traces from the standard wired EKG machine and the wireless chest strap (i.e. the Bioharness) of the WMMS.

Full size image

Blood Pressure. Figure 4 shows an example of the BPs obtained from the WMMS and a standard cuff BP machine for two patients. While the BP fluctuated throughout the day, the blood pressure taken by the machine at intermittent intervals (three times for Patient 1 and twice for Patient 2) was very close to the reading given by the wristwatch device (i.e. the BPro). It is important to note that the starting value for both devices is the same because the wristwatch requires an initial calibration value from the blood pressure machine.

Fig. 4.
figure 4

Comparison of recorded blood pressure readings from the BPro wristwatch (solid lines) and the standard machine (black diamonds and circles) in the clinic for two patients.

Full size image

3.3 Capturing Relevant Events

During the time while patients were in the clinic, any relevant events were noted on the smartphone, so that those events were timelocked to the physiological data streams.

Tremor. During the pilot study in the clinic one patient displayed behavior that was identified clinically as tremor at rest typically observed in parkinsonism. The accelerometer outputs (from three devices) that corresponded to this time period can be found in Fig. 5 (right panel). Compared to normal movement without tremor (left panel), the tremor is shown by the regular oscillation in the leg and hand in contrast to normal movement patterns where acceleration increases and decreases without an oscillation pattern.

Fig. 5.
figure 5

Accelerometer data streams for one patient during normal movement with no tremor (left panel) and a resting tremor (right panel).

Full size image

Near-fall. One patient experienced two near-falls due to balance impairment, and these events were captured in the multimodal data stream. Figure 6 compares the accelerometer streams for one of the near-falls (left panel) with the same ten second time period where the patient sat down and stood back up again according to instructions in the Baseline Measurements App (right panel). The near-fall created a large but brief increase in peak acceleration values from the chest sensor (where the patient sat down abruptly to catch the fall). This brief but sharp increase is not seen when the patient sits down and stands up normally. These data demonstrate that the WMMS can distinguish between different types of actions, including distinguishing falls and near-falls from normal actions. Moreover, the lack of simultaneous or preceding EKG and blood pressure abnormalities (not shown) excluded the cardiovascular and autonomic systems as potential causes for the near-fall and pointed to postural instability as a cause of this near-fall.

Fig. 6.
figure 6

Accelerometer data streams for one patient during a near-fall to sitting (left panel) and during normal standing to sitting (right panel).

Full size image

4 Fall Prediction Algorithm Development

Successful fall detection algorithms have been developed using acceleration data collected from various parts of the body utilizing techniques such as less sensitive simple threshold based algorithms [15] or more complex and precise machine learning algorithms [16] including the use of Bayesian models [17]. While a wide literature has established that basic fall detection is a relatively simple process requiring only acceleration data from a few sites on the body, fall prediction algorithms have not yet been attempted on real-world data. Moreover, to our knowledge diagnosing the multiple causes for a fall has not been attempted. The WMMS offers an integrated view as it will utilize real-world continuous multimodal data streams collected on healthy controls and fall-prone subjects to begin to build prediction profiles for the various causes of falls. These profiles could be used for the prediction and intervention of falls in this population and the ageing population at large. We will use the recently validated definition of near-falls by Maidan et al. [18]: A stumble event or loss of balance that would result in a fall if sufficient recovery mechanisms were not activated. At least two of the following mechanisms should be activated to be determined as a near-fall: (1) unplanned movement of arms or/and legs; (2) unplanned change in stride length, (3) lowering of the center of mass, (4) unplanned change in stride velocity and (5) trunk tilt.

Once fall and near-fall events are identified via user input and/or basic fall detection algorithms, we will then utilize the rich multimodal dataset (including those data obtained through the pathways, questionnaires, and motor apps) to probe for which preceding features are predictive of falls. We will do this by applying recent groundbreaking machine learning techniques [19–23] to these multimodal data streams. Techniques include but are not limited to fuzzy logic [24] and Bayesian probability models [17]. The techniques will not only mine physiological and physical activities from multimodal sensors, but also handle context awareness, and subject specific models and personalization. Once an optimal fall prediction algorithm has been created, it will then be deployed in real-time on the smartphone and will trigger questions for the user or an emergency call to the study coordinators or 911 if questions are left unanswered or the patient reports an injury.

5 Conclusion

This paper has described a novel multimodal system designed to study falls in movement disorder patients. Overall the WMMS functioned well during the short pilot study. Patients and younger healthy adults found the system to be comfortable and easy to use throughout the study. The system provided continuous high-quality datastreams from a variety of users (from young healthy adults to older movement disorder patients). These datastreams were of high enough quality to be comparable to gold-standard medical equipment and to discriminate between events of interest and normal movement patterns. Moving forward, we aim to continue to collect data for the pilot study and to incorporate any advances in technology in order to update the system. For example, we chose the HealthStats BPro to measure blood pressure because it is the best non-cuff semi-continuous blood pressure device available on the market. However, during the pilot study we noted that the device lacked sufficient data collection frequency to accurately measure sudden blood pressure drops. New emerging technology utilizing Pulse Transit Time (PTT) [25, 26] may provide an alternative option. The WMMS employs a very flexible architecture and can be modified to incorporate new sensor technology as it becomes available. With continued improvement and data collection, we can deploy fall detection algorithms and initialize fall prediction algorithm development. We believe the WMMS holds the promise of safer and more independent living, not only for movement disorder patients but also for the ageing population at large.