1 Introduction

Parkinson’s disease (PD) is a one of the most common neurodegenerative disorders that threaten the living quality of the elderly. It is estimated that there are over 10 million people worldwide suffering from PDFootnote 1 and usually the onset happens after 60 years old. The patients suffer from the declines in motor and cognitive functions. The cardinal symptoms include rigidity, bradykinesia (slowness of movements), postural instability and tremor.

Some medications, such as L-DOPA and dopamine agonists, have been applied for managing PD symptoms and delaying disease progression. However, as PD symptoms are high variable throughout the day, the conditions of patients’ symptoms need to monitored carefully to ensure the effectiveness of appropriate dosage of medications and other necessary treatments to be applied in case that the medications become less effective. However, it is challenging to measure symptoms and treatment-related complications in advanced PD. In current clinical settings, quantification of PD symptoms is usually dependent on two aspects. Firstly, rating scales, such as the Unified Parkinson’s Disease Rating Scale (UPDRS), are employed to evaluate the severity of PD symptoms. The rating scales are derived based on the observations and judgements by clinician. Secondly, patients’ self-reporting provides complimentary information to facilitate the evaluation of the severity of PD symptoms. However, such approaches are not suitable for a continuously long-term monitoring due to the following reasons. Firstly, the derivation of the rating scales is very time-consuming [1, 2] and requires considerable clinical experience [3,4,5]. Brief clinic consultations can not provide sufficient time to clinicians to obtain a holistic picture of patients’ symptoms and progress. Secondly, the observation of clinicians and self-reporting of patients are too subjective to generate an objective evaluation outcome for the severity of PD symptoms.

Therefore, it is necessary to provide objective and convenient measurements to continuously capture the severity of PD symptoms and fluctuations. In view of this, we propose a mobile game platform to facilitate patients and clinicians to have an objective assessment of the daily conditions of PD symptoms and monitor the effectiveness of treatment. The game is implemented in the iOS environment to be installed on iPhone, which makes it easier for patients to use at any time without the need of expert supporting. In the game, we specially designed 9 mini-tasks in a farming-themed game environment. Each task is designed to align with commonly used PD tests. Furthermore, the patients’ in-game behaviors are collected and analyzed to generate reports for patients and clinicians to track the response to medications and the conditions of disease progression. The game has been launched together with a world renowned PD research centre for clinical trial to gather feedbacks and study the effectiveness.

The paper is organized as follows. Section 2 reviews related work in quantitative measurement of PD systems. The proposed mobile game framework is introduced in Sect. 3, followed by the presentation of the details with respect to the designed game and mini-tasks in Sect. 4. Some issues are discussed in Sect. 5. Section 6 concludes the paper and introduces some potential future directions.

2 Related Work

The quantitative measurement of the severity of PD systems has been studied for a long time. There are massive works exploring the quantitative measurements of motor performances for PD patients. In [3], a computer-interfaced musical keyboard is employed to study the correlation between the assessment using quantitative digitography (QDG) and UPDRS motor score. The study results substantiate the use of QDG to measure motor symptoms of PD patients.

In [6], a computerized keyboard test based on the alternative finger tapping test is developed to quantify upper limb motor function. Through analyzing the performance of users in the test, several variables are generated, including kinesia score, akinesia time, dysmetria score, and incoordination score. An assessment based on 35 patients with idiopathic PD, 12 patients with cerebellar dysfunction, and 27 normal control subjects shows that the results of the test correlate with clinical rating scales in PD and cerebellar dysfunction.

In [7], the authors design a magnetic sensing system to detect the finger movement of PD patients. The system consists of a magnetic induction coil, a sensing coil, and a circuit unit. The study conducted by the authors demonstrates the possibility of quantitatively detecting finger movement disorders in PD patients by using the designed system. In [8], the authors further propose to use log-linearized Gaussian mixture networks (LLGMNs) based on 11 indices generated through magnetic sensors for the assessment of finger tapping movements. An evaluation performed in 33 PD patients and 32 normal elderly subjects shows that the patients could be classified in terms of their impairment status with a high degree of accuracy.

In [9], a computer based at-home testing device (AHTD) is developed and the feasibility of applying the device in early-stage and unmedicated PD patients is studied. The AHTD is a computerized assessment battery and the collected data are stored on a USB memory stick and sent by Internet to a central data repository. The study results show that the device provides a feasible format for assessing PD symptoms from home.

In [10], the authors introduce a method for enabling quantitative and automatic scoring of alternating tapping performance of PD patients using a touch-pad handled computer. Twenty-four quantitative parameters are derived through the tapping signals based on time series analysis and statistical methods. Principal component analysis is used to reduce the size of the parameters. Scores are obtained for the dimensions related with finger tapping, including speed, accuracy, fatigue, and arrhythmia. A logistic regression classifier is trained to map the reduced parameters to global tapping severity (GTS) scores. The study based on 95 PD patients and 10 healthy elderly subjects shows that the computed scores correlate well to GTS scores and are significantly different across UPDRS scores.

In [11], optical hand tracking technologies are employed for the assessment of bradykinesia from the aspects of velocity, amplitude, and rhythm. The study conducted on 57 PD patients and 57 controls performing repetitive finger tapping, alternating hand movements, and alternating forearm movements confirm that optical hand tracking technologies can be applied for the quantification of bradykinesia of the upper extremity in PD.

In [12], a system consisting of an accelerometer and touch sensor is developed to find objective parameters for the finger tapping test in PD patients. A total of 14 parameters of finger tapping related to velocity, amplitude, rhythm and number are measured. The study based on 16 PD patients and 32 age-matched healthy volunteers shows that maximum opening velocity is the best of the 14 parameters to differentiate the healthy volunteers and PD patients due to its sensitivity and association with the UPDRS score.

In recent years, with the advancement in the technologies of embedded sensor components in smartphones and portable media devices, some works have been conducted to employ these devices to assess the severity of PD symptoms. In [13,14,15,16,17], iPhone and iPod are adopted for the quantitative measurement of hand tremor and gait characteristics. It is indicated that the usage of iPhone or iPod is dependent on the context of the application, such as the weight restrictions of devices, wireless Internet connectivity for iPod, and cell-phone coverage for iPhone.

To sum up, there are a lot of devices, systems, and approaches developed and applied to qualitatively evaluate the severity of the PD symptoms. However, most of the existing works focused on one symptom only and ignore the correlations among symptoms, which may lead to incomplete evaluation outcomes for PD patients.

3 The Proposed Framework

In this work, we aim to provide a convenient and effective way to PD patients and clinicians in the management of PD treatment. As aforementioned, the embedded sensors in smartphones have made it possible to employ smartphones to quantitatively evaluate the severity of PD symptoms. At the same time, the elderly are increasingly used to smartphones, who constitute the major population suffering from PD. Therefore, we propose a smartphone based interactive game platform – the Pumpkin Garden (PG) for PD patients to have daily monitoring. The framework of the proposed platform is as shown in Fig. 1.

Fig. 1.
figure 1

The conceptual framework of Pumpkin Garden.

Full size image

In particular, PG provides a farming-themed game environment on smartphones, and the game platform is currently implemented in iOS and installed on iPhone. Interactivity designs in the developed game enable a user to display behaviors reflecting cardinal PD related symptoms. More specifically, a user is asked to play the game after taking medications. As medications taking effects vary from person to person, the user is asked to indicate the time they take medications, as shown in Fig. 2(a).

Fig. 2.
figure 2

Interaction designs for assessing various PD symptoms: (a) Input for medication taking; (b) The designed 9 mini-tasks.

Full size image

There are a total of 9 mini-tasks designed as shown in Fig. 2(b) to assess the severity of PD symptoms, such as resting and kinetic tremor, bradyskinesia (slowness in movement), rigidity and posture disturbances, movement unbalance, micrographia, and gait freezing. During user playing the game, the raw data with respect to user behaviors will be collected, such as the finger positions and the accelerometer values from iPhone at each sampling time (i.e., the sampling interval is 30 ms). These raw data will be uploaded to a longitudinal behavior database and assessed by an assessment agent. In the next section, we will introduce the details of each task and the related features extracted from the collected user behavior data and iPhone embedded sensor readings. After assessment, a comprehensive report will be generated and sent to clinicians for review.

4 The Details of Tasks

We currently design 9 mini-tasks in the game platform to assess the severity of PD symptoms. There will be instructions shown to users before the beginning of each task. Users can simply follow the instructions to complete the tasks.

4.1 Single Finger Tapping

In this task, there is a circle shown in the middle of the screen as shown in Fig. 3(a). The user is asked to tap the screen within the range of the circle as shown in Fig. 3(b).

Fig. 3.
figure 3

Single finger tapping task: (a) The circle for users to tap within; (b) User tapping within the circle.

Full size image

In our design, before each task begins, the user can set parameters for the task based on his own profile. For the task of single finger tapping, the parameters that can be adjusted include the duration of the task and the size of the circle. The task will be more difficult if the size of the circle is smaller. The features to be extracted from raw data include: (1) the duration of the task; (2) the number of total taps; (3) the number of taps in range, which is the number of the user’s tapping position falling into the range of the circle; and (4) the accuracy of the taps, which is equal to the number of taps in range divided by the number of total taps.

4.2 Alternating Tapping

In this task, there are two circles shown in horizontally parallel in the screen as shown in Fig. 4(a). The user is asked to alternatively tap the screen at the position where one of the circles is displayed as show in Fig. 4(b), (c).

Fig. 4.
figure 4

Alternating tapping task: (a) The circles for users to tap within; (b) User tapping within one circle; (c) User tapping within the other circle.

Full size image

The parameters that can be adjusted include the duration of the task, the size of the circles, and the distance between the two circles. As in the first task, we extract the features of the duration of the task, the number of total taps, and the number of taps in range. Besides these features, we also extract the following two more features. The first one is the mean and standard deviation of tapping speed, which is calculated as the mean and standard deviation of the rate of change of tapping distance with time. The second one is the accuracy of alternating tapping. As this task involves alternating taps for two circles, a correct tapping is not only dependent on whether the tapping position is located in the range of circles but also rely on the previous taps. For example, if the previous tap happens at the left side, the next tap should go to the right side. If the user continues tapping on the left side, this tap cannot be counted as a correct one. Therefore, the accuracy of alternating tapping is calculated as the number of correct alternating tapping divided by the number of total taps.

4.3 Finger Tapping

In this task, there are two circles shown on the upper right and bottom left corners of the screen as shown in Fig. 5(a). The user is asked to place the index finger and thumb on the upper right and bottom left circles, respectively as shown in Fig. 5(b), and then bring the index finger and thumb to the center of the screen to make the two fingers contact as shown in Fig. 5(c). After contacting, the user will release his fingers from screen and put them back to the original positions and continue the procedure until the duration of the task ends.

Fig. 5.
figure 5

Finger tapping task: (a) The circles for testing; (b) Place the index finger and thumb within the circles; (c) Bring the index finger and thumb to the center of the screen.

Full size image

The parameters that can be adjusted include the duration of the task, the distance between the two circles, and which hand to be assessed. Except the duration of task time, we also extract the following features. Firstly, we extract the number of finger contacting. As the circles will move following fingers, we can measure whether two fingers contact by measuring the distance between the circle centers. If the distance between the circle centers is smaller than the diameter of the circles, then the fingers are considered as contacting. Secondly, we extract the feature of the mean and standard deviation of the distance of movements for each finger, which is calculated as the mean and standard deviation of a finger moving from the original position to the farthest position before it changes direction back to the original position. Based on the finger movement distance, we will further get the feature of finger movement speed.

4.4 Rest Tremor

In this task, the user is asked to sit still, put his hand on leg, and hold the iPhone flat on the palm of the most affected hand as shown in Fig. 6(a). We collect the raw 3-dimensional data reflected by the iPhone embedded accelerometer for feature extraction. In particular, the collected raw data are time series data of 3-dimensional coordinates. We then use the Fourier Transform to get the signal frequency and amplitude along each axis. If there is rest tremor, there will be a higher frequency and amplitude along at least one axis.

Fig. 6.
figure 6

Rest and postural tremor tasks: (a) Rest tremor; (b) Postural tremor.

Full size image

4.5 Postural Tremor

In this task, the user is asked to sit still, extend his hand forwards at shoulder height, and hold the iPhone flat on the palm of the most affected hand as shown in Fig. 6(b). As in the task of rest tremor, we also collect the raw 3-dimensional data reflected by the iPhone embedded accelerometer for feature extraction. The collected raw data are still time series data of 3-dimensional coordinates, based on which we can get the signal frequency and amplitude along each axis by using the Fourier Transform. If there is postural tremor, there will be a higher frequency and amplitude along at least one axis.

4.6 Kinetic Tremor

In this task, the user is asked to sit still and hold the iPhone using the most affected hand. Then the user starts from a position of outstretched arm extending sideways from the body at shoulder level, bends the arm at the elbow, brings his hand above his head to his heart in a semi-circle motion, and returns to original position using the same motion. An example of the movement in the task is shown in Fig. 7. The user is required to continue the movement until the specified task duration ends.

Fig. 7.
figure 7

Kinetic tremor task: (a) Arm extending sideways from the body at shoulder level; (b) Bring the hand to heart.

Full size image

As in the previous tasks of rest tremor and postural tremor, the feature extraction is still dependent on the collected 3-dimensional data reflected by the iPhone embedded accelerometer, which are time series data of 3-dimensional coordinates. Fourier Transform is applied to get the signal frequency and amplitude along each axis. If there is kinetic tremor, there will be a higher frequency and amplitude along at least one axis.

4.7 Micrographia

In this task, there are two modes designed: cued and un-cued. In the cued mode, there is a circle shown in the middle of the screen, and the user is asked to trace the circle with a finger of the most affected hand as shown in Fig. 8(a). The user can set the circle radius to adjust the size of the circle. In the un-cued mode, the user is asked to draw as large circles as they can, as shown in Fig. 8(b). The radius is calculated for each circle that the user draws to see whether there a trend of drawing smaller and smaller circles.

Fig. 8.
figure 8

Micrographia task: (a) Cued mode; (b) Un-cued mode.

Full size image

4.8 Coordination

In this task, there are two symmetric trails shown in the screen as shown in Fig. 9(a). The user is asked to trace the trails simultaneously from the bottom of the trails using the index finger from each hand as shown in Fig. 9(b). The user can set the parameter of repeated times to adjust how many times he wants to have the test. We extract the symmetry of finger movements as a feature, which is calculated as the mean distance between the original trails and the symmetric trails of finger movement against the y-axis. The feature of average distance from the finger movement trails to the original trails is also extracted.

4.9 Gait

In this task, there are two modes provided: waist and ankle. The user is asked to secure the iPhone to his waist or ankle, walk straight for 20 s, and then turn around to walk for anther 20 s as shown in Fig. 10. We collect the raw 3-dimensional data reflected by the iPhone embedded accelerometers to extract the acceleration feature as we have done in the tasks of rest tremor, postural tremor, and kinetic tremor.

Fig. 9.
figure 9

Coordination task: (a) Trails shown on the screen; (b) Finger moving following the trails

Full size image
Fig. 10.
figure 10

Gait task: (a) Waist mode; (b) Ankle mode.

Full size image

5 Discussions

In this paper, we have proposed a game platform for an objective assessment of the severity of PD symptoms to facilitate patients and clinicians to have a continuous monitoring on the progress of disease and the effectiveness of PD management. In this section, we will discuss some issues which require further exploration.

Firstly, it is necessary to investigate an appropriate way to secure iPhone to a user’s hand or ankle. As the user may have the problems of tremor and movement unbalance, it is very important to secure the iPhone to his hand or ankle to prevent the happening of accidents because of the user’s unstable holding. Currently there are already some works conducted to study how to secure iPhone. For example, we can use a glove to help the user to mount the iPhone to the hand [18] as shown in Fig. 11(a). Elastic bands and socks [18] are equipped to secure iPhone to ankles as shown in Fig. 11(b), (c). However, we still need to explore a convenient way for the user to secure iPhone to his hand or ankle without changing materials.

Fig. 11.
figure 11

Secure iphone to hand or ankle: (a) Glove; (b) Elastic bands; (c) Socks.

Full size image

Secondly, one of the purposes of the proposed game platform is to encourage users to have symptom assessment continuously. Therefore, exploring how to attract the users to play the game is worthy of further exploration. Although we currently already integrated some game design considerations in the platform, we need to incorporate more factors into the game to make it more interesting for the users to play. For example, we may add in the game factors of “score board” and “badges” to make users feel the gain of achievement, and then continue to play the game.

Thirdly, we need further study the correlation between UPDRS and the analysis of the features extracted from user in-game behavior data. As the features extracted from one or more tasks may relate with multiple items in UPDRS, we need to explore which items the task features are correlated to and how they are correlated with each other. For example, in UPDRS-part III, which is to evaluate the severity of upper limb motor systems, items 23 (Finger Tapping), 24 (Hand Movements), and 25 (Rapid Alternative Movements of Hands) are related to the features extracted from the tasks of single finger tapping, alternating finger tapping, and finger tapping in the designed game. In addition, exploring more features to be correlated with UPDRS may also need more research works.

6 Conclusion

Parkinson’s Disease (PD) is the second most common neurodegenerative disorder (after Alzheimer’s disease) in world. Though some medications have been found to be effective in PD management to optimize the quality of life and reduce burden of care, the conditions of PD symptoms of patients need to be monitored carefully to facilitate clinicians to adjust treatment for more effective PD management. However, current practice based on subjective observations and reporting from clinicians and patients in brief clinic consultations can not provide a holistic picture of patients’ symptoms and progress to the clinicians. Therefore, we proposed Pumpkin Garden, a game platform to facilitate the PD patients to have a continuous assessment of the severity of PD symptoms in a convenient and objective manner. The data collected from its on-going clinic trial will constitute a large scale interactive digital media personal dataset with respect to monitoring PD related symptoms. It may also provide medical researchers with a brand new perspective in assessing PD symptoms daily and treatment adjustment. In the future, we will continue research in the directions of how to secure the iPhone to users’ hands and ankles, the design of games to attract users to play games more frequently, and the correlation analysis of users’ in-game behavior data and UPDRS.