1 Introduction

Memory is the ability of our neural system to encode, store and retrieve information. It is the accumulation of one person’s past activity, feeling and many other information. Episodic memory is a part of long-term memory, together with semantic memory, it forms the integrity of memory system. The concept of episodic memory first appeared around 1970s [1]. Unlike semantic memory which concentrates on the actual facts about this world, episodic memory is the memory of autobiographical events. These events including contextual information (who, what, when, where, why) which can be explicitly elaborated and memorized. Generally speaking, episodic memory allows an individual to vividly go back in time to recall the event that occurs at a particular time and space. Episodic memory is vital to human being and Tulving [2] muse it as “a true marvel of nature”. Moreover, according to the experiment done by Klein [3], the loss of episodic memory not only impact individual’s memory for the past, but also may extend the influence of people’s ability to anticipate the future. Meanwhile, because of the limitation of brain capacity or some severe memory illness, it is essential for us to find some external solutions for aid episodic memory retrieving.

Lifelog is the detailed chronicle of personal life involving large number of data that can be captured automatically by wearable technology. It is a process of gathering, processing and recalling life experience passively. Individuals, defined as lifeloggers, will carry out multiple sensors which can sense individual’s living environment and activities. Visual information captured by wearable technology in personal lifelog can be very helpful for human memory aiding. Lee [4] proved that lifelogging technologies have the potential to provide memory cues for people who struggle with episodic memory impairment. And Aizawa [5] have put forward five cues based on context and content for efficient retrieval of lifelog data. Moreover, Chen [6] introduced a work-in-process prototype lifelog searching system to augment human memory, which has obtained positive experimental results.

2 Goal and Approach

For aiding episodic memory in lifelog system, we used to define cues as memory triggers. Most existing cues in lifelog system mainly focus on contextual information, which is captured from lifelog image or by using sensors like GPS [7]. We can find that these cues have nothing to do with user’s own behavior and care little about users themselves. In this research, we aim to involve user’s own status into lifelog system, so as to help finding important memory for lifeloggers. Here, user’s own status including inner psychological mood and external physiological activity. For a clearer explanation of our research perspective, we corresponds the user status to two cues: Special sentiment cue and special movement cue.

We innovatively use these two cues to propose a new user-related lifelog viewing prototype for aiding episodic memory more efficiently. Our proposed prototype mainly consists of two parts: a wearable sensor system and a web-based lifelog viewer. Unlike traditional contextual lifelog cues, we introduce personal sentiment and movement condition as new cues to enhance the effectiveness of memory recall.

To achieve this goal, we set up a wearable sensor system (see Fig. 1). We use Autographer attached on user’s head to automatically capture face photo, which is used in emotion detection. Meanwhile, we use an Android smartphone embedded with motion sensors like accelerometer and gyroscope to continuously get data to acquire movement situation. Moreover, we use another Autographer hang on user’s neck to capture lifelog image constantly. Because of the cross-platform compatibility and usability of web, we build a web-based lifelog viewer as the output. The appearance of the viewer is shown in Fig. 1. Lifeloggers can easily access important memory by using two proposed new cues in this viewer.

Fig. 1.
figure 1

Prototype overview

Full size image

3 System Design

To achieve our goal, we come up with the idea of providing two user-related cues for lifeloggers. The system overview is shown in Fig. 2. The system is mainly consisting of 1 web-based lifelog viewer and two accessory sub systems: a sentiment detection system used to capture sentiment data and a movement detection system used to capture movement data. In this section, we will describe special sentiment cue and special movement cue in detailed separately.

Fig. 2.
figure 2

System overview

Full size image

3.1 Special Sentiment Cue

Human’s emotions are complex and diverse. Ekman formally put forward six basic emotions in 1972 [8]. Sentiment classification in our system uses six basic emotions put forward by Ekman. Six emotions are: Happiness, sad, surprise, anger, fear and disgust.

In order to get sentiment data for forming special sentiment cue, we use Face++ API [9] to detect sentiment situation through face photo. Figure 3 shows the main procedure of processing sentiment data vividly. Firstly, we need to capture face photo automatically using Autographer. Secondly, the face photo will be input into sentiment detection system to get sentiment data. Here, emotion data consist of two parts: the emotion status and the time information which shows when this emotion has been detected. Thirdly, we match the time of each emotion with the record time of each lifelog image. Finally, we output the result to lifelog viewer for lifeloggers. In the example, the face photo was detected as happiness and it was detected at 2018/03/18 15:12:38. Then, we can find the lifelog image taken at this time. Finally, we can judge that this lifelog image was detected as “happiness”.

Fig. 3.
figure 3

Sentiment detection process

Full size image

3.2 Special Movement Cue

Move is the change in position of an object over time in physics, which is described in terms of displacement, distance, velocity, acceleration, time and speed. Human movement describes the way human moves. The classification of movement is quite complicated. To simplify the result, our proposed method involves mainly five kinds of movement: still, walking, running, on bike and on vehicle. On the other hand, in order to display user’s movement situation more vividly, we introduce movement timeline in our method.

We use accelerometer embedded in Android smartphone to capture the pure movement data of lifeloggers. Then, we import the raw accelerometer data into movement detection system. Together with Moves API [10], we can clearly get the result of movement situation. The result consists of variety information, including date, movement, start time, end time, etc. We use movement, start time and end time to form a movement timeline. After getting movement situation, we will match movement time with the time of lifelog image. By matching time, we can clearly find out the movement situation of each lifelog image. Finally, we will output the result to lifelog viewer. The process is shown in Fig. 4.

Fig. 4.
figure 4

Movement detection process

Full size image

4 System Implementation

4.1 Hardware Implementation

For using our system, users need to carry the following three wearable devices: an Autographer attached to a hat on lifelogger’s head for face photo acquisition, another Autographer hang on lifelogger’s neck for lifelog images capturing, an Android smartphone in lifelogger’s pocket for movement data acquisition. To sum up, our system mainly involve two kinds of hardware: Autographer and Android smartphone.

Autographer. Autographer is a hands-free wearable camera that is used in current research. It is 90 × 36 mm in size and weighs approximately 58 g which makes it easy to clip on any clothes or hang on a neck. Without external interference, the capturing frequency of Autographer is 30 s per image and this frequency can change due to the changes in external environment. Autographer is embedded with some sensors, including color sensor, magnetometer, PIR, temperature GPS, etc. Therefore, Autographer is favored by lifelog researchers for its superior performance and relatively intelligent shooting.

Android Smartphone. Smartphone is a mobile phone that has a mobile operating system and can expand functions by installing applications, games and other programs. Android is a free and open source based operating system which is s led and developed by Goggle and the open mobile alliance. As one of the most popular mobile operating systems, Android smartphone have covered more than half in the market. Nowadays, with the development of mobile technique, smartphones are always embedded with variety of sensors like accelerometer, gyroscope, gravity sensor, magnetic field sensor, etc. These sensors can be read by developer using Android SDK and is useful in detecting smartphone holder’s movement condition.

4.2 Software Implementation

For the software part, we mainly use Browser/Server (B/S) structure to build our web viewer. The structure is consisting of three layers: presentation layer, application layer and data layer. The presentation layer is the web browser which can be run on plenty of platform like desktop, smartphone and laptop. When lifelogger is viewing our system, presentation layer will send a request to application layer, which is a web server. Then, web server will send SQL request to the database and get a reply. It will send the reply back to web browser and the data will be formed into a more legible result and finally be shown to users. Moreover, we mainly use Java and JavaScript as developing language. We use Apache Tomcat 7.0 x as our local web server and use MySQL for data management.

4.3 Lifelog Viewing System

This part will describe the detailed interaction of our web-based lifelog viewing system. We will show each page respectively.

Default Page. Figure 5 shows the default page of our lifelog viewer. The default page demonstrates all lifelog images which is ordered by time. Lifeloggers can click “sentiment” button to enter sentiment page, click “movement” button to enter movement page and click “combination” button to enter combination cue.

Fig. 5.
figure 5

Default page

Full size image

Sentiment Page. Figure 6 shows the result of using special sentiment cue to classification all lifelog images. We can find that in the sentiment page, we mainly involve six kinds of sentiment, they are happiness, sad, surprise, anger, fear and disgust. For presenting sentiment vividly, we use emoji to represent each sentiment. The corresponding lifelog image of each sentiment will appear under each sentiment’s signal.

Fig. 6.
figure 6

Sentiment page

Full size image

Movement Timeline. Figure 7 shows movement timeline. we can see that we involve still, walking, running, on bike and on vehicle and there is a movement timeline showing in the middle of the page. By clicking the “view” button of each movement phase, we can enter to browse the detailed lifelog image of each segmented event.

Fig. 7.
figure 7

Movement page

Full size image

Movement Page. Figure 8 shows detailed movement page. The page consists of three parts: slides section(a), choose bar(b) and result section(c). In slide section, the images corresponded to each movement will be played automatically. This can reduce the time of browsing all images. As choose bar shows 6 sentiment emoji, lifeloggers can click the button to view the images satisfy both specific movement and specific sentiment. The result of images will be shown in result section ch sentiment will appear under each sentiment’s signal.

Fig. 8.
figure 8

Movement detail page

Full size image

5 Preliminary Evaluation

5.1 Participants

In order to evaluate usability and efficiency of using our system, we recruited 6 participants, aging from 22 to 26. All participants are students who have general knowledge of computer and have the experience of using web browser. Before our experiment, there is a pre-description of how to use our devices and system.

5.2 Method

The method we used for evaluation is described in the following 3 steps:

  1. 1.

    Each participant is asked to use our wearable devices for successive 5 h in 1 day. There are no strict regulations on when to start. However, we strongly recommend choosing 10:00am to 3:00pm as recording period.

  2. 2.

    After recording, hardware devices can generate three data: sentiment data, movement data and lifelog images. These data are imported into our proposed lifelog system and a related lifelog viewer is generated.

  3. 3.

    Participant then tried to use our web viewer by their own. Our viewer contains two parts: Sentiment cue viewer extracts vital memory using participant’s sentiment situation. Movement cue viewer extracts important images and orders by using movement timeline, which also enables the function of using sentiment cue. Participants are required to record how many events they can reminisce by using each viewer to test the efficiency of our system.

  4. 4.

    After using our system, a personal interview is given to each participant. Participants are asked to answer a few questions and score their feeling based on Likert Scale (1 = very negative, 5 = very positive).

5.3 Result

In our evaluation, all participants have successfully completed the plan and have given effective feedback. In order to get the result of our evaluation, we collected 6 questionnaires from 6 participants (3 males and 3 females) and analyzed their feedback. Since our evaluation is divided into two aspects: efficiency and usability, our results analysis will also be carried out separately in these two aspects.

Table 1 reflects the efficiency of our proposed system. To get the result, we calculate the average amount of each question. As shown in Table 1, the result of each question is 6.50, 8.00 and 9.83. From the result, we can see that compare to recalling memory in default manner (not using any assistant), using any method in our proposed system can increase the number of recalling events. With the combination of movement cue in its interface, we can find the result of using movement cue shows better performance.

Table 1. Result for efficiency evaluation.
Full size table

Another important aspect of our evaluation is the usability of proposed wearable device system and web-based viewing system. As the answer is based on Likert Scale, the average score of each question can clearly represent use experience of our hardware and software system. Table 2 shows the result. We can see that the score of all question is above average level, which shows positive result in our system’s usability. Moreover, we can see that the usability of hardware system gets lowest score. This reminds us we need to improve our hardware devices to make it easier to be carried out.

Table 2. Result for usability evaluation.
Full size table

6 Related Work

6.1 Related Work on Sentiment Detection

Emotion is a general term for a series of subjective cognitive experience. Whether positive or negative, emotions can motivate people to act and influence individual’s future somehow. Ever since human beings are aware of the importance of emotions to ourselves, with the increase of sensor accuracy and the rise of computer vision, we are becoming more and more concerned with the recognition and interpretation of emotions. Nowadays, there are mainly three methods to realize emotion detection: (1) Speech emotion detection [11, 12]. (2) Physiological signal detection [13]. (3) Facial emotion detection [14, 15].

In our proposed methods, we choose face photo to detect emotion changes among three methods which is mentioned above. To finally decide which method to choose, we have investigated the merits and demerits of three methods. For speech emotion detection, the accuracy is the highest one. However, it only detects emotional changes when people speak, making detection discontinuity. As for physiological signal detection, subjects need to wear all kinds of heavy and tedious sensors, which make daily life inconvenience and not natural enough. Compare to the former two methods, the accuracy of facial emotion detection has become higher due to deep learning staff. Subjects only need to wear a hat hanged with a camera and detection can be persistent [16].

6.2 Related Work on Movement Detection

Movement detection, or activity detection aims to recognize the actions of human from the observation on human’s actions. It has captured the attention of computer science communities ever since 1980s. Up to now, mobile devices are becoming increasingly sophisticated, and the latest mobile phones usually embedded with all kinds of sensors. Among them, accelerometer is widely-used for movement detection [17]. Accelerometer measures the combination of gravitational acceleration and object’s motion acceleration in the direction of x, y, z axis. Gravitational acceleration is always vertical to the earth, it can measure the angel change between objects and ground. Motion acceleration can detect the speed changing of the object [18].

6.3 General Related Work

Our work is closely related to the system developed by Harvey [19]. In their proposed system, they innovatively arose the idea of involving sitting behavior into the lifelog system. They use a sensor, named activePal, to catch subject’s sitting event. Meanwhile, they used Vicon Revue, one kind of wearable camera, to capture lifelog image automatically. In their research, they extracted lifelog images when user is sitting for observation purpose. They used their system to observe the living habit of elder adults. The idea of involving user’s own status into the lifelog system and making behavior as a new cue inspired the birth of our proposed system.

In this paper, we include Special Moment Approach and Spatial Frequency Approach for reviewing the lifelogs, which is very closely related to Wang’s [20] work. Special moment approach is a technique for extracting episodic events. Spatial frequency approach is a technique for associating visual with temporal and location information. Heatmap is applied as the spatial data for expressing frequency awareness.

7 Conclusion

In general, our research explains the limitation of memory cue in lifelog system for aiding episodic memory. In order to improve the situation and enhance the efficiency of aiding episodic memory, we propose two user-related cues in this research and has implemented them into a web-based lifelog viewer.

In the proposed system, we mainly implement special sentiment cue and special movement cue as two new user-related cues for lifeloggers to retrieve memory. We use two Autographers and an Android smartphone to capture sentiment data, movement data and lifelog images. After processing the obtained data, we generate a web viewer for lifeloggers to use sentiment cue and movement cue to view lifelog images and retrieve memory.

We assume that lifeloggers can wear our proposed hardware devices in their daily life. After recording for the whole day, lifeloggers can upload their data onto our system and generate their own web viewer. They can retrieve their memory efficiently and conveniently by using our proposed system.

In order to test the usability and efficiency of our proposed system, we have included some participates in our evaluation. The feedback is quite positive.