Ten scientific problems in human behavior understanding
- 132 Downloads
Human behavior understanding is of great importance for a variety of applications, such as personalized recommendations, smart home, urban planning, and anti-terrorism. Although there has been significant progress on the understanding of human behaviors, we still face a number of theoretical and technical challenges that need be further explored. In this article, we first outline the basic research process of human behavior understanding based on existing studies, and illustrate important issues in each step. Afterwards we describe main research challenges from the aspects of human behavior itself, the behavior related data, as well as the modeling and evaluations, respectively. Then we identify and explore ten most important fundamental open problems in this field. The proposed problem list is expected to provoke innovative studies on human behavior understanding, e.g., theory improvement and data collaboration. In this article, we also discuss possible ways that would be helpful for resolving the challenging problems.
KeywordsHuman behavior understanding Scientific problems Behavior characteristics Data analysis Modeling and evaluation
Human behavior refers to people’s actions and statuses shown in their daily lives, including body movements (e.g., hand gesture), daily activities (e.g., working, shopping), and crowd gathering, etc (Chen et al. 2012). In addition to these physical activities, psychological activities (e.g., emotions) and online activities (e.g., surfing the Internet) are also considered as human behaviors. As we known, human behaviors usually follow certain patterns or laws (e.g., bursts and heavy tails Barabási 2005), and revealing such behavior patterns has been an important research area in social science for many years. Recently, with technology advances in sensor-equipped devices and online social networks, it becomes prevalent that human behaviors are recorded as digital footprints. Moreover, academics in computer science have explored such digital footprints and made much progress in developing related models and techniques for human behavior understanding. In particular, these achievements have been applied in many fields, including smart home, urban transportation planning, health care, emergency warning and disaster monitoring.
Due to its scientific and practical values, human behavior understanding is attracting attentions from a growing number of governments and academia/industry communities. For instance, DAPRA’s 2015 report on 5-year plan for science and technology development1 proposed six disruptive basic research areas worthy of future study, one of which is computational models of human behavior. In 2017, Nature launched a new sub-journal named Nature Human Behavior, focusing on human behavior capture, modeling and analysis. Meanwhile, a series of studies on human behavior understanding have been conducted, ranging from theoretical foundations to technology innovation, system development, and practical applications. For example, Song et al. (2010) studied the limits of predictability in human mobility; Subrahmanian and Kumar (2017) reported four major challenges of human behavior predictive models; Youyou et al. (2015) inferred user’s personality using Facebook dataset; Kramer et al. (2014) studied the emotion contagion behavior across users on the social network platform.
While there has been significant progress on the understanding of human behaviors, we still face a number of theoretical and technical challenges that need be further explored. In this article, we first outline the basic research process of human behavior understanding based on existing studies, and illustrate important issues in each step. Afterwards, possible challenges are analyzed from the aspects of human behavior itself, the behavior related data, as well as the modeling and evaluations, respectively. In particular, we explore ten fundamental open problems, with hoped-for potential breakthroughs promising to advanced human behavior understanding models and techniques. Finally, we envision several possible ways that would be helpful for resolving the challenging problems.
2 Research framework
Data capture We can collect data by utilizing various sensing devices, including infrastructures deployed (e.g., the pre-deployed cameras) and sensor-equipped mobile devices (e.g., the smart watches and smart phones). Meanwhile, social media, especially mobile social networks (e.g., Foursquare, Twitter), can also be used for data collection, which is prevalent with the advanced sensor and network technology. For example, the cameras on the road are capable of monitoring large scale human mobility behaviors, the sensor-equipped smart watch is able to capture fine-grained user-level activities, and location-based social networks (e.g., Foursquare) allow users to record and share their locations with corresponding comments. In addition, due to the diversity of data sources, researchers can capture various formats of data, such as audio, video, text, and image, which increases the possibility of human behavior modeling from multiple aspects.
Modeling and analysis As human behaviors follow certain patterns, we can propose certain hypotheses onto the deduced observations from human behavior digital footprints, and apply corresponding models to understand human behavior to some extent. All the analysis procedures that manage to fit human behaviors into mathematical models are included in this component. In specific, it mainly consists of three subcomponents, i.e., observed patterns, inference algorithm, and model hypotheses. We should explore how to match these three modules together for problem solving, and the key is to develop both effective inference algorithms and appropriate model hypotheses that can eventually reflect the observed patterns in a decent and comprehensive manner. Specifically, a significant issue is to determine the computationality, which aims to theoretically prove the extent or bound that human behavior can be understood, e.g., whether there is available data and model that can be used to understand human behaviors, what the optimal experimental results could be.
Performance evaluation For performance evaluation of human behavior understanding, comparability should be firstly considered. In particular, we expect there should be several universal benchmarks that can be used for all researchers to test their studies equivalently. Further, the performances can be evaluated using a set of metrics, for example, the employed algorithm should be efficient in time, space and energy, and the results are expected to be accurate and precise. Moreover, the applied method should be robust and maintain effective when facing uncertain factors. For the understanding of some special behaviors, we might need to consider some other performance measurements, e.g., scalability, which depends on the characters of human behaviors and the application scenarios.
Applications Effective human behavior understanding can be applied into many applications, such as personalized recommendations, smart home, public opinion monitoring, urban planning, disease treatment, and anti-terrorism. For example, recognizing human’s indoor activities helps provide context-aware services, e.g., turning on the light; according to the repost behaviors, we could measure one’s characteristics on social networks to help with information diffusion and analysis; understanding human’s mobility patterns can help transportation planning in a city; monitoring crowd gathering is useful for emergency management and control.
3 Research challenges
Human behavior understanding faces many challenges, which can be summarized from the following three perspectives.
3.1 Challenges from human behavior itself
Human behavior can be influenced by various factors, including culture diversity, different physical spaces, and multi-level social relations. Accordingly, there exist some characteristics of human behavior itself that are difficult to sense and analyze, e.g., capriciousness, evolution, and multiple granularity. Taking the capriciousness as an example, one person’s emotion can be easily affected by the behaviors of other people, while the influence factors are usually dynamic and uncertain. Therefore, capturing and quantifying these influence factors becomes a challenge. In addition, as human behavior ranges from fine-grained hand gestures to large-scale crowd movements, it could be another technical challenge to understand human behaviors in multiple granularity. For instance, understanding hand gestures requires techniques sensitive to capturing subtle changes, while for crowd behavior monitoring it is crucial to determine the global features from various individual behaviors. Yu et al. (2018) used a smartphone to recognize human computer operations from the keyboard input. To capture the input audio, this work considered the position of the smartphone, and meanwhile, the technology and algorithms for signal processing and semantic analysis are also significant. In Du et al. (2018), the mobility behavior of a group is characterized by collective motion features of all the members in the group.
3.2 Challenges from the data
Not only because of the development of sensor-equipped mobile devices and online social network services, but also due to the dynamics of human behavior across different domains and spaces, there are unprecedented as well as diverse “Big Data” related to human behaviors. In specific, multiple interfaces (e.g., smartphones, mobile apps, and websites) are broadly used that can capture human behaviors with different formats (e.g., text, audio, and image) and various spatial-temporal information (e.g., different places and time slices). However, when researchers manage to conduct analysis on such plenty of datasets, they undoubtedly will face challenges during procedures such as data acquisition and data processing/analyzing. We identify following key research challenges that may be involved, i.e., data fragmentation, data heterogeneity, data representativeness, data sparsity, imbalanced data distribution, and spatial-temporal correlation. For example, due to the fact that population biases exist across different social media platforms (Mislove et al. 2011), it becomes extremely difficult to gather human behavior data that could accurately and completely reflect/represent the real-world behavior, which hence causes the challenge of data representativeness. In addition, there also exists imbalanced data distribution problem since valuable data records may only have a small proportion in the whole dataset. That thus makes difficulty in modeling rare behavior patterns using such insufficient dataset. In short, all the identified challenges are crucial for human behavior modeling and understanding, and researchers need to take proactive actions to deal with them.
3.3 Challenges from modeling and evaluation
There have been various models proposed for human behavior recognition and prediction, and these models usually regard the behavior computable as the default. However, human behavior is usually related to many dynamic and uncertain factors. Therefore, theoretical verification on computationality is essential yet difficult. From the aspect of evaluation, a valuable research should be comparable, whereas there is no standard evaluation metrics or systems as the context of human behavior varies. The determining of techniques applied, the baseline method, and the experimental environment is still to be further studied for building a standard of human behavior understanding. In addition, it is also crucial to determine what performance should be considered (e.g., accuracy, time-efficiency, energy efficiency, robustness), which depends on the behavior characteristics. Taking human activity recognition (such as fall detection) as an example, firstly, recognizing the fall of older adults should be in time so that effective services can be provided. Second, the technique employed cannot be affected by the environments, which means it should be robust. Besides, for continuous behavior monitoring, energy efficiency is also important, especially when mobile devices are used (Liang et al. 2014). In brief, the evaluation for human behavior understanding makes a demand on the comprehensive consideration of behavior characteristics as well as the optimized model.
4 Ten most important problems
Based on the challenges discussed above, we identify ten most important problems in human behavior understanding. The first three come from challenges related to human behavior itself, problems No. 4–8 are related to the data, and the last two are derived from the challenges of modeling and evaluation.
(1) Behavior evolution Human behavior can change over time, for example, a person preferring literatures could possibly turn to reading social science when she grows older. We call this characteristic behavior evolution, which requires the model constructed to adapt to such evolution as well as identify when the behavior changes. However, most existing models only output the results at present without any temporal/evolutionary details, and they cannot support the detection of behavior evolution. To address this problem, there are several questions that should be considered. For instance, which types of human behavior can evolve? Can the applied model detect such behavior changes, including when and why they happen? Can the model adapt to the change and output time-variant results? Has the range of all possible results been known? Machine learning algorithms are widely used for human behavior understanding, especially pattern recognition, and existing studies usually utilize predefined features to identify one or multiple specific behavior patterns. Apparently, such algorithms may not be suitable for the understanding of dynamic behaviors. A smart model is to be developed that can make adjustments adaptively, e.g., a multi-stage model (Subrahmanian and Kumar 2017). More importantly, as we hope to reveal the law of human behavior, the descriptive information is expected to be obtained (e.g., why the pattern changes), which also helps decision making.
(2) Multi-aspect of human behavior Multi-aspect refers that the appearance of a specific human behavior is not unique, i.e., it shows difference in multiple aspects, such as human’s personality. For example, a person can show calm when communicating with colleagues, while she can be lively when facing the families. Moreover, there also may be some unknown appearances. As our understanding for a person is usually one sided, it is difficult to obtain an overall picture. As a result, many questions need to be addressed, for instance, what reflects the multi-aspect of human behavior? Can all aspects be understood? Which aspect can/cannot be sensed and analyzed? Is each aspect equal for characterizing human behavior? What are their weights? Facing these problems, we can first explore the method that can capture an overall portrait, and then make further analysis for each behavior aspect.
(3) Capriciousness Human behavior is capricious for the others and the environment. What factors can influence human behavior easily? Can these influences be quantified? As we know, a person’s emotion could be affected by others, e.g., conformity psychology, emotional contagion. In most cases the influence is caused by subjective factors, the analysis of which needs to be combined with the knowledge in both psychology and sociology. Meanwhile, the influence is difficult to measure, and we are not sure whether all factors can be quantified. For the factors that are difficult to quantify, how can we estimate or model the influence? The solution may be studied by referring to similar models in other fields.
(4) Data fragmentation To capture a person’s daily life activities, researchers have to apply various sensing devices that are distributed at separate places and time slots, which eventually leads to the fragmentation of the collected dataset. For example, the captured data can be distributed at different places (e.g., home, office, and gym) with different time logs (e.g., morning, noon, and night). Hence, how to merge/aggregate the fragmented data together is necessary for understanding a comprehensive human behavior pattern. The exploration may need to consider the following questions. What are the relations among fragmented data? Which part of the fragmented data is important? According to different research problems, do we have to balance the weights across all the fragmented datasets? Or is there any criterion that could guide us for dealing with data fragmentation in a more effective way?
(5) Data heterogeneity Heterogeneity emphasizes the fact that the collected data are constituted by different formats, such as image, text, video, and audio. Human behaviors can be tracked by various sensing devices (e.g., cameras, smartphones, and laptops), and meanwhile they are recorded as heterogeneous datasets. Yu et al. (2018) identified on-site users for social events by utilizing the information of text, time, and coordinates of social media users. Although heterogeneous data have been utilized in prior studies, how we can collaboratively fuse all these different formats of data for human behavior understanding remains to be explored. For instance, when there is a parade happening in a city, it is usually expected that the event can be characterized by not only text-based news reports, but also pictures, videos and user comments. It is a challenging issue to match all these heterogeneous data together, e.g., how to discover the relationship between text-based contents and pixel-based images? How can we complement one type of data with other types of data? In what situations can the merging of heterogeneous data achieve better results than using singular data format? How can we balance the importance between various formats of data? Only if we could address these questions, can our model be effective in human behavior modeling.
(6) Spatial-temporal correlation Location and time are two key features of human behaviors, and usually there are spatial and temporal correlations among the collected behavior digital footprints. For example, user’s periodic activities (such as breathing events) could result in temporal correlated data records; similar activities at similar places (such as shopping in a mall) would produce spatial correlated data; trajectories in workdays or on weekends present obvious spatial-temporal correlation (Yu et al. 2015). Pan et al. (2013) have proved the power of collaboratively exploring the spatial-temporal correlation between traffic information and online reviews. However, we still need to address a number of issues to fully understand such spatial-temporal correlations, e.g., in what ways could we match spatial and temporal data for problem solving? Is the employed algorithm capable of taking data’s spatial-temporal correlation into consideration during its learning process? If it has such a capability, then how can the spatial-temporal correlation constraint be formalized? Further, how can we evaluate the effectiveness of using spatial-temporal correlation features? Are there methods that can be used to adjust different weights across spatial and temporal factors? We also need to take care of fake correlations and over-usage of correlations. Some facts may seem to be correlated from the data, but indeed they are not. Moreover, correlations sometimes may be regarded as causality by mistake.
(7) Data representativeness The data collected may be not representative due to several reasons. (1) Population biases across different platforms. According to the investigation of Sprout Social,2 we know that social media demographics vary a lot across different platforms, and such deviation may lead to misjudgments when applying knowledge from one platform to another. Therefore, we need methods and criterions to measure the similarity between population distributions of different platforms. Further, what sampling methods can be applied to collect a representative dataset? (2) Data protection from providers. It has been discussed in Ruths and Pfeffer (2014) that researchers are usually left in the dark about when and how social media providers change the sampling/filtering method of their data streams. Hence, the data available for researchers on the public platforms may be pre-processed by their providers due to privacy, safety, and other issues, which does not truly reflect users’ real behavior patterns and might eventually affect the results of human behavior modeling. Therefore, we should focus on reasoning whether the obtained data is qualified for the problem or not, especially on modeling human behavior patterns.
(8) Data sparsity Although we are in the era of big data, we may still lack useful data for human behavior understanding. For example, trajectory prediction is a popular research topic, while in most circumstances, we can only obtain a sparse trajectory dataset due to privacy concerns, budget constraints, and many other factors. Wang et al. (2017) utilized spatial-temporal correlation for destination prediction with sparse GPS trajectories. Another case, researchers may be interested in user’s abnormal/rare activities to fully characterize human beings, whereas such data records usually only take a little proportion of the whole dataset, which causes difficulty in conducting this kind of research. Therefore, how to deal with data sparsity becomes crucial. There are several off-the-shelf methods that can be applied, while they are usually incapable of addressing such emerging challenges. We still need to seek effective methods to handle imbalanced data distribution.
(9) Computationality Can all human behaviors be computed (including captured and modeled) by using the data and techniques available? How can the computationality be proved theoretically? In which conditions can they be computed? What limits the computationality? For example, Song et al. (2010) explored the limits of predictability in human mobility by measuring the entropy of each individual’s trajectory. Zhang et al. (2017) proposed the Fresnel zone model to reveal the precision boundary of Wi-Fi based activity recognition, which is different from traditional pattern-based behavior recognition approaches (Wang et al. 2018). Inspired by these studies, we should first think about a fundamental question: to what extent is human behavior computable? Only based on the computationality, can we improve the performance in human behavior modeling, and also broaden the current field on human behavior understanding. This part still needs further exploration in theory.
(10) Comparability Is it possible to build a benchmark for human behavior understanding? Are there a set of public datasets for evaluation? Are there any techniques or algorithms that can be used as baselines? For specific behavior understanding, what are the basic evaluation criteria? And what is the numerical standard? For similar research problems (e.g., user positioning), researchers usually apply different sensing techniques, datasets, models and algorithms. Moreover, the experimental environments are distinguishable. Thus, these studies cannot be compared completely because of these differences, and we cannot determine what result is acceptable, what result is perfect, and what result is unreasonable. It still lacks a complete evaluation system to highlight the experimental conditions and performance requirements.
Interdisciplinary collaboration Understanding our human being needs the collaboration of experts from multiple related disciplinaries. For instance, understanding the evolution and capriciousness of human behavior should combine the knowledge in both psychology and sociology, which can explore how human behavior evolves and why human behavior is capricious and what can affect human behaviors. It is studied users’ shopping intention can be affected by others’ comments (Morris et al. 2014). With the support of these studies, we can further explore whether and how the factors can be quantified.
Cross-organization collaboration Different organizations need to collaborate to deal with the challenges derived from the data. The most primary task for data fragmentation is merging/aggregating fragmented data from different devices or platforms to comprehensively understand human behavior pattern. Only if we could link the same identity from different data sources, can we merge the fragmented data of human behavior to build universal models. Some possible approaches are discussed in Yi et al. (2018), including structure/content-based identity linking methods by leveraging graph-based techniques. Furthermore the data from different sources would be heterogeneous. Thus different organizations should work together to share the data and use a universal model to describe the data.
Leveraging crowd power Crowd power can be leveraged for human behavior understanding in several aspects. First, crowdsourced footprints can be used for understanding large-scale community activities. This is effective to solve the problem of collecting sensory data of community-scale users. Second, training data uploaded by the crowd would be valuable for behavior recognition. Third, ground truth data is significant in behavior understanding. While manual annotation of large-scale dataset is time-consuming, we can leverage human intelligence of crowd power to fulfill the annotation task (von Ahn et al. 2008).
Focus on fundamentals There have been early works that attempt to tackle the challenging problems and in some time make the field forward. But we need to focus more on the fundamentals, such as common principles, theories, and benchmarks. For instance, it is necessary to conduct the theoretical exploration on the extent that human behavior can be sensed or understood. This is related to the properties of specific human behavior, and may also depend on the sensing technology as well as the characteristics of collected data. It is also essential to build benchmarks for human behavior understanding. For example, based on the characteristics of different localization technologies (e.g., infrared, RFID, GPS, Bluetooth, Wi-Fi), we should try to define a standard experimental environment for each technology and propose the bound of the location results.
Human behavior understanding is of great importance for various areas, ranging from personalized services, community-based recommendations to city-scaled planning, and other specific fields such as disease treatment and anti-terrorism. Considering the characters of human behavior itself, the behavior related data, as well as the modeling and evaluation of human behaviors, we outlined the ten most important scientific problems, but certainly not limited to. For example, we leave out specific modelling problems such as building a high dimensional model, optimization algorithms for model learning, and combining advanced technologies like deep learning into existing models, which all play significant roles in human behavior understanding. When a scientific field runs out of challenging problems, it will also run out of steam. We expect the proposed problem list will provoke innovative studies on human behavior understanding, and most importantly, inspire a constantly updated unsolved problems as well as solutions for the field.
This work was supported in part by the National Science Fund for Distinguished Young Scholars (No. 61725205), the National Basic Research Program of China (No. 2015CB352400), and the National Natural Science Foundation of China (Nos. 61332005, 61772428).
- Mislove, A., Lehmann, S., Ahn, Y.-Y., Onnela, J.-P., Rosenquist, J.N.: Understanding the demographics of twitter users. ICWSM 11, 5 (2011)Google Scholar
- Pan, B., Zheng, Y., Wilkie, D., Shahabi, C.: Crowd sensing of traffic anomalies based on human mobility and social media. In: Proceedings of the 21st ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, ACM, pp. 344–353 (2013)Google Scholar
- Wang, L., Yu, Z., Guo, B., Ku, T., Yi, F.: Moving destination prediction using sparse dataset: a mobility gradient descent approach. ACM Trans. Knowl. Discov. Data 11(3), Article No. 37 (2017)Google Scholar
- Morris, M. R., Inkpen, K., Venolia, G.: Remote shopping advice: enhancing in-store shopping with social technologies. In: Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work and Social Computing, ACM, pp. 662–673 (2014)Google Scholar