PDPNN: Modeling User Personal Dynamic Preference for Next Point-of-Interest Recommendation
- 143 Downloads
Abstract
Next Point of Interest (POI) recommendation is an important aspect of information feeds for Location Based Social Networks (LSBNs). The boom in LSBN platforms such as Foursquare, Twitter, and Yelp has motivated a considerable amount of research focused on POI recommendations within the last decade. Inspired by the success of deep neural networks in many fields, researchers are increasingly interested in using neural networks such as Recurrent Neural Network (RNN) to make POI recommendation. Compared to traditional methods like Factorizing Personalized Markov Chain (FPMC) and Tensor Factorization (TF), neural network methods show great improvement in general sequences prediction. However, the user’s personal preference, which is crucial for personalized POI recommendation, is not addressed well in existing works. Moreover, the user’s personal preference is dynamic rather than static, which can guide predictions in different temporal and spatial contexts. To this end, we propose a new deep neural network model called Personal Dynamic Preference Neural Network(PDPNN). The core of the PDPNN model includes two parts: one part learns the user’s personal long-term preferences from the historical trajectories, and the other part learns the user’s short-term preferences from the current trajectory. By introducing a similarity function that evaluates the similarity between spatiotemporal contexts of user’s current trajectory and historical trajectories, PDPNN learns the user’s personal dynamic preference from user’s long-term and short-term preferences. We conducted experiments on three real-world datasets, and the results show that our model outperforms current well-known methods.
Keywords
Point-of-interest recommendation User personal dynamic preference Recurrent Neural Network1 Introduction
Due to the prevalence of smart mobile devices, people frequently use Location-Based Social Networks (LBSNs) such as Foursquare, Twitter, and Yelp to post check-ins and share their life experiences. Point-of-interest (POI) recommendation has become an important way to help people discover attractive and interesting venues based on their historical preferences. Many merchants also use POI recommendation as an important channel to promote their products.
POI recommendation has been extensively studied in recent years. In the literature, Markov Chains (MC) [1, 2], collaborative filtering [3, 4, 5, 6], and Recurrent Neural Networks [7, 8, 9, 10, 11] are three main approaches. These methods commonly focus on capturing either the user’s short-term preferences or long-term preferences. User’s short-term and long-term preferences are both important for achieving higher accuracy of recommendation [12]. The short-term preference is greatly influenced by time and POIs the user has visited recently, while the long-term preference reflects the user’s personal habits thus all historically visited POIs need to be considered. However, the user’s personal long-term preference, which is crucial for personalized POI recommendation, is not addressed well in existing work. Moreover, the user’s personal preference is dynamic rather than static, which can guide predictions in different temporal and spatial contexts. To this end, we propose a new deep neural network model called Personal Dynamic Preference Neural Network (PDPNN). The core of the PDPNN model includes two parts: one part learns the user’s personal long-term preferences from the historical trajectories, and the other part learns the user’s short-term preferences from the current trajectory. By introducing a similarity function that evaluates the similarity between spatiotemporal contexts of user’s current trajectory and historical trajectories, PDPNN learns the user’s personal dynamic preference from user’s long-term and short-term preferences.
We propose a new approach to better model dynamic user preference for their next POI from their current trajectory in conjunction with their trajectory context. Based on the attention-enhanced Long Short-Term Memory (LSTM) neural network model, we build a new neural network model named PDPNN for next POI recommendation.
To accelerate training of trajectory contexts, we propose inner epoch cache mechanism to store the trajectory context output states during the process of model training. This reduces the complexity of the user’s personal dynamic preference module from \(O(n^2)\) to O(n) in each training epoch.
We conducted experiments on three real world datasets and the results show that our model outperforms current well known methods.
2 Related Work
Plenty of approaches have been proposed that focus on sequential data analysis and recommendation. Collaborative filtering based models, such as Matrix Factorization (MF) [3] and Tensor Factorization(TF) [4] are widely used for recommendation. These methods aim to cope with data sparsity and the cold start problem by capturing common short-term preferences among users, but cannot capture the user’s personal long-term preferences. Other existing studies employ the properties of a Markov chain to capture sequential patterns, such as Markov Chain (MC) [1] and Factorizing Personalized Markov Chains (FPMC) [2]. FPMC models user’s preference and sequential information jointly by combining factorization method and Markov Chains for next-basket recommendation. However, both MC and FPMC methods fall short in learning the long-term preference and the periodicity of the user’s movement.
Recently, as a result of the success of deep learning in speech recognition, vision and natural language processing, Recurrent Neural Networks (RNNs) have been widely used in sequential item recommendation [7, 8, 9, 10, 11]. Spatial Temporal Recurrent Neural Networks (STRNN) [8] utilizes a RNN architecture and linear interpolation to learn the regularity of sequential POI transition. However, traditional RNNs suffer from the issues of vanishing gradients and error propagation when they learn long-term dependencies [13]. Therefore, special gating mechanisms such as Long Short-Term Memory network (LSTM) [14] have been developed and widely used in recent work [9, 10, 11]. By controlling access to memory cells, LSTMs can alleviate the problem of long-term dependencies.
Attention mechanism is a key advancement in deep learning in recent years, and it shows a promising performance improvement for RNNs [15, 16, 17]. By introducing the attention mechanism, Attention-based Spatio-Temporal LSTM network (ATST-LSTM) [9] can focus on the relevant historical check-in records in a check-in sequence selectively using the spatiotemporal contextual information. However, these models are designed to learn short-term preferences and are not well suited to learning personal long-term preferences. They commonly add user embedding to RNN outputs, and reduce the problem of user context learning to the problem of learning a static optimal user embedding representation. It is more difficult to learn long-term preferences with simple attention mechanisms, especially for users with only a small set of historical trajectories.
3 Model Description
3.1 Problem Formulation
Let \(U=\{u_{1},\ldots ,u_{m} \}\) denote the user set and \(P= \{ p_{1} , \ldots , p_{j} \}\) denote the POI set, where \(\Vert U \Vert \) and \(\Vert P \Vert \) are the total numbers of users and POIs, respectively. Each POI \(p_{k}\) is associated with a geographic location \(l_k = ( l_{a},l_o\)), where \(l_{a}\) and \(l_o\) denote the latitude and longitude of the POI location. For a user \(u \in U\), a check-in behavior means \(u_{i}\) visits a POI \(p_{k}\) at time \(t_{k}\), which is denoted as tuple (\(u,p_{k}, t_{k}\)). Any check-in sequence with all the time intervals of successive check-ins less than a threshold value \(T_{delta}\), is called a trajectory. Obviously, a user’s historical check-ins will be segmented into many trajectories. We denote \(T_i^u=\{(u, p_{1},t_{1}),\ldots ,(u, p_{n},t_{n})\}\) as the i-th trajectory of the user u, with \(t_{k}-t_{k-1} \le T_{delta}\) for neighbor check-ins of the trajectory, and \(\Vert T_i^u \Vert \) is the total number of check-ins. All the historical trajectories of the user u are denoted as \(T^{u}=\{T_{1}^{u}, T_{2}^{u}, \ldots , T_{n}^{u} \}\).
When we are processing the i-th trajectory \(T_i^u\), we need to take into account the previous historical trajectories of user u, which is called the trajectory context. We denote \(C_{i}^{u} = \{ T_{1}^{u}, T_{2}^{u}, \ldots , T_{i-1}^{u} \}\) as the context of the current trajectory \(T_i^u\). When there is no ambiguity, we will omit the subscript u.
Formally, given a user u and all of their historical trajectories \(T^{u}=\{T_{1}^{u}, T_{2}^{u}, \ldots , T_{n}^{u} \}\), the problem of making the next POI recommendation is to predict the next POI \(p_{n+1}\) that the user is most likely to visit.
3.2 PDPNN Model
(I) Attention-enhanced recurrent neural network (ARNN) module: We use a LSTM network to capture short-term and long-term spatial-temporal sequence patterns. Additionally, an attention mechanism is introduced to capture the weight of all hidden states of the trajectory sequence. This component is used as an important part of the personal dynamic preference and the short-term preference learning module.
(II) Personal dynamic preference learning module: The user’s personal preferences is important for personalized POI recommendation, which is implicit in their historical trajectories. This module obtains a representation of personal dynamic preference from the user’s current trajectory and trajectory context \(C_i\). We first utilize the ARNN module to calculate the output hidden states of all trajectories in the trajectory context. Obviously, not every state can help to predict a POI, and the historical trajectories that are highly correlated with the current trajectory are supposed to have greater weight. To this end, we introduce attention mechanism into the module to measure the spatial temporal context similarity of the trajectory context and the current trajectory, and aggregate all the hidden states of the trajectory context as the personal dynamic preference representation.
(III) Short-term preference learning module: The short-term preference, which is usually referred to as sequential preference, is commonly learned from the current trajectory. We simply apply the ARNN module described above to get the hidden state of the current trajectory \(T_i\) .
(IV) Classifier: This is the final output component which unifies the user embedding output, the last hidden state from current trajectory and the aggregated attention from historical trajectories into a feature representation; after this it predicts the next POI. The output of this module is the probability vector of every POI that the user is likely to visit next. Cross-entropy loss and L2 regularization is used to measure the total loss in this module.
ARNN: ARNN is an attention-enhanced recurrent neural network, which receives a trajectory \(T^u\) as input and outputs the weighted sum of the hidden states of the user’s trajectory via a attention mechanism. The ARNN model consists of an embedding layer, a recurrent layer and an attention layer.
3.3 Model Training
Training the model with short-term preference and personal dynamic preference jointly is time-consuming, so we use pre-trained model and inner epoch cache method to accelerate the training process.
Pre-trained Model: The ARNN is the core component of the PDPNN model, and it takes most of the time required by the training process. In other words, if the ARNN component could be trained as quickly as possible, the training speed of the whole model would be accelerated. To this end, we try to pre-train the ARNN component in a more simple model, which connects the output of the ARNN and the classifier directly. Then, we load the pre-trained model and modify its network structure to adapt to our current model in training.
Inner Epoch Cache: Each input of the PDPNN model contains the current trajectory and the trajectory context, which makes the training process time consuming. For a certain user u with n trajectories {\(T_1, T_2,\ldots , T_n\)}, the PDPNN needs to process all of the n trajectories and the corresponding trajectory contexts. For each trajectory \(T_i\) , the PDPNN learns a short-term preference from \(T_i\), and learns the user’s personal dynamic preference from its trajectory context \(C_i=\{T_1, T_2, \ldots , T_{i-1}\}\), which means the PDPNN needs to calculate the output state of \(C_i\) with ARNN component for i times. So, the whole training complexity of the n trajectories user u is \(O(n^2)\).
As a part of trajectory contexts of \(T_{i+1}, T_{i+2},\ldots , T_n\), the i-th trajectory \(T_i\) is repeatedly calculated for \(n-i\) times during each training epoch. The optimization we can intuitively think of is to eliminate the repeated calculations. However, trajectory context state values are updated during training synchronously, which means we can not simply cache the state value in the whole training process. Considering the increment of state value updates between batches is very small in each training epoch, if the trajectory context output states can be cached within one training epoch and updated between epochs, a large number of approximately repeated calculations can be eliminated. To this end, we apply the inner epoch cache to store the trajectory context output state value during model training, and the calculation of each user’s personal dynamic preference can be reduced to linear complexity.
4 Experiment Analysis and Evaluation
4.1 Experiment Settings
The statistics of datasets.
Dataset | #Users | #check-ins | #location | #trajectories |
---|---|---|---|---|
LA | 1,083 | 22,7428 | 38,333 | 31,941 |
TKY | 2,293 | 57,3703 | 61,858 | 66,663 |
CA | 4,163 | 48,3805 | 2,9529 | 47,276 |
RNN [7]: This is a basic method for POI prediction, which has been successfully applied in word embedding and ad click prediction.
AT-RNN: This method empowers the RNN model with an attention mechanism, which has been successfully applied in machine translation and vision processing.
LSTM [14]: This is a variant of the RNN model, which contains a memory cell and three multiplicative gates to allow long-term dependency learning.
ATST-LSTM [9]: This is a state-of-the-art method for POI prediction, which applies an attention mechanism to a LSTM network.
PDPNN: This is our approach, which learns a user’s personal dynamic preference base on the current spatial and temporal context.
Prediction performance comparison on three dataset.
Dataset | Method | Recall@1 | Recall@5 | Recall@10 | Recall@20 |
---|---|---|---|---|---|
LA | RNN | 0.7901% | 3.0536% | 5.2317% | 9.4597% |
AT-RNN | 1.9218% | 7.4098% | 14.4352% | 24.685% | |
LSTM | 6.8332% | 25.71% | 39.0562% | 50.5659% | |
ATST-LSTM | 7.9650% | 26.009% | 39.9317% | 51.5909% | |
PDPNN | 8.5629% | 28.2084% | 41.0634% | 52.0606% | |
TKY | RNN | 2.5639% | 7.2056% | 9.4411% | 11.8282% |
AT-RNN | 4.0859% | 16.8172% | 26.8203% | 36.9498% | |
LSTM | 2.6271% | 7.2182% | 9.8958% | 13.5965% | |
ATST-LSTM | 4.4143% | 18.5475 | 28.9233% | 38.8317% | |
PDPNN | 4.8058% | 18.5665% | 28.2918% | 37.0445% | |
CA | RNN | 0.5711% | 2.4112% | 4.3993% | 8.0795% |
AT-RNN | 6.5144% | 17.8299% | 24.6616% | 32.0643% | |
LSTM | 2.0093% | 7.0008% | 10.7445% | 15.4399% | |
ATST-LSTM | 6.6836% | 17.7453% | 24.4078% | 32.1489% | |
PDPNN | 7.1912% | 18.6548% | 25.5499% | 32.9315% |
Metrics: To evaluate the performance of all methods for the POI recommendation problem, we employ a commonly used metric known as recall@N. The recall@N metric is popular in ranking tasks, which evaluates where the ground truth next POI appears in a ranked prediction list. A larger metric value indicates better performance.
4.2 Comparison of Recommendation Performance
We conducted experiments on a machine with Intel a Xeon CPU and a NVIDIA Tesla P4 GPU. The performance comparison of methods on three datasets is illustrated in Table 2. The RNN has a lower baseline performance than other methods with no extra optimization. The AT-RNN improves the performance greatly by 2 or 3 times over the RNN on all three datasets, which shows that the attention mechanism can alleviate the problem of long-term dependencies with gradient descent of RNN. The LSTM outperforms the RNN largely because of its memory and forget gate design, but fails to outperform the AT-RNN for the TKY and CA datasets. ATST-LSTM turns out to be the strongest baseline and shows significant improvement on the TKY and CA datasets. The reason is that ATST-LSTM works better at learning long-term dependencies through the gate and attention mechanisms. This shows that the attention mechanism is an effective supplement to LSTM models.
4.3 Inner Epoch Cache Evaluation
To evaluate the efficiency of the inner epoch cache, we compare the average time consumption of models with and without the inner epoch cache during one epoch training. Figure 2 shows that the PDPNN with an inner epoch cache trains 45\(\times \), 38\(\times \) and 30\(\times \) faster than the model without a cache on LA, TKY and CA dataset, respectively. By introducing the inner epoch cache optimization in training, the per-epoch training time is reduced from more than one day to one hour on the TKY data set. It shows that the inner epoch cache can greatly accelerate the model training process.
5 Conclusion
In this paper, we have proposed a new approach to better model the user’s personal dynamic preference of next POI from current trajectory supplemented with a trajectory context. Based on an attention empowered LSTM neural network, we build a new neural network model named PDPNN for next POI recommendation. Moreover, to accelerate the training of trajectory contexts, we proposed an inner epoch cache to store the trajectory context output state value during model training, and reduce the complexity of the user’s personal dynamic preference module from \(O(n^2)\) to O(n) in each training epoch. We conduct experiments on three real world data set and show that our model outperforms current well-known methods.
Notes
Acknowledgment
This work was supported by Beijing Municipal Science & Technology Commission (Z191100007119003).
References
- 1.Gambs, S., Killijian, M.-O., del Prado Cortez, M.N.: Next place prediction using mobility markov chains. In: MPM (2012)Google Scholar
- 2.Steffen, R., Christoph, F., Lars, S.T.: Factorizing personalized Markov chains for nextbasket recommendation. In: WWW 811–820 (2010)Google Scholar
- 3.Yehuda Koren; Robert Bell; and Chris Volinsky: Matrix factorization techniques for recommender systems. IEEE Comput. 42(8), 30–37 (2009)CrossRefGoogle Scholar
- 4.Xiong, L., Chen, X., Huang, T.-K., Schneider, J., Carbonell, J.G.: Temporal collaborative filtering with bayesian probabilistic tensor factorization. In: SDM, pp. 211–222 (2010)Google Scholar
- 5.Lian, D., Zhao, C., Xie, X., Sun, G., Chen, E., Yong, R.: GeoMF, joint geographical modeling and matrix factorization for point-of-interest recommendation. In: KDD (2014)Google Scholar
- 6.Zheng, V.W., Cao, B., Zheng, Y., Xie, X., Yang, Q.: Collaborative filtering meets mobile recommendation, a user-centered approach. In: AAAI (2010)Google Scholar
- 7.Zhang, Y., et al.: Sequential click prediction for sponsored search with recurrent neural networks. In: AAAI, pp. 1369–1376 (2014)Google Scholar
- 8.Liu. Q., Wu, S., Wang, L., Tan, T.: Predicting the next location : a recurrent model with spatial and temporal contexts. In: AAAI 2016 (2016)Google Scholar
- 9.Liwei, H., Yutao, M., Shibo, W., Yanbo, L.: An attention-based spatio-temporal LSTM network for next POI recommendation. In: IEEE Transactions on Services Computing, p. 1 (2019). https://doi.org/10.1109/TSC.2019.2918310
- 10.Zhao, P., Zhu, H., Liu, Y., Li, Z., Xu, J., Victor, S.: Where to go next: a spatio-temporal LSTM model for next POI recommendation. In: AAAI 2019 (2019)Google Scholar
- 11.Yao, D., Zhang, C., Huang, J., Bi, J.: SERM: a recurrent model for next location prediction in semantic trajectories. In: CIKM (2017)Google Scholar
- 12.Jannach, D., Lerche, L., Jugovac, M.: Adaptation and evaluation of recommendations for short-term shopping goals. In: RecSys, Adaptation and evaluation of recommendations for short-term shopping goals. pp. 211–218 (2015)Google Scholar
- 13.Bengio, Y., Frasconi, P., Simard, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5(2), 157–166 (1994)CrossRefGoogle Scholar
- 14.Schuster, M., Paliwal, K.P.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45(11), 2673–2681 (1997)CrossRefGoogle Scholar
- 15.Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., Hovy, E.: Hierarchical attention networks for document classification. In: Proceedings of NAACL (2016)Google Scholar
- 16.Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: Proceedings of ICLR (2015)Google Scholar
- 17.Vaswani, A., et al.: Attention is all you need. In: Proceedings of NIPS (2017)Google Scholar
- 18.Yang, D., Zhang, D., Zheng, V.W., Yu, Z.: Modeling user activity preference by leveraging user spatial temporal characteristics in LBSNs. IEEE Trans. Syst. Man Cybern. Syst. (TSMC) 45(1), 129–142 (2015)CrossRefGoogle Scholar
- 19.Yang, D., Zhang, D., Qu, B.: Participatory cultural mapping based on collective behavior data in location based social networks. ACM Trans. Intell. Syst. Technol. (TIST) 7(3), 30 (2016)Google Scholar