Advertisement

Modeling Marked Temporal Point Process Using Multi-relation Structure RNN

  • Hongyun Cai
  • Thanh Tung Nguyen
  • Yan Li
  • Vincent W. ZhengEmail author
  • Binbin Chen
  • Gao Cong
  • Xiaoli Li
Article
  • 49 Downloads

Abstract

Event sequences with marker and timing information are available in a wide range of domains, from machine log in automatic train supervision systems to information cascades in social networks. Given the historical event sequences, predicting what event will happen next and when it will happen can benefit many useful applications, such as maintenance service schedule for mass rapid transit trains and product advertising in social networks. Temporal point process (TPP) is one effective solution to solve the next event prediction problem due to its capability of capturing the temporal dependence among events. The recent recurrent temporal point process (RTPP) methods exploited recurrent neural network (RNN) to get rid of the parametric form assumption in the density functions of TPP. However, most existing RTPP methods focus only on the temporal dependence among events. In this work, we design a novel multi-relation structure RNN model with a hierarchical attention mechanism to capture not only the conventional temporal dependencies but also the explicit multi-relation topology dependencies. We then propose an RTPP algorithm whose density function conditioned on the event sequence embedding learned from our RNN model for cognitively predict the next event marker and time. The experiments show that our proposed MRS-RMTPP outperforms the state-of-the-art baselines in terms of both event marker prediction and event time prediction on three real-world datasets. The capability of capturing both ontology relation structure and temporal structure in the event sequences is of great importance for the next event marker and time prediction.

Keywords

Structure RNN Recurrent temporal point process Multi-layer attention 

Notes

Funding Information

This study was financially supported in part by the National Research Foundation (NRF), Prime Minister’s Office, Singapore, under its National Cybersecurity R&D Programme (Award No. NRF2014NCR-NCR001-31) and administered by the National Cybersecurity R&D Directorate. This research is also financially supported in part by the National Research Foundation, Prime Ministers Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) program.

Compliance with Ethical Standards

Conflict of Interests

This research was mainly conducted when Hongyun Cai and Vincent W. Zheng worked at Advanced Digital Sciences Center. This research was supported in part by the National Research Foundation (NRF), Prime Ministers Office, Singapore, under its National Cybersecurity R&D Programme (Award No. NRF2014NCR-NCR001-31) and administered by the National Cybersecurity R&D Directorate. It is also supported in part by the National Research Foundation, Prime Ministers Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.

References

  1. 1.
    Dong H, Ning B, Cai B, Hou Z. Automatic train control system development and simulation for high-speed railways. IEEE Circuits Syst Mag 2010;10(2):6–18.CrossRefGoogle Scholar
  2. 2.
    Wang H, Feng R, Leung AC, Tsang KF. Lagrange programming neural network approaches for robust time-of-arrival localization. Cogn Comput 2018;10(1):23–34.CrossRefGoogle Scholar
  3. 3.
    Zhang H, Wu L, Song Y, Su C, Wang Q, Su F. An online sequential learning non-parametric value-at-risk model for high-dimensional time series. Cogn Comput 2018;10(2):187–200.CrossRefGoogle Scholar
  4. 4.
    Wang J, Zheng VW, Liu Z, Chang KC. Topological recurrent neural network for diffusion prediction. ICDM; 2017.Google Scholar
  5. 5.
    Daley DJ, Vere-Jones D. 2008. An introduction to the theory of point processes, vol II. 2nd ed. Probability and its applications (New York), General theory and structure.Google Scholar
  6. 6.
    Kingman JFC, Vol. 3. Poisson processes. Oxford: Oxford University Press; 1993.Google Scholar
  7. 7.
    HAWKES AG. Spectra of some self-exciting and mutually exciting point processes. Biometrika 1971;58(1):83–90.CrossRefGoogle Scholar
  8. 8.
    Isham V, Westcott M. A self-correcting point process. Adv Appl Probab 1979;37:629–46.Google Scholar
  9. 9.
    Engle R, Duration RJR. Autoregressive conditional a new model for irregularly spaced transaction data. Econometrica 1998;66(5):1127–62.CrossRefGoogle Scholar
  10. 10.
    Grossberg S. REcurrent neural networks. Scholarpedia 2013;8 (2):1888.CrossRefGoogle Scholar
  11. 11.
    Du N, Dai H, Trivedi R, Upadhyay U, Gomez-Rodriguez M, Song L. Recurrent marked temporal point processes: embedding event history to vector. KDD; 2016. p. 1555–64.Google Scholar
  12. 12.
    Xiao S, Yan J, Yang X, Zha H, Chu SM. Modeling the intensity function of point process via recurrent neural networks. AAAI; 2017. p. 1597–603.Google Scholar
  13. 13.
    Wang Y, Shen H, Liu S, Gao J, Cheng X. Cascade dynamics modeling with attention-based recurrent neural network. IJCAI; 2017. p. 2985–91.Google Scholar
  14. 14.
    Li Y, Yang L, Xu B, Wang J, Lin H. Improving user attribute classification with text and social network attention. Cogn Comput 2019;11(4):459–68.CrossRefGoogle Scholar
  15. 15.
    Ma Y, Peng H, Khan T, Cambria E, Hussain A. Sentic LSTM: a hybrid network for targeted Aspect-Based sentiment analysis. Cogn Comput 2018;10(4):639–50.CrossRefGoogle Scholar
  16. 16.
    Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput 1997;9(8):1735–80.CrossRefGoogle Scholar
  17. 17.
    Cho K, van Merriënboer B, Gülçehre Ç, Bahdanau D, Bougares F, Schwenk H, et al. Learning phrase representations using RNN encoder–decoder for statistical machine translation. EMNLP; 2014. p. 1724–34.Google Scholar
  18. 18.
    Yang H, Cheung LP. Implicit heterogeneous features embedding in deep knowledge tracing. Cogn Comput 2018;10(1):3–14.CrossRefGoogle Scholar
  19. 19.
    Lauren P, Qu G, Yang J, Watta P, Huang G, Lendasse A. Generating word embeddings from an extreme learning machine for sentiment analysis and sequence labeling tasks. Cogn Comput 2018;10(4):625–38.CrossRefGoogle Scholar
  20. 20.
    Tai KS, Socher R, Manning CD. Improved semantic representations from tree-structured long short-term memory networks. ACL; 2015. p. 1556–66.Google Scholar
  21. 21.
    Zheng J, Cai F, Chen W, Feng C, Chen H. Hierarchical neural representation for document classification. Cogn Comput 2019;11(2):317–27.CrossRefGoogle Scholar
  22. 22.
    Zhang B, Yin X. SSDM2: a two-stage semantic sequential dependence model framework for biomedical question answering. Cogn Comput 2018;10(1):73–83.CrossRefGoogle Scholar
  23. 23.
    Shuai B, Zuo Z, Wang B, Wang G. DAG-recurrent neural networks for scene labeling. CVPR; 2016. p. 3620–29.Google Scholar
  24. 24.
    Liu Z, Zheng VW, Zhao Z, Zhu F, Chang KC, Wu M, et al. Semantic proximity search on heterogeneous graph by proximity embedding. AAAI; 2017. p. 154–60.Google Scholar
  25. 25.
    Liu Z, Zheng VW, Zhao Z, Yang H, Chang KCC, Wu M, et al. Subgraph-augmented path embedding for semantic user search on heterogeneous social network. WWW; 2018.Google Scholar
  26. 26.
    Brillinger DR, Guttorp PM, Schoenberg FP. . Point processes, temporal. American Cancer Society; 2013.Google Scholar
  27. 27.
    Aalen O, Borgan O, Gjessing H. 2008. Survival and event history analysis: a process point of view statistics for biology and health.Google Scholar
  28. 28.
    Mei H, Eisner J. The neural Hawkes process: a neurally self-modulating multivariate point process. NIPS; 2017.Google Scholar
  29. 29.
    Song H, Rajan D, Thiagarajan JJ, Spanias A. Attend and diagnose: clinical time series analysis using attention models. AAAI; 2018. p. 4091–98.Google Scholar
  30. 30.
    Ma F, Chitta R, Zhou J, You Q, Sun T, Gao J. Dipole: diagnosis prediction in healthcare via attention-based bidirectional recurrent neural networks. KDD; 2017. p. 1903–11.Google Scholar
  31. 31.
    Xiao S, Yan J, Yang X, Zha H, Chu SM. Modeling the intensity function of point process via recurrent neural networks. AAAI; 2017. p. 1597–1603.Google Scholar
  32. 32.
    Press WH, Teukolsky SA, Vetterling WT, Flannery BP. Numerical recipes in C. Cambridge: Cambridge University Press; 1992.Google Scholar
  33. 33.
    Werbos PJ. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE 1990;78 (10):1550–60.CrossRefGoogle Scholar
  34. 34.
    Kingma DP, Ba J. 2014. Adam: a method for stochastic optimization. CoRR arXiv:1412.6980.
  35. 35.
    Du N, Dai H, Trivedi R, Upadhyay U, Gomez-Rodriguez M, Song L. Recurrent marked temporal point processes: embedding event history to vector. 1555–64; 2016.Google Scholar
  36. 36.
    Hodas NO, Lerman K. 2013. The simple rules of social contagion. CoRR arXiv:1308.5015.
  37. 37.
    Leskovec J, Backstrom L, Kleinberg J. 2009. Meme-tracking and the dynamics of the news cycle.Google Scholar
  38. 38.
    Team TD. 2016. Theano: a python framework for fast computation of mathematical expressions. CoRR arXiv:1605.02688.
  39. 39.
    Hyndman RJ, Koehler AB. Another look at measures of forecast accuracy. International Journal of Forecasting 2006;22(4):679–88.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.TencentShenzhenChina
  2. 2.Nanyang Technological UniversitySingaporeSingapore
  3. 3.Advanced Digital Sciences CenterSingaporeSingapore
  4. 4.WeBankShenzhenChina
  5. 5.Singapore University of Technology and Design & Advanced Digital Sciences CenterSingaporeSingapore

Personalised recommendations