Adaptive Online First-Order Monitoring

  • Joshua SchneiderEmail author
  • David Basin
  • Frederik Brix
  • Srđan KrstićEmail author
  • Dmitriy TraytelEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11781)


Online first-order monitoring is the task of detecting temporal patterns in streams of events carrying data. Considerable research has been devoted to scaling up monitoring using parallelization by partitioning events based on their data values and processing the partitions concurrently. To be effective, partitioning must account for the event stream’s statistics, e.g., the relative event frequencies, and these statistics may change rapidly. We develop the first parallel online first-order monitor capable of adapting to such changes. A central problem we solve is how to manage and exchange states between the parallel executing monitors. To this end, we develop state exchange operations and prove their correctness. Moreover, we extend the implementation of the MonPoly monitoring tool with these operations, thereby supporting parallel adaptive monitoring, and show empirically that adaptation can yield up to a tenfold improvement in run-time.



Christian Fania helped us implement and evaluate our adaptive monitoring framework. The anonymous reviewers gave numerous helpful comments on earlier drafts of this paper. Joshua Schneider is supported by the US Air Force grant “Monitoring at Any Cost” (FA9550-17-1-0306). Srđan Krstić is supported by the Swiss National Science Foundation grant “Big Data Monitoring” (167162).


  1. 1.
    Afrati, F.N., Ullman, J.D.: Optimizing multiway joins in a map-reduce environment. IEEE Trans. Knowl. Data Eng. 23(9), 1282–1298 (2011)CrossRefGoogle Scholar
  2. 2.
    Alexandrov, A., et al.: The Stratosphere platform for big data analytics. VLDB J. 23(6), 939–964 (2014)CrossRefGoogle Scholar
  3. 3.
    Alur, R., Mamouras, K., Stanford, C.: Modular quantitative monitoring. PACMPL 3(POPL), 50:1–50:31 (2019)Google Scholar
  4. 4.
    Aniello, L., Baldoni, R., Querzoni, L.: Adaptive online scheduling in Storm. In: DEBS 2013, pp. 207–218. ACM (2013)Google Scholar
  5. 5.
    Barre, B., Klein, M., Soucy-Boivin, M., Ollivier, P.-A., Hallé, S.: MapReduce for parallel trace validation of LTL properties. In: Qadeer, S., Tasiran, S. (eds.) RV 2012. LNCS, vol. 7687, pp. 184–198. Springer, Heidelberg (2013). Scholar
  6. 6.
    Basin, D., Caronni, G., Ereth, S., Harvan, M., Klaedtke, F., Mantel, H.: Scalable offline monitoring of temporal specifications. Form. Methods Syst. Des. 49(1–2), 75–108 (2016)CrossRefGoogle Scholar
  7. 7.
    Basin, D., Klaedtke, F., Müller, S., Zălinescu, E.: Monitoring metric first-order temporal properties. J. ACM 62(2), 15:1–15:45 (2015)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Basin, D., Klaedtke, F., Zălinescu, E.: The MonPoly monitoring tool. In: Reger, G., Havelund, K. (eds.) RV-CuBES 2017. Kalpa Publications in Computing, vol. 3, pp. 19–28. EasyChair (2017)Google Scholar
  9. 9.
    Basin, D., Klaedtke, F., Zălinescu, E.: Greedily computing associative aggregations on sliding windows. Inf. Process. Lett. 115(2), 186–192 (2015)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Beame, P., Koutris, P., Suciu, D.: Communication steps for parallel query processing. J. ACM 64(6), 40:1–40:58 (2017)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Bersani, M.M., Bianculli, D., Ghezzi, C., Krstić, S., Pietro, P.S.: Efficient large-scale trace checking using MapReduce. In: Dillon, L.K., Visser, W., Williams, L. (eds.) ICSE 2016. pp. 888–898. ACM (2016)Google Scholar
  12. 12.
    Bianculli, D., Ghezzi, C., Krstić, S.: Trace checking of metric temporal logic with aggregating modalities using MapReduce. In: Giannakopoulou, D., Salaün, G. (eds.) SEFM 2014. LNCS, vol. 8702, pp. 144–158. Springer, Cham (2014). Scholar
  13. 13.
    Cormode, G., Garofalakis, M.N., Haas, P.J., Jermaine, C.: Synopses for massive data: samples, histograms, wavelets, sketches. Found. Trends Databases 4(1–3), 1–294 (2012)zbMATHGoogle Scholar
  14. 14.
    Elseidy, M., Elguindy, A., Vitorovic, A., Koch, C.: Scalable and adaptive online joins. PVLDB 7(6), 441–452 (2014)Google Scholar
  15. 15.
    Hallé, S., Soucy-Boivin, M.: MapReduce for parallel trace validation of LTL properties. J. Cloud Comput. 4(1), 8 (2015)CrossRefGoogle Scholar
  16. 16.
    Hoffmann, M., Lattuada, A., McSherry, F., Kalavri, V., Liagouris, J., Roscoe, T.: Megaphone: latency-conscious state migration for distributed streaming dataflows. PVLDB 12(9), 1002–1015 (2019)Google Scholar
  17. 17.
    Kalavri, V., Liagouris, J., Hoffmann, M., Dimitrova, D.C., Forshaw, M., Roscoe, T.: Three steps is all you need: fast, accurate, automatic scaling decisions for distributed streaming dataflows. In: Arpaci-Dusseau, A.C., Voelker, G. (eds.) OSDI 2018, pp. 783–798. USENIX Association (2018)Google Scholar
  18. 18.
    Kolchinsky, I., Schuster, A.: Efficient adaptive detection of complex event patterns. PVLDB 11(11), 1346–1359 (2018)Google Scholar
  19. 19.
    Mamouras, K., Raghothaman, M., Alur, R., Ives, Z.G., Khanna, S.: StreamQRE: modular specification and efficient evaluation of quantitative queries over streaming data. In: Cohen, A., Vechev, M.T. (eds.) PLDI 2017, pp. 693–708. ACM (2017)CrossRefGoogle Scholar
  20. 20.
    Mayer, R., Tariq, M.A., Rothermel, K.: Minimizing communication overhead in window-based parallel complex event processing. In: DEBS 2017, pp. 54–65. ACM (2017)Google Scholar
  21. 21.
    Murray, D.G., McSherry, F., Isaacs, R., Isard, M., Barham, P., Abadi, M.: Naiad: a timely dataflow system. In: Kaminsky, M., Dahlin, M. (eds.) SOSP 2013, pp. 439–455. ACM (2013)Google Scholar
  22. 22.
    Reger, G., Rydeheard, D.: From first-order temporal logic to parametric trace slicing. In: Bartocci, E., Majumdar, R. (eds.) RV 2015. LNCS, vol. 9333, pp. 216–232. Springer, Cham (2015). CrossRefGoogle Scholar
  23. 23.
    Roşu, G., Chen, F.: Semantics and algorithms for parametric monitoring. Log. Methods Comput. Sci. 8(1:9), 1–47 (2012)Google Scholar
  24. 24.
    Schneider, J., Basin, D., Brix, F., Krstić, S., Traytel, D.: Scalable online first-order monitoring. In: Colombo, C., Leucker, M. (eds.) RV 2018. LNCS, vol. 11237, pp. 353–371. Springer, Cham (2018). Scholar
  25. 25.
    Schneider, J., Basin, D., Brix, F., Krstić, S., Traytel, D.: Artifact associated with this paper (2019).
  26. 26.
    Schneider, J., Basin, D., Krstić, S., Traytel, D.: A formally verified monitor for metric first-order temporal logic. In: Finkbeiner, B., Mariani, L. (eds.) RV 2019. LNCS, Springer (2019, to appear).
  27. 27.
    Shah, M.A., Hellerstein, J.M., Chandrasekaran, S., Franklin, M.J.: Flux: an adaptive partitioning operator for continuous query systems. In: Dayal, U., Ramamritham, K., Vijayaraman, T.M. (eds.) ICDE 2003, pp. 25–36. IEEE (2003)Google Scholar
  28. 28.
    Vitorovic, A., et al.: Squall: Scalable real-time analytics. PVLDB 9(13), 1553–1556 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Institute of Information Security, Department of Computer ScienceETH ZürichZurichSwitzerland

Personalised recommendations