Cluster Computing

, Volume 12, Issue 1, pp 59–72 | Cite as

Tracking transaction footprints for non-intrusive end-to-end monitoring

  • Bikram Sengupta
  • Nilanjan Banerjee
  • Chatschik Bisdikian
  • Paul Hurley


Existing transaction monitoring solutions are either platform-specific or rely on instrumentation techniques, which limit their applicability. Consequently, transaction monitoring in enterprise environments often involves the manual collation of information spread across a variety of infrastructure elements and applications, and is a time-consuming and labor-intensive task. To address this problem, we have developed an online, non-intrusive and platform-agnostic solution for transaction monitoring. The solution includes a transaction model discovery component that leverages historical system log files, containing transaction footprints and generates a model of the transaction in terms of valid sequence of steps that a transaction instance may execute and the expected footprint patterns at each step. The online monitoring system, in turn, takes in only (a) online system log files and (b) the transaction model, as inputs and generates a dynamic execution profile of ongoing transaction instances that allows their status to be tracked at individual and aggregate levels, even when transaction footprints do not necessarily carry correlating identifiers as those injected through instrumentation. In this paper, we describe the transaction model discovery and monitoring system including the architecture and algorithms, followed by results from an empirical study, ongoing work on run-time model validation and directions for future research.


Transactions Monitoring Log analysis Correlation Footprints 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
  2. 2.
    Agrawal, R., Gunopulos, D., Leymann, F.: Mining process models from workflow logs. Int’l Conf. on Extending Database Technology. LNCS, vol. 1377 (1998) Google Scholar
  3. 3.
  4. 4.
    Chen, M.Y., Kiciman, E., Fratkin, E., Fox, A., Brewer, E.: Pinpoint: problem determination in large, dynamic internet services. In Proc. of DSN, pp. 595–604 (2002) Google Scholar
  5. 5.
  6. 6.
    Krishnamurthy, B., Sengupta, B., Neogi, A., Singh, R.: Data tagging architecture for system monitoring in dynamic environments. NOMS 2008 (main technical track) Google Scholar
  7. 7.
    Vaardi, R.: A data clustering algorithm for mining patterns from event logs. In 3rd Workshop on IP Operations and Management (IPOM 2003), Kansas City, MO, USA, Oct. 1–3, 2003 Google Scholar
  8. 8.
  9. 9.
    Google Web Toolkit.
  10. 10.
    GraphML File Format.
  11. 11.
    Logfile Clustering Tool.
  12. 12.
    Cook, J.E., Wolf, A.L.: Discovering models of software processes from event-based data. ACM Trans. Softw. Eng. Methodol. 7(3), 215–249 (1998) CrossRefGoogle Scholar
  13. 13.
    Herbst, J., Karagiannis, D.: An inductive approach to the acquisition and adaptation of workflow models. IJCAI Workshop on Intelligent Workflow and Process Management, pp. 52–57 (1999) Google Scholar
  14. 14.
    van der Aalst, W.M.P., van Dongen, B.F., et al.: Workflow mining: a survey of issues and approaches. Data Knowl. Eng. 47(2), 237–267 (2003) CrossRefGoogle Scholar
  15. 15.
    Peng, W., Li, T., Ma, S.: Mining logs files for data-driven system management. SIGKDD Explor. 7, 44–51 (2005) CrossRefGoogle Scholar
  16. 16.
  17. 17.
    Hellerstein, J., Ma, S.: Mining event data for actionable patterns. The Computer Measurement Group, Orlando, December 2000 Google Scholar
  18. 18.
    Aman, J., Eilert, C.K., Emmes, D., Yocom, P., Dillenberger, D.: Adaptive algorithms for managing a distributed data processing workload. IBM Syst. J. 36, 2 (1997) CrossRefGoogle Scholar
  19. 19.
    Schmid, M., Thoss, M., et al.: A generic application-oriented performance instrumentation for multi-tier environments. Integr. Netw. Manag. 304–313 (2007) Google Scholar
  20. 20.
    Hasselmeyer, P.: Managing dynamic service dependencies. In Proc. of 12th International Workshop on Distributed Systems: Operations and Management, Nancy, France, pp. 141–150 (2001) Google Scholar
  21. 21.
    E2EProf: Automated end-to-end performance management for enterprise systems. In S. Agarwala, F. Alegre et al. (eds.) 37th Int’l Conf. on Dependable Systems and Networks (DSN), pp. 749–758 (2007) Google Scholar
  22. 22.
    Agarwal, M., Gupta, M., Kar, G., Neogi, A., Sailer, A.: Mining Activity Data for Dynamic Dependency Discovery in e-Business Systems. IEEE eTrans. Netw. Serv. Manag. J. (eTNSM), Fall 2004 Google Scholar
  23. 23.
    Brown, A., Kar, G., Keller, A.: An active approach to characterizing dynamic dependencies for problem determination in distributed environment. In Proc. of 7th IEEE/IFIP Int’l Symposium on Integrated Network Management, Seattle, pp. 377–390, May 2001 Google Scholar
  24. 24.
  25. 25.
  26. 26.
  27. 27.
  28. 28.
  29. 29.
    Branch, J.W., Bisdikian, C., (Starsky) Wong, H.Y., Agrawal, D.: A semantically agnostic framework supporting model composition for business management solutions. In IFIP/IEEE International Symposium on Integrated Network Management (IM 2009), June 2009 Google Scholar
  30. 30.
    Anandkumar, A., Bisdikian, C., Agrawal, D.: Tracking in a spaghetti bowl: monitoring transactions using footprints. In ACM SIGMETRICS 2008, Annapolis, Maryland, USA, June 2–6, 2008 Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  • Bikram Sengupta
    • 1
  • Nilanjan Banerjee
    • 1
    • 2
  • Chatschik Bisdikian
    • 3
  • Paul Hurley
    • 4
  1. 1.IBM India Research LabBangaloreIndia
  2. 2.IBM India Research LabNew DelhiIndia
  3. 3.IBM T.J.Watson Research CenterHawthorneUSA
  4. 4.IBM Zurich Research LabZurichSwitzerland

Personalised recommendations