Skip to main content
SpringerLink
Log in
Book cover

Workshop on Enterprise and Organizational Modeling and Simulation

EOMAS 2018: Enterprise and Organizational Modeling and Simulation pp 107–122Cite as

Enhanced Benchmark Datasets for a Comprehensive Evaluation of Process Model Matching Techniques

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 332))

Abstract

Process Model Matching (PMM) refers to the automatic identification of corresponding activities between a pair of process models. Recognizing the pivotal role of PMM in numerous application areas a plethora of matching techniques have been developed. To evaluate the effectiveness of these techniques, researchers typically use PMMC’15 datasets and three well-established performance measures, precision, recall and F1 score. The performance scores of these measures are useful for a surface level evaluation of a matching technique. However, these overall scores do not provide essential insights about the capabilities of a matching technique. To that end, we enhance the PMMC’15 datasets by classifying corresponding pairs into three types and compute performance scores of each type, separately. We contend that the performance scores for each type of corresponding pairs, together with the surface level performance scores, provide valuable insights about the capabilities of a matching technique. As a second contribution, we use the enhanced datasets for a comprehensive evaluation of three prominent semantic similarity measures. Thirdly, we use the enhanced datasets for a comprehensive evaluation of the results of twelve matching systems from the PMM Contest 2015. From the results, we conclude that there is a need for developing the next generation of matching techniques that are equally effective for the three types of pairs.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    The results can be downloaded from https://ai.wu.ac.at/emisa2015/contest.php.

  2. 2.

    Gold standard refers to the benchmark correspondences generated by BPM experts.

  3. 3.

    The results can be downloaded from https://ai.wu.ac.at/emisa2015/contest.php.

References

  1. Kuss, E., Leopold, H., van der Aa, H., Stuckenschmidt, H., Reijers, H.A.: Probabilistic evaluation of process model matching techniques. In: Comyn-Wattiau, I., Tanaka, K., Song, I.-Y., Yamamoto, S., Saeki, M. (eds.) ER 2016. LNCS, vol. 9974, pp. 279–292. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46397-1_22

    Chapter  Google Scholar 

  2. Jabeen, F., Leopold, H., Reijers, Hajo A.: How to make process model matching work better? An analysis of current similarity measures. In: Abramowicz, W. (ed.) BIS 2017. LNBIP, vol. 288, pp. 181–193. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59336-4_13

    Chapter  Google Scholar 

  3. Rodríguez, C., Klinkmüller, C., Weber, I., Daniel, F., Casati, F.: Activity matching with human intelligence. In: La Rosa, M., Loos, P., Pastor, O. (eds.) BPM 2016. LNBIP, vol. 260, pp. 124–140. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-45468-9_8

    Chapter  Google Scholar 

  4. Awad, A., Polyvyanyy, A., Weske, M.: Semantic querying of business. Process models. In: Proceedings of the 12th IEEE International Conference on Enterprise Distributed Object Computing Conference (EDOC 2008), pp. 85–94, Munich, Germany (2008)

    Google Scholar 

  5. Dumas, M., García-Bañuelos, L., La Rosa, M., Uba, R.: Fast detection of exact clones in business process model repositories. Inf. Syst. 38(4), 619–633 (2012)

    Article  Google Scholar 

  6. La Rosa, M., Dumas, M., Uba, R., Dijkman, R.M.: Business process model merging: an approach to business process consolidation. ACM Trans. Softw. Eng. Methodol. 22(2), 11–42 (2012)

    Google Scholar 

  7. Meilicke, C., Leopold, H., Kuss, E.S., Reijers, H.: Overcoming individual process model matcher weaknesses using ensemble matching. Decis. Support Syst. 100(1), 15–26 (2017)

    Article  Google Scholar 

  8. Antunes, G., et al.: The process model matching contest 2015. In: Kolb, J., Leopold, H., Mendling, J. (eds.) Proceedings of the 6th International Workshop on Enterprise Modelling and Information Systems Architecture (EMISA 2015), Innsbruck, Austria. LNI, pp. 1–29. Springer, Heidelberg (2015)

    Google Scholar 

  9. Kuss, E., Leopold, H., Aa, H., Stuckenschmidt Reijers, H.A.: A probabilistic evaluation procedure for process model matching techniques. DKE J. (2018, in press)

    Google Scholar 

  10. Sonntag, A., Hake, P., Fettke, P., Loos, P.: An approach for semantic business process model matching using supervised machine learning. In: Proceedings of the 24th European Conference on Information Systems, pp. 1– 12. AIS, Istanbul (2016)

    Google Scholar 

  11. Clough, P., Gaizauskas, R., Piao, S., Wilks, Y.: METER: MEasuring TExt Reuse. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (ACL 2002), Philadelphia, USA, pp. 152–159 (2002)

    Google Scholar 

  12. Clough, P., Stevenson, M.: Developing a corpus of plagiarized short answers. Lang. Resour. Eval. 45(1), 5–24 (2011)

    Article  Google Scholar 

  13. Sameen, S., Sharjeel, M., Nawab, R.M.A., Rayson, P., Muneer, I.: Measuring short text reuse for the Urdu language. IEEE Access 6(1), 7412–7421 (2018)

    Article  Google Scholar 

  14. Xiao, C., Wang, W., Lin, X., Yu, J.X.: Efficient similarity joins for near duplicate detection. ACM Trans. Database Syst. 36(3), 1–15 (2011)

    Article  Google Scholar 

  15. Miller, A.G.: WordNet: a lexical database for English. Commun. ACM 38(11), 39–41 (1995)

    Article  Google Scholar 

  16. Patwardhan, S., Banerjee, S., Pedersen, T.: Using measures of semantic relatedness for word sense disambiguation. In: Proceedings of the 4th International Conference on Intelligent Text Processing and Computational Linguistics, Maxico City, Mexico, pp. 241–257 (2003)

    Google Scholar 

  17. Budanitsky, A., Hirst, G.: Evaluating WordNet-based measures of lexical semantic relatedness. Comput. Linguist. 32(1), 13–47 (2006)

    Article  Google Scholar 

  18. Lin, D.: An information-theoretic definition of similarity. In: Proceedings of the 15th International Conference on Machine Learning (ICML 1998), Madison, USA, pp. 296–304 (1998)

    Google Scholar 

  19. Lesk, M.: Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from a ice cream cone. In: Proceedings of the 5th Annual International Conference on Systems Documentation (SIGDOC 1986), Toronto, Canada, pp. 24–26 (1986)

    Google Scholar 

  20. Niemann, M., Siebenhaar, M., Schulte, S., Steinmetz, R.: Comparison and retrieval of process models using related cluster pairs. Comput. Ind. 63(2), 168–180 (2012)

    Article  Google Scholar 

  21. Sebu, M.L.: Similarity of business process models in a modular design. In: Proceedings of the Applied Computational Intelligence and Informatics (SACI 2016), Timisoara, Romania, pp. 31–36 (2016)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Software Development and Maintenance Center, University of Gujrat, Gujrat, Pakistan

    Muhammad Ali & Khurram Shahzad

  2. Punjab University College of Information Technology, University of the Punjab, Lahore, Pakistan

    Muhammad Ali & Khurram Shahzad

Authors
  1. Muhammad Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Khurram Shahzad
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Khurram Shahzad .

Editor information

Editors and Affiliations

  1. Czech Technical University in Prague, Prague, Czech Republic

    Robert Pergl

  2. Higher School of Economics, National Research University, Nizhny Novgorod, Russia

    Eduard Babkin

  3. Loughborough University, Loughborough, UK

    Russell Lock

  4. Higher School of Economics, National Research University, Nizhny Novgorod, Russia

    Pavel Malyzhenkov

  5. Czech Technical University in Prague, Prague, Czech Republic

    Vojtěch Merunka

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ali, M., Shahzad, K. (2018). Enhanced Benchmark Datasets for a Comprehensive Evaluation of Process Model Matching Techniques. In: Pergl, R., Babkin, E., Lock, R., Malyzhenkov, P., Merunka, V. (eds) Enterprise and Organizational Modeling and Simulation. EOMAS 2018. Lecture Notes in Business Information Processing, vol 332. Springer, Cham. https://doi.org/10.1007/978-3-030-00787-4_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-00787-4_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-00786-7

  • Online ISBN: 978-3-030-00787-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation