A Systematic Approach for Evaluating BPM Systems: Case Studies on Open Source and Proprietary Tools

  • Andrea DelgadoEmail author
  • Daniel Calegari
  • Pablo Milanese
  • Renatta Falcon
  • Esteban García
Conference paper
Part of the IFIP Advances in Information and Communication Technology book series (IFIPAICT, volume 451)


Business Process Management Systems (BPMS) provide support for modeling, developing, deploying, executing and evaluating business processes in an organization. Selecting a BPMS is not a trivial task, not only due to the many existing alternatives, both in the open source and proprietary realms, but also because it requires a thorough evaluation of its capabilities, contextualizing them in the organizational environment in which they will be used. In this paper we present a methodology to guide the systematic evaluation of BPMS that takes into account the specific needs of each organization. It provides a list of key characteristics of BPMS which are ranked by the organization and evaluated using test cases and quantitative criteria. We also present case studies of open source and proprietary BPMS evaluations following our proposal.


Business Process Management Systems (BPMS) Open source and proprietary BPMS Evaluation methodology Systematic approach 


  1. [1]
    Weske, M.: BPM: Concepts, Languages, Architectures. Springer (2007)Google Scholar
  2. [2]
    van der Aalst, W.M.P., ter Hofstede, A.H.M., Weske, M.: Business process management: A survey. In: van der Aalst, W.M.P., ter Hofstede, A.H.M., Weske, M. (eds.) BPM 2003. LNCS, vol. 2678, pp. 1–12. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  3. [3]
    Chang, J.F.: BPM Systems: strategy and implementation. Auerbach Publications, Taylor & Francis Group (2006)Google Scholar
  4. [4]
    OMG. Business Process Model And Notation (BPMN) v2.0 (2011)Google Scholar
  5. [5]
    WfMC, XML Process Definition Language (XPDL) (2008)Google Scholar
  6. [6]
    OASIS, WS Business Process Execution Language (WS-BPEL) (2007)Google Scholar
  7. [7]
    van der Aalst, W.M.P., ter Hofstede, A., Kiepuszewski, B., Barros, A.: Workflow patterns. Dist. & Parallel Databases 14(3) (2003)Google Scholar
  8. [8]
    Garcês, R., Jesus, T., Cardoso, J., Valente, P.: Open source workflow management systems: A concise survey. In: 2009 BPM & Workflow Handbook, Future Strategies Inc., pp. 179-190 (2009)Google Scholar
  9. [9]
  10. [10]
  11. [11]
    ISO/IEC 9126, Software engineering, Product quality, superseded by ISO/IEC 25000 SQuaRE.
  12. [12]
    Tsalgatidou, A.: Selection criteria for tools supporting business process transformation for electronic commerce. In: Proceedings of EURO-MED NET (1998)Google Scholar
  13. [13]
    Wohed, P., Andersson, B., ter Hofstede, A., Russell, N.C., van der Aalst, W.M.P.: Patterns-based Evaluation of Open Source BPM Systems: The Cases of jBPM, OpenWFE, and Enhydra Shark. Inf. Softw. Technol. 51(8), 1187–1216 (2009)CrossRefGoogle Scholar
  14. [14]
    Harrer, S., Lenhard, J., Wirtz, G.: Open source versus proprietary software in service-orientation: The case of BPEL engines. In: Basu, S., Pautasso, C., Zhang, L., Fu, X. (eds.) ICSOC 2013. LNCS, vol. 8274, pp. 99–113. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  15. [15]
    Morera, D: COTS evaluation using desmet methodology & analytic hierarchy process (AHP). In: Oivo, M., Komi-Sirviö, S. (eds.): PROFES 2002. LNCS vol. 2559, pp. 485-493. Springer, Heidelberg (2002)Google Scholar
  16. [16]
    Lawlis, P., Mark, K., Thomas, D., Courtheyn, T.: A Formal Process for Evaluating COTS Software Products. IEEE Comput. 34(5), 58–63 (2001)CrossRefGoogle Scholar
  17. [17]
    Taibi, D., Lavazza, L., Morasca, S.: OpenBQR: a framework for the assessment of OSS. In: OSS 2007, pp. 173-186 (2007)Google Scholar
  18. [18]
    Tarawneh, F., Baharom, F., Yahaya, J., Ahmad, F.: Evaluation and Selection COTS Software Process: The State of the Art. International Journal on New Computer Architectures and Their Applications (IJNCAA), 344-357Google Scholar
  19. [19]
    Richardson, C., Miers, D., Cullen, A., Keenan, J.: The Forrester Wave: BPM Suites, Q1 2013, How The Top 10 Vendors Stack Up For Next-Generation BPM Suites, March 2013Google Scholar
  20. [20]
    Sinur, J., Hill, J.: Magic Quadrant for Business Process Management Suites. Gartner Inc. (2010)Google Scholar
  21. [21]
    Sinur, J., Schulte, W., Hill, J., Jones, T.: Magic Quadrant for Intelligent Business Process Management Suites. Gartner Inc. (2012)Google Scholar
  22. [22]
    Yin, R.: Case Study Research: Design and Methods. Sage Publications, Inc. (2002)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2015

Authors and Affiliations

  • Andrea Delgado
    • 1
    Email author
  • Daniel Calegari
    • 1
  • Pablo Milanese
    • 1
  • Renatta Falcon
    • 1
  • Esteban García
    • 1
  1. 1.Instituto de Computación, Facultad de IngenieríaUniversidad de La RepúblicaMontevideoUruguay

Personalised recommendations