Towards a Method for the Evaluation of Reference Architectures: Experiences from a Case

  • Samuil Angelov
  • Jos J. M. Trienekens
  • Paul Grefen
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5292)


Reference architectures provide major guidelines for the structure of a class of information systems. Because of their fundamental role, reference architectures have to be of high quality. Before accepting a reference architecture, it has to go through a rigorous evaluation process. A number of methods exist for the evaluation of software architectures. In this paper, we analyze the main differences between concrete software architectures and reference architectures. We discuss the effects of these differences on the evaluation of reference architectures and show that existing methods cannot be directly applied for the evaluation of reference architectures. For the evaluation of a reference architecture for e-contracting systems, we used the Architecture Tradeoff Analysis Method with a number of adaptations and extensions. We present our approach and share our experiences from this evaluation process. Based on the analysis and our experiences gained, we present our vision for a method for the evaluation of reference architectures.


software architecture reference architecture evaluation method 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Angelov, S.: Foundations of B2B Electronic Contracting, PhD Thesis. Eindhoven University of Technology, Eindhoven (2006)Google Scholar
  2. 2.
    Angelov, S.: Defining E-Contracting and Measuring its Significance. In: Post-Conference Workshop at the 4th Annual Contract Management Conference, Dubai. Institute for International Research (2007)Google Scholar
  3. 3.
    Angelov, S.: Evaluation of the E-Contracting Reference Architecture. Technical report, Beta Working Paper, WP 225, Eindhoven University of Technology (2007)Google Scholar
  4. 4.
    Angelov, S., Grefen, P.: An E-contracting Reference Architecture. Journal of Systems and Software (to appear) (February 26, 2008)Google Scholar
  5. 5.
    Babar, M., Gorton, I.: Comparison of Scenario-Based Software Architecture Evaluation Methods. In: 11th Asia-Pacific Software Engineering Conference (APSEC 2004), pp. 600–607. IEEE Computer Society, Washington (2004)Google Scholar
  6. 6.
    Bass, L., Clements, P., Kazman, R.: Software Architecture in Practice, 2nd edn. Addison-Wesley Professional, Reading (2003)Google Scholar
  7. 7.
    Bengtsson, P., Bosch, J.: Scenario-Based Software Architecture Reengineering. In: Fifth International Conference on Software Reuse, 1998, Victoria, Canada, pp. 308–317 (1998)Google Scholar
  8. 8.
    Clements, P., Kazman, R., Klein, M.: Evaluating Software Architectures: Methods and Case Studies. Addison-Wesley Professional, Reading (2002)Google Scholar
  9. 9.
    Clements, P., Northrop, L.: Software Product Lines: Practices and Patterns. Addison-Wesley Professional, Reading (2001)Google Scholar
  10. 10.
    Dobrica, L., Niemelä, E.: A Survey on Software Architecture Analysis Methods. IEEE Transactions on Software Engineering 28(7), 638–653 (2002)CrossRefGoogle Scholar
  11. 11.
    Dolan, T.: Architecture Assessment of Information-System Families - A Practical Perspective, PhD Thesis. Eindhoven University of Technology, Eindhoven (2001)Google Scholar
  12. 12.
    Ferrara, F.: The Standard Healthcare Information Systems Architecture and the DHE middleware. International Journal of Medical Informatics 52(1), 39–51 (1998)CrossRefGoogle Scholar
  13. 13.
    Grefen, P., Remmerts de Vries, R.: A Reference Architecture for Workflow Management Systems. Data & Knowledge Engineering 27(1), 31–57 (1998)zbMATHCrossRefGoogle Scholar
  14. 14.
    Hollingsworth, D.: The Workflow Reference Model. Technical report, Workflow Management Coalition Documents, TC00-1003, Workflow Management Coalition (1995)Google Scholar
  15. 15.
    Bontempo, C., Zagelow, G.: The IBM Data Warehouse Architecture. Communications of the ACM 41(9), 38–48 (1998)CrossRefGoogle Scholar
  16. 16.
    Ionita, M., Hammer, D., Obbink, H.: Scenario-Based Software Architecture Evaluation Methods: An Overview. In: Workshop on Methods and Techniques for Software Architecture Review and Assessment at the International Conference on Software Engineering, Orlando, Florida, USA (2002)Google Scholar
  17. 17.
    Kusters, R., Solingen, R., Trienekens, J.: Identifying Embedded Software Quality: Two Approaches. Quality and Reliability Engineering International 15(6), 485–492 (1999)CrossRefGoogle Scholar
  18. 18.
    Norta, A.: Exploring Dynamic Inter-Organizational Business Process Collaboration, PhD Thesis. Eindhoven University of Technology, Eindhoven (2007)Google Scholar
  19. 19.
    Reed, P.: Reference Architecture: The Best of Best Practices (2002),
  20. 20.
    Rozanski, N., Woods, E.: Software Systems Architecture: Working With Stakeholders Using Viewpoints and Perspectives. Addison-Wesley Professional, Reading (2005)Google Scholar
  21. 21.
    Shaw, M., Garlan, D.: Software Architecture: Perspectives on an Emerging Discipline. Prentice-Hall, Englewood Cliffs (1996)zbMATHGoogle Scholar
  22. 22.
    Wu, H.: A Reference Architecture for Adaptive Hypermedia Applications, PhD Thesis. Eindhoven University of Technology, Eindhoven (2002)Google Scholar
  23. 23.
    Zimmermann, H.: OSI Reference Model - the IS0 Model of Architecture for Open Systems Interconnection. IEEE Transactions on Communications 28(4), 425–432 (1980)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Samuil Angelov
    • 1
  • Jos J. M. Trienekens
    • 1
  • Paul Grefen
    • 1
  1. 1.Department of Technology ManagementEindhoven University of TechnologyThe Netherlands

Personalised recommendations