, Volume 101, Issue 3, pp 1627–1656 | Cite as

A framework for systematic analysis of open access journals and its application in software engineering and information systems

  • Daniel Graziotin
  • Xiaofeng Wang
  • Pekka Abrahamsson


This article is a contribution towards an understanding of open access (OA) publishing. It proposes an analysis framework of 18 core attributes, divided into the areas of bibliographic information, activity metrics, economics, accessibility, and predatory issues. The framework has been employed in a systematic analysis of 30 OA journals in software engineering (SE) and information systems (IS), which were selected from among 386 OA journals in Computer Science from the Directory of OA Journals. An analysis was performed on the sample of the journals, to provide an overview of the current situation of OA journals in the fields of SE and IS. The journals were then compared between-group, according to the presence of article processing charges. A within-group analysis was performed on the journals requesting article processing charges from authors, in order to understand what is the value added according to different price ranges. This article offers several contributions. It presents an overview of OA definitions and models. It provides an analysis framework born from the observation of data and the existing literature. It raises the need to study OA in the fields of SE and IS while offering a first analysis. Finally, it provides recommendations to readers of OA journals. This paper highlights several concerns still threatening the adoption of OA publishing in the fields of SE and IS. Among them, it is shown that high article processing charges are not sufficiently justified by the publishers, which often lack transparency and may prevent authors from adopting OA.


Open access Predatory publishers Software engineering Information systems Research Systematic analysis 



The authors are thankful to Elena Borgogno for her valuable help during the study and when writing this article. The authors would like to thank Christian Gumpenberger for the insightful comments he offered to improve the manuscript. Lastly, the authors are thankful to two anonymous reviewers for the several suggestions that significantly improved the article.


  1. ACM. (2012). The 2012 ACM computing classification system. Association for Computing Machinery. Retrieved August 01, 2013, from
  2. Antelman, K. (2004). Do open-access articles have a greater research impact? College Research Libraries, 65(5), 372–382. Retrieved from
  3. Arunachalam, S. (2008). Open access to scientific knowledge. DESIDOC Journal of Library & Information Technology, 28(1), 7–14. Retrieved from
  4. Bailey, C. (2008). Open access and libraries. Collection Management, 32(3), 351–383. doi: 10.1300/J105v32n03_07.Google Scholar
  5. Beall, J. (2012a). Predatory publishers and opportunities for scholarly societies. In American Educational Research Association meeting, Washington, D.C. (Vol. 489, pp. 1–5). doi: 10.1038/489179a.
  6. Beall, J. (2012b). Predatory publishers are corrupting open access. Nature, 489(7415), 179. doi: 10.1038/489179a.CrossRefGoogle Scholar
  7. Beall, J. (2013a). Criteria for determining predatory open-access publishers (2nd ed.). Scholarly Open Access. Retrieved May 21, 2013, from
  8. Beall, J. (2013b). OA journal stops publishing, deletes Website. Scholarly Open Access. Retrieved August 01, 2013, from
  9. Beall, J. (2013c). The open-access movement is not really about open access. tripleC, 11(2), 589–597.Google Scholar
  10. Beall, J. (2013d). Potential, possible, or probable predatory scholarly open-access publishers. Scholarly Open Access. Retrieved January 28, 2014, from
  11. Björk, B. C. (2003). Open access to scientific publications - an analysis of the barriers to change. Information Research, 9(2), 170. Retrieved from
  12. Björk, B. C., & Solomon, D. (2012). Open access versus subscription journals: A comparison of scientific impact. BMC Medicine, 10(1), 73. doi: 10.1186/1741-7015-10-73.CrossRefGoogle Scholar
  13. Bjørnshauge, L., Brage, R., Brage, S., & Jørgensen, L. (2013a). DOAJ. Directory of Open Access Journals. Retrieved May 20, 2013, from
  14. Bjørnshauge, L., Brage, R., Brage, S., & Jørgensen, L. (2013b). DOAJ announces new selection criteria. Directory of Open Access Journals. Retrieved August 06, 2013, from
  15. BOAI. (2002). Budapest open access initiative. Budapest Open Access Initiative. Retrieved May 27, 2013, from
  16. Bohannon, J. (2013). Who’s afraid of peer review? Science (New York, N.Y.), 342(6154), 60–65. doi: 10.1126/science.342.6154.60.CrossRefGoogle Scholar
  17. Boisvert, R. F., & Davidson, J. W. (2013). Positioning ACM for an open access future. Communications of the ACM, 56(2), 5. doi: 10.1145/2408776.2408777.CrossRefGoogle Scholar
  18. Brown, P. O., Cabell, D., Chakravarti, A., Cohen, B., Delamothe, T., Eisen, M.,… Watson, L. (2003). Bethesda statement on open access publishing. Harvard Dash. Retrieved May 29, 2013, from
  19. Butler, D. (2012). Investigating journals: The dark side of publishing. Nature, 495(7442), 433–435. doi: 10.1038/495433a.CrossRefGoogle Scholar
  20. CLOCKSS. (2013). Contribute to CLOCKSS. CLOCKSS. Retrieved August 06, 2013, from
  21. Crawford, S., & Stucki, L. (1990). Peer review and the changing research record. Journal of the American Society for Information Science, 41(3), 223–228. doi:10.1002/(SICI)1097-4571(199004)41:3%3C223:AID-ASI14%3E3.0.CO;2-3.Google Scholar
  22. CrossRef. (2013). 2013 CrossRef Deposit Fees. CrossRef. Retrieved May 25, 2013, from
  23. Davis, P. (2009). Open access publisher accepts nonsense manuscript for dollars. The Scholarly Kitchen. Retrieved July 31, 2013, from
  24. DOAJ. (2013). About. Directory of Open Access Journals. Retrieved January 27, 2014, from
  25. Eysenbach, G. (2008). Black sheep among open access journals and publishers. Gunther Eysenbach Random Research Rants Blog. Retrieved July 31, 2013, from
  26. Falagas, M. E., Pitsouni, E. I., Malietzis, G. A., & Pappas, G. (2008). Comparison of PubMed, Scopus, Web of Science, and Google Scholar: Strengths and weaknesses. FASEB Journal: Official Publication of the Federation of American Societies for Experimental Biology, 22(2), 338–342.CrossRefGoogle Scholar
  27. Franceschet, M. (2010). The role of conference publications in CS. Communications of the ACM, 53(12), 129. doi: 10.1145/1859204.1859234.CrossRefGoogle Scholar
  28. Graziotin, D. (2013, May 18). Comparison of open access software engineering and information systems journals. figshare. doi: 10.6084/m9.figshare.704442.
  29. Gumpenberger, C., Ovalle-Perandones, M.-A., & Gorraiz, J. (2012). On the impact of gold open access journals. Scientometrics, 96(1), 221–238. doi: 10.1007/s11192-012-0902-7.CrossRefGoogle Scholar
  30. Gutman, S. A. (2011). Copyright in the age of digital scholarship. American Journal of Occupational Therapy, 65(2), 123–124. doi: 10.5014/ajot.2011.000877.CrossRefGoogle Scholar
  31. Harnad, S., & Brody, T. (2004). Comparing the impact of open access (OA) vs. non-oa articles in the same journals. DLib Magazine, 10(6), 2–6. doi: 10.1045/june2004-harnad.
  32. Harnad, S., Brody, T., Vallières, F., Carr, L., Hitchcock, S., Gingras, Y.,… Hilf, E. R. (2008). The access/impact problem and the green and gold roads to open access: An update. Serials Review, 34(1), 36–40. doi: 10.1016/j.serrev.2007.12.005 Google Scholar
  33. Holmstrom, A. (2012). What happened to online articles published in K-theory (Springer journal)? Mathoverflow. Retrieved August 01, 2013, from
  34. Houghton, J. W., & Oppenheim, C. (2010). The economic implications of alternative publishing models. Prometheus: Critical Studies in Innovation, 28(1), 41–54. doi: 10.1080/08109021003676359.
  35. IEEE. (2013). IEEE open-article processing charges. Institute of Electrical and Electornics Engineers. Retrieved May 20, 2013, from
  36. ISSN. (2013). ISSN and electronic publications. ISSN International Centre. Retrieved August 01, 2013, from
  37. Karen, C. (2013). Predatory publishers. Library Journal. Retrieved January 29, 2014, from
  38. Koehler, W. (2002). Web page change and persistence? A four-year longitudinal study. Journal of the American Society for Information Science and Technology, 53(2), 162–171. doi: 10.1002/asi.10018.CrossRefGoogle Scholar
  39. Kuhn, T. S. (1970). The structure of scientific revolutions. In O. Neurath (Ed.), Philosophical review (Vol. II, p. 210). London: University of Chicago Press. doi: 10.1119/1.1969660.
  40. Laakso, M., Welling, P., Bukvova, H., Nyman, L., Björk, B. C., & Hedlund, T. (2011). The development of open access journal publishing from 1993 to 2009. PLoS ONE, 6(6), e20961. doi: 10.1371/journal.pone.0020961.CrossRefGoogle Scholar
  41. Laika Spoetnik, J. (pseudonym). (2011). Jeffrey Beall’s list of predatory, open-access publishers. Laika’s MedLibLog.Google Scholar
  42. Langston, M., & Tyler, J. (2004). Linking to journal articles in an online teaching environment: The persistent link, DOI, and OpenURL. The Internet and Higher Education, 7(1), 51–58. doi: 10.1016/j.iheduc.2003.11.004.CrossRefGoogle Scholar
  43. Leubsdorf, C. (2012). Annotum: Launching a peer-reviewed journal online for free. Learned Publishing, 25(2), 8. doi: 10.1087/20120204.CrossRefGoogle Scholar
  44. Ley, M. (2002). The DBLP computer science bibliography: Evolution, research issues, perspectives. In 9th international symposium on string processing and information retrieval (SPIRE 2013) (Vol. 2476, pp. 1–10). Lisbon, Portugal: Springer, Berlin. doi: 10.1007/3-540-45735-6.
  45. Liu, Z. (2006). Print vs. electronic resources: A study of user perceptions, preferences, and use. Information Processing and Management, 42(2), 583–592. doi: 10.1016/j.ipm.2004.12.002.CrossRefGoogle Scholar
  46. Maniatis, P., Roussopoulos, M., Giuli, T. J., Rosenthal, D. S. H., & Baker, M. (2005). The LOCKSS peer-to-peer digital preservation system. ACM Transactions on Computer Systems, 23(1), 2–50. doi: 10.1145/1047915.1047917.CrossRefGoogle Scholar
  47. Mattern, F. (2008). Bibliometric evaluation of computer science—Problems and pitfalls. In European computer science summit (Vol. 2008, p. 6). Zurich, Switzerland. Retrieved from
  48. Max Planck Society. (2003). Berlin declaration. Open Access at the Max Planck Society. doi: 10.1353/hrq.2005.0002.
  49. Molloy, J. C. (2011). The open knowledge foundation: Open data means better science. PLoS Biology, 9(12), e1001195. doi: 10.1371/journal.pbio.1001195.CrossRefGoogle Scholar
  50. Mosallahnezhad, R. (2007). REMOVED: Cooperative, compact algorithms for randomized algorithms. Applied Mathematics and Computation. doi: 10.1016/j.amc.2007.03.011.
  51. Navidi, W. (2010). Statistics for engineers and scientists. Education (3rd ed., Vol. 6). McGraw-Hill. doi: 10.2307/2288012.
  52. OASPA. (2013). Code of conduct. Open access scholarly publishers association. Retrieved from
  53. Parks, R. P. (2002). The Faustian grip of academic publishing. Journal of Economic Methodology, 9(3), 317–335. doi: 10.1080/1350178022000015122.CrossRefGoogle Scholar
  54. Piwowar, H. (2013). Altmetrics: Value all research products. Nature, 493(7431), 159. doi: 10.1038/493159a.Google Scholar
  55. Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). Altmetrics: A manifesto. Retrieved from
  56. Pringle, J. (2013). Do Open Access journals have impact? Nature Web focus: Access to the literature. Retrieved June 05, 2013, from
  57. Regazzi, J. (2004). The shifting sands of open access publishing, a publisher’s view. Serials Review, 30(4), 275–280. doi: 10.1016/j.serrev.2004.09.010.CrossRefGoogle Scholar
  58. Reich, V., & Rosenthal, D. (2004). Preserving today’s scientific record for tomorrow. BMJ (Clinical research ed.), 328(7431), 61–62. doi: 10.1136/bmj.328.7431.61.CrossRefGoogle Scholar
  59. Schroter, S., & Tite, L. (2006). Open access publishing and author-pays business models: A survey of authors’ knowledge and perceptions. Journal of the Royal Society of Medicine, 99(3), 141–148. doi: 10.1258/jrsm.99.3.141.CrossRefGoogle Scholar
  60. Schroter, S., Tite, L., & Smith, R. (2005). Perceptions of open access publishing: Interviews with journal authors. BMJ (Clinical research ed.), 330(7494), 756. doi: 10.1136/bmj.38359.695220.82.CrossRefGoogle Scholar
  61. Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. BMJ, 314(7079), 497. doi: 10.1136/bmj.314.7079.497.CrossRefGoogle Scholar
  62. SHERPA. (2002). RoMEO—Search—Publisher copyright policies & self-archiving. SHERPA Services based at the University of Nottingham. Retrieved May 29, 2013, from
  63. Smith, R. (1999). Opening up BMJ peer review. BMJ, 318(7175), 4–5. doi: 10.1136/bmj.318.7175.4.CrossRefGoogle Scholar
  64. Sol, H. G. (1983). A feature analysis of information systems design methodologies: Methodological considerations. In Information Systems design methodologies: A feature analysis (pp. 1–7).Google Scholar
  65. Song, X., & Osterweil, L. J. (1992). Toward objective, systematic design-method comparisons. IEEE Software, 9(3), 43–53. doi: 10.1109/52.136166.CrossRefGoogle Scholar
  66. Sotudeh, H., & Horri, A. (2007). Tracking open access journals evolution: Some considerations in open access data collection validation. Journal of the American Society for Information Science and Technology, 58(11), 1578–1585. doi: 10.1002/asi.20639.CrossRefGoogle Scholar
  67. Stribling, J., Krohn, M., & Aguayo, D. (2005). SCIgen—An automatic CS paper generator. PDOS Research. Retrieved July 31, 2013, from
  68. Suber, P. (2008). Gratis and libre open access. SPARC Open Access Newsletter. Retrieved May 29, 2013, from
  69. Suber, P. (2009). Timeline of the open access movement. Open Access Directory. Retrieved July 29, 2013, from
  70. Suber, P. (2012). Open Access (1st ed., pp. 1–255). Cambridge, MA: The MIT Press.Google Scholar
  71. Swoger, B. (2013). It’s not about predators, it’s about journal quality. Scientific American Blogs. Retrieved January 29, 2014, from
  72. Taylor, M. (2012). Crowdsourcing a database of “predatory OA journals.” Sauropod Vertebra Picture of the Week. Retrieved January 29, 2014, from
  73. Van Noorden, R. (2012). Journal offers flat fee for “all you can publish”. Nature, 486(7402), 166. doi: 10.1038/486166a.CrossRefGoogle Scholar
  74. Van Noorden, R. (2013). Open access: The true cost of science publishing. Nature, 495(7442), 426–429. doi: 10.1038/495426a.CrossRefGoogle Scholar
  75. Vardi, M. Y. (2012). Predatory scholarly publishing. Communications of the ACM, 55(7), 5. doi: 10.1145/2209249.2209250.CrossRefGoogle Scholar
  76. Vision, T. J. (2010). Open data and the social contract of scientific publishing. BioScience, 60(5), 330–331. doi: 10.1525/bio.2010.60.5.2.CrossRefGoogle Scholar
  77. Wainer, J., Billa, C., & Goldenstein, S. (2011). Invisible work in standard bibliometric evaluation of computer science. Communications of the ACM, 54(5), 141. doi: 10.1145/1941487.1941517.CrossRefGoogle Scholar
  78. Willinsky, J. (2005). Open journal systems: An example of open source software for journal management and publishing. Library Hi Tech, 23(4), 504–519. doi: 10.1108/07378830510636300.CrossRefGoogle Scholar
  79. Willinsky, J. (2006). The access principle. electronic publishing (1st ed., Vol. 85, pp. 227–232). Cambridge, MA: MIT Press.Google Scholar
  80. Worlock, K. (2013). The pros and cons of open access. Nature Web Focus: Access to the Literature. Retrieved June 05, 2013, from
  81. Young, B. (1988). International standard serial numbers. Serials: The Journal for the Serials Community, 1(2), 48–50. doi: 10.1629/010248.

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2014

Authors and Affiliations

  • Daniel Graziotin
    • 1
  • Xiaofeng Wang
    • 1
  • Pekka Abrahamsson
    • 1
  1. 1.Faculty of Computer ScienceFree University of Bozen-BolzanoBolzanoItaly

Personalised recommendations