Advertisement

Measuring Science

Capita Selecta of Current Main Issues
  • Anthony F.J. van Raan

Abstract

After a review of developments in the quantitative study of science, particularly since the early 1970s, I focus on two current main lines of ‘measuring science’ based on bibliometric analysis. With the developments in the Leiden group as an example of daily practice, the measurement of research performance and, particularly, the importance of indicator standardisation are discussed, including aspects such as interdisciplinary relations, collaboration, ‘knowledge users’. Several important problems are addressed: language bias; timeliness; comparability of different research systems; statistical issues; and the ‘theory-invariance’ of indicators. Next, an introduction to the mapping of scientific fields is presented. Here basic concepts and issues of practical application of these ‘science maps’ are addressed. This contribution is concluded with general observations on current and near-future developments, including network-based approaches, necessary ‘next steps’ are formulated, and an answer is given to the question ‘Can science be measured?’

Keywords

Research Performance Science Citation Index Citation Analysis Bibliometric Analysis Bibliometric Indicator 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Adam, D. (2002). The counting house. Nature, 415, 726–729.Google Scholar
  2. Albert, M.B., Avery, D., Narin, F., MacAllister, P. (1991). Direct validation of citation counts as indicators of industrially important patents. Research Policy, 20, 251–259.CrossRefGoogle Scholar
  3. Arunachalam, S., Srinivasan, R., Raman, V. (1994). International collaboration in science-participation by the Asian giants. Scientometrics, 30, 7–22.CrossRefGoogle Scholar
  4. Bar-Ilan, J. (2001). Data collection methods on the Web for informetric purposes — A review and analysis. Scientometrics, 50, 7–32.CrossRefGoogle Scholar
  5. Bayer, A.E., Smart, J.C., McLaughlin, G.W. (1990). Mapping intellectual structure of a scientific subfield through author cocitations. Journal of the American Society for Information Science, 41, 444–452.CrossRefGoogle Scholar
  6. Beaver, D. de B., Rosen, R. (1978). Studies in scientific collaboration, 1: Professional origins of scientific co-authorship. Scientometrics, 1, 65–84.CrossRefGoogle Scholar
  7. Björneborn, L., Ingwersen, P. (2001). Perspectives of webometrics. Scientometrics, 50, 65–82.CrossRefGoogle Scholar
  8. Borgman, C.L. (ed.) (1990). Scholarly Communication and Bibliometrics. Newbury Park: Sage.Google Scholar
  9. Braam, R.R., Moed, H.F., van Raan, A.F.J. (1991a). Mapping of science by combined cocitation and word analysis, I: Structural Aspects. Journal of the American Society for Information Science (JASIS), 42, 233–251, and, II: Dynamical Aspects. Journal of the American Society for Information Science (JASIS), 42, 252–266.Google Scholar
  10. Braun, T., Glänzel, W., Schubert, A. (1988). World flash on basic research — The newest version of the facts and figures on publication output and relative citation impact of 100 countries 1981–1985. Scientometrics, 13, 181–188.Google Scholar
  11. Braun, T., Glänzel, Grupp, H. (1995). The scientometric weight of 50 nations in 27 science areas, 1989–1993. 1: All fields combined, mathematics, engineering, chemistry and physics. Scientometrics, 33, 263–293; and 2: Life sciences. Scientometrics, 34, 207–237.Google Scholar
  12. Brooks, T.A. (1986). Evidence of complex citer motivations. Journal of the American Society for Information Science, 37, 34–36.Google Scholar
  13. Butler, L. (2003). Modifying publication practices in response to funding formulas. Research Evaluation, 17, 39–46.Google Scholar
  14. Callon, M., Bauin, S., Courtial, J.P., Turner, W. (1983). From translation to problematic networks: an introduction to co-word analysis. Social Science Information, 22, 191–235.Google Scholar
  15. Cole, S., Cole, J.R., Dietrich, L. (1978). Measuring the cognitive state of scientific disciplines. In: Lederberg, J., Merton, R.K., Thackray, A., Zuckerman, H. (Eds.). Toward a metric of science: the advent of science indicators. New York: John Wiley Elkana et al., op. cit..Google Scholar
  16. de Candolle, A. (1873, 2nd. edition 1885). Histoire des sciences et des savants depuis deux siècles. Genève/Basel: H.Georg. Reprint in 1987 by Fayard.Google Scholar
  17. Egghe L., Rousseau, R. (2000). The influence of publication delays on the observed aging distribution of scientific literature. Journal of the American Society for Information Science, 51, 158–165.Google Scholar
  18. Elkana, Y., Lederberg, J., Merton, R.K., Thackray, A., Zuckerman, H. (Eds.) (1978). Toward a metric of science: the advent of science indicators. New York: John Wiley.Google Scholar
  19. Etzkowitz, H., Leydesdorff, L. (2000). The dynamics of innovation: from National Systems and “Mode 2” to a Triple Helix of university-industry-government relations. Research Policy, 29, 109–123.Google Scholar
  20. Garfield, E. (1979). Is citation analysis a legitimate evaluation tool? Scientometrics, 1, 359–375.CrossRefGoogle Scholar
  21. Garfield, E. (1980). Premature discovery or delayed recognition — Why? Current Contents, 21, May 26, 5–10.Google Scholar
  22. Gilbert, G.N. (1978). Measuring the growth of science-review of indicators of scientific growth. Scientometrics, 1, 9–34.CrossRefGoogle Scholar
  23. Glänzel, W. (1996). A bibliometric approach to social sciences, national research performances in 6 selected social science areas, 1990–1992. Scientometrics, 35, 291–307.Google Scholar
  24. Glänzel, W. (2001). National characteristics in international scientific co-authorship relations. Scientometrics, 51, 69–115.Google Scholar
  25. Glänzel, W., Meyer, M. (2003). Patents cited in the scientific literature: An exploratory study of ‘reverse’ citation relations. Scientometrics, 58, 415–428.Google Scholar
  26. Glänzel, W., Schlemmer, B., Thijs, B. (2003). Better late than never? On the chance to become highly cited only beyond the standard bibliometric time horizon. Scientometrics, 58, 571–586Google Scholar
  27. Glänzel, W., Debackere, K. (2003). On the opportunities and limitations in using bibliometric indicators in a policy relevant context. In: Bibliometric analysis in science and research. Applications, Benefits and Limitations. Second Conference of the Central Library, Forschungszentrum Jülich, (pp. 225–236). (ISBN 3-89336-334-3).Google Scholar
  28. Gläser, J., Laudel, G. (2001). Integrating scientometric indicators into sociological studies: methodical and methodological problems. Scientometrics, 52, 411–434.CrossRefGoogle Scholar
  29. Grupp, H., Schmoch, U., Hinze, S. (2001). International alignment and scientific regard as macro-indicators for international comparisons of publications. Scientometrics, 51, 359–380.CrossRefGoogle Scholar
  30. Haitun, S.D. (1982). Stationary scientometric distributions. 1: Different approximations. Scientometrics, 4, 89–104.Google Scholar
  31. Hicks, D. (1999). The difficulty of achieving full coverage of international social science literature and the bibliometric consequences. Scientometrics, 44, 193–215.Google Scholar
  32. Holton, G. (1978). Can science be measured? In: Lederberg, J., Merton, R.K., Thackray, A., Zuckerman, H. (Eds.). Toward a metric of science: the advent of science indicators. New York: John Wiley Elkana et al., op. cit.Google Scholar
  33. Horrobin, D.F. (1990). The philosophical basis of peer review and the suppression of innovation. Journal of the American Medical Association (JAMA), 263, 1438–1441.Google Scholar
  34. Kamerlingh Onnes, H. (1882). De betekenis van kwantitatief onderzoek in de natuurkunde (The meaning of quantitative research in physics). Inaugural Address as Professor of Physics, Leiden University.Google Scholar
  35. Koenig, M.E.D. (1983). Bibliometric indicators versus expert opinion in assessing research performance. Journal of the American Society for Information Science, 34, 136–145.Google Scholar
  36. Kostoff, R.N. (1995). Federal research impact assessment — Axioms, approaches, applications. Scientometrics, 34, 163–206.CrossRefGoogle Scholar
  37. van Leeuwen, Th.N., Moed, H.F., Tijssen, R.J.W., Visser, M.S., van Raan, A.F.J. (2001). Language biases in the coverage of the Science Citation Index and its consequences for international comparisons of national research performance. Scientometrics, 51, 335–346.Google Scholar
  38. van Leeuwen, Th.N. (2004). Second generation bibliometric analysis. Ph.D. Thesis Leiden University.Google Scholar
  39. Lewison, G. (2001). The quantity and quality of female researchers: a bibliometric study of Iceland. Scientometrics, 52, 29–43.CrossRefGoogle Scholar
  40. Lewison, G. (2002). Researchers’ and users’ perceptions of the relative standing of biomedical papers in different journals. Scientometrics, 53, 229–240.CrossRefGoogle Scholar
  41. Lotka, A.J. (1926). The frequency distribution of scientific productivity. J. Washington Acad. Sci., 16, 317–323.Google Scholar
  42. MacRoberts, M.H., MacRoberts, B.R. (1996). Problems of citation analysis. Scientometrics, 36, 435–444.CrossRefGoogle Scholar
  43. MacRoberts, M.H., MacRoberts, B.R. (1988). Author motivation for not giving citing influences — A methodological note. Journal of the American Society for Information Science, 39, 432–433.CrossRefGoogle Scholar
  44. Martin, B.R., Irvine, J. (1983). Assessing basic research: some partial indicators of scientific progress in radio astronomy. Research Policy, 12, 61–90.CrossRefGoogle Scholar
  45. May, R.M. (1997). The scientific wealth of nations. Science, 275, 793–796.CrossRefGoogle Scholar
  46. McCain, K.W. (1984). Longitudinal author cocitation mapping — The changing structure of macroeconomics. Journal of the American Society for Information Science, 35, 351–359.Google Scholar
  47. McCain, K.W. (1990). Mapping authors in intellectual space — A technical overview. Journal of the American Society for Information Science, 41, 433–443.CrossRefGoogle Scholar
  48. Melin, G., Persson, O. (1996). Studying research collaboration using co-authorships. Scientometrics, 36, 363–377.CrossRefGoogle Scholar
  49. Moed, H.F., van Leeuwen, Th.N. (1995). Improving the accuracy of the Institute for Scientific Information’s Journal Impact Factors. J. of the American Society for Information Science (JASIS), 46, 461–467.Google Scholar
  50. Moed, H.F., van Leeuwen, Th.N. (1996). Impact factors can mislead. Nature, 381, 186.CrossRefGoogle Scholar
  51. Moed, H.F., Luwel, M., Nederhof, A.J. (2002). Towards research performance measurement in the humanities. Library Trends, 50, 498–520.Google Scholar
  52. Moravcsik, M.J. (1975). Phenomenology and models of growth of science. Research Policy, 4, 80–86.CrossRefGoogle Scholar
  53. Moravcsik, M.J., Murugesan, P. (1979). Citation patterns in scientific revolutions. Scientometrics, 1, 161–169.CrossRefGoogle Scholar
  54. Moxham, H., Anderson, J. (1992). Peer review. A view from the inside. Science and Technology Policy, February 1992, 7–15.Google Scholar
  55. Narin, F. (1976). Evaluative bibliometrics: The use of publication and citation analysis in the evaluation of scientific activity. Washington D.C.: National Science Foundation.Google Scholar
  56. Narin, F. (1978). Objectivity versus relevance in studies of scientific advance. Scientometrics, 1, 35–41.CrossRefGoogle Scholar
  57. Narin, F. (1994). Patent bibliometrics. Scientometrics, 30, 147–155.CrossRefGoogle Scholar
  58. Narin, F., Hamilton, K.S., Olivastro, D. (1997). The increasing linkage between US technology and public science. Research Policy, 26, 317–330.CrossRefGoogle Scholar
  59. National Science Board (1973). Science Indicators 1972. Washington DC: Government Printing Office.Google Scholar
  60. Nederhof, A.J. (1988). The validity and reliability of evaluation of scholarly performance. In: A.F.J. van Raan (ed). (1988), Handbook of Quantitative Studies of Science and Technology (pp. 193–228). Amsterdam: Elsevier/North-Holland, (ISBN 0-444-70537-6).Google Scholar
  61. Noma, E. (1982). An improved method for analysing square scientometric transaction matrices. Scientometrics, 4, 297–316.Google Scholar
  62. Noyons, E.C.M., van Raan, A.F.J. (1998). Monitoring scientific developments from a dynamic perspective: self-organized structuring to map neural network research. J. of the American Society for Information Science and Technology (JASIST), 49, 68–81.Google Scholar
  63. Noyons, E.C.M., Luwel, M., Moed, H.F. (1999). Combining mapping and citation analysis for evaluative bibliometric purpose. A bibliometric study on recent development in microelectronics. Journal of the American Society for Information Science and Technology (JASIST), 50, 115–131.Google Scholar
  64. Noyons, E.C.M. (1999). Bibliometric mapping as a science policy and research management tool. Ph.D. Thesis Leiden University. Leiden: DSWO Press (ISBN 90-6695-152-4).Google Scholar
  65. Noyons, E.C.M., Buter, R.K., van Raan, A.F.J., Schmoch, U., Heinze, T., Hinze, S., Rangnow, R. (2003). Mapping excellence in science and technology across Europe (Part 1: Life sciences, Part 2: Nanoscience and nanotechnology). Report to the European Commission by the Centre for Science and Technology Studies (CWTS), Leiden University, and the Fraunhofer Institute for Systems and Innovation Research (Fraunhofer-ISI), Karlsruhe.Google Scholar
  66. Noyons, E.C.M. (2004). Science Maps within in a Science Policy Context. This Handbook.Google Scholar
  67. OECD (1963). The measurement of scientific and technological activities, ‘Frascati Manual’, Paris: Organization for Economic Co-operation and Development (OECD).Google Scholar
  68. Peritz, B.C. (1983). A classification of citation roles for the social sciences and related fields. Scientometrics, 5, 303–312.Google Scholar
  69. Porter, A.L., Chubin. D.E. (1985). An indicator of cross-disciplinary research. Scientometrics, 8, 161–176.CrossRefGoogle Scholar
  70. De Solla Price, D.J. (1978). Toward a model for Science Indicators. In: Lederberg, J., Merton, R.K., Thackray, A., Zuckerman, H., (Eds.). Toward a metric of science: the advent of science indicators. New York: John Wiley Elkana et al., op.cit.Google Scholar
  71. De Solla Price, D.J. (1981). The analysis of scientometric matrices for policy implications. Scientometrics, 3, 47–53.Google Scholar
  72. Prime, C., Bassecoulard, E., Zitt, M. (2002). Co-citations and co-sitations: A cautionary view on an analogy. Scientometrics, 54, 291–308.CrossRefGoogle Scholar
  73. Prpić, K. (2002). Gender and productivity differentials in science. Scientometrics, 55, 27–58.Google Scholar
  74. van Raan, A.F.J. (ed). (1988). Handbook of Quantitative Studies of Science and Technology. Amsterdam: Elsevier/North-Holland (ISBN 0-444-70537-6).Google Scholar
  75. van Raan, A.F.J. (1990). Fractal dimension of co-citations. Nature, 347, 626.Google Scholar
  76. van Raan, A.F.J. (1996). Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises. Scientometrics, 36, 397–420.CrossRefGoogle Scholar
  77. van Raan, A.F.J. (1997). Scientometrics: State-of-the-Art. Scientometrics, 38, 205–218.Google Scholar
  78. van Raan, A.F.J. (1998). In matters of quantitative studies of science the fault of theorists is offering too little and asking too much. Scientometrics, 43, 129–139.Google Scholar
  79. van Raan, A.F.J. (2000a). The Pandora’s box of citation analysis: measuring scientific excellence, the last evil? In: B. Cronin and H. Barsky Atkins (eds.). The Web of Knowledge. A Festschrift in honor of Eugene Garfield. Ch. 15, p. 301–319. Medford (New Jersey): ASIS Monograph Series, 2000 (ISBN 1-57387-099-4).Google Scholar
  80. van Raan, A.F.J. (2000b). On growth, ageing, and fractal differentiation of science. Scientometrics 47, 347–362.Google Scholar
  81. van Raan, A.F.J. (2001). Two-step competition process leads to quasi power-law income distributions. Application to scientific publication and citation distributions. Physica A, 298, 530–536.Google Scholar
  82. van Raan, A.F.J., Noyons, E.C.M. (2002). Discovery of patterns of scientific and technological development and knowledge transfer. In W. Adamczak, A. Nase (Eds.), Gaining Insight from Research Information. Proceedings of the 6th International Conference on Current Research Information Systems, University of Kassel, August 29–31, 2002 (pp. 105–112). Kassel: University Press, (ISBN 3-933146-844).Google Scholar
  83. van Raan, A.F.J., van Leeuwen, Th.N. (2002). Assessment of the scientific basis of interdisciplinary, applied research. Application of bibliometric methods in nutrition and food research. Research Policy, 31, 611–632Google Scholar
  84. van Raan, A.F.J. (2003). Reference-based publication networks with episodic memories. E-print ArXiv cond-mat/0311318.Google Scholar
  85. van Raan, A.F.J. (2004). Sleeping Beauties in Science. Scientometrics, 59, 461–466.Google Scholar
  86. van Raan, A.F.J., van Leeuwen, Th.N. (2004). Statistical aspects of research group performance, journal impact, and peer judgement. To be published.Google Scholar
  87. Rinia, E.J., van Leeuwen, Th.N., van Vuren, H.G., van Raan, A.F.J. (1998). Comparative analysis of a set of bibliometric indicators and central peer review criteria. Evaluation of condensed matter physics in the Netherlands. Research Policy, 27, 95–107.CrossRefGoogle Scholar
  88. Rinia, E.J., van Leeuwen, Th.N., van Vuren, H.G., van Raan, A.F.J. (2001). Influence of interdisciplinarity on peer-review and bibliometric evaluations. Research Policy, 30, 357–361.CrossRefGoogle Scholar
  89. Rip, A., Courtial, J.P. (1984). Co-word maps of biotechnology — An example of cognitive scientometrics. Scientometrics, 6, 381–400.CrossRefGoogle Scholar
  90. Schmoch, U. (1993). Tracing the knowledge transfer from science to technology as reflected in patent indicators. Scientometrics, 26, 193–211.CrossRefGoogle Scholar
  91. Schwechheimer, H., Winterhager, M. (2001). Mapping interdisciplinary research fronts in neuroscience: a bibliometric view to retrograde amnesia. Scientometrics, 51, 311–318.CrossRefGoogle Scholar
  92. Schubert A., Glänzel, W. (1983). Statistical reliability of comparisons based on the citation impact of scientometric publications. Scientometrics, 5, 59–74.Google Scholar
  93. Seglen, P.O. (1992). The skewness of science. Journal of the American Society for Information Science, 43, 628–638.CrossRefGoogle Scholar
  94. Seglen, P.O. (1994). Causal relationship between article citedness and journal impact. Journal of the American Society for Information Science, 45, 1–11.CrossRefGoogle Scholar
  95. Small, H. (1973). Co-citation in the Scientific Literature: A New Measure of the Relationship Between Publications. Journal of the American Society for Information Science, 24, 265–269.Google Scholar
  96. Small, H., Greenlee, E. (1980). Citation context analysis of a co-citation cluster-recombinant DNA. Scientometrics, 2, 1980.CrossRefGoogle Scholar
  97. Small, H., Sweeney, E. (1985). Clustering the Science Citation Index using co-citations, I: A Comparison of Methods. Scientometrics, 7, 393–404.CrossRefGoogle Scholar
  98. Small, H., Sweeney, E., Greenlee, E. (1985). Clustering the Science Citation Index using cocitations, II: Mapping Science. Scientometrics, 8, 321–340.CrossRefGoogle Scholar
  99. Small, H. (1999). Visualizing science by citation mapping. Journal of the American Society for Information Science, 50, 799–813.CrossRefGoogle Scholar
  100. Swanson, D.R. (1986). Fish oil, Raynaud’s syndrome, and undiscovered public knowledge. Perspectives in Biology and Medicine, 30, 7–18.Google Scholar
  101. Swanson, D.R. (1987). Two medical literatures that are logically but not bibliographically connected. Journal of the American Society for Information Science, 38, 228–233.CrossRefGoogle Scholar
  102. Sullivan D., Koester, D., White, D.H., Kern, R. (1980). Understanding rapid theoretical change in particle physics — a month-by-month co-citation analysis. Scientometrics, 2, 309–319.CrossRefGoogle Scholar
  103. Thelwall, M., Smith, A. (2002). Interlinking between Asia-Pacific University Web sites. Scientometrics, 55, 363–376.CrossRefGoogle Scholar
  104. Thelwall, M., Harries, G. (2003). The connection between the research of a university and counts of links to its web pages: An investigation based upon a classification of the relationships of pages to the research of the host university. Journal of the American Society for Information Science, 54, 594–602.Google Scholar
  105. Vinkler, P. (1993). Research contribution, authorship and team cooperativeness. Scientometrics 26, 213–230.CrossRefGoogle Scholar
  106. Vinkler, P. (1998). Comparative investigation of frequency and strength of motives toward referencing, the reference threshold model-comments on theories of citation? Scientometrics, 43, 107–127.Google Scholar
  107. Vlàchy, J. (1979). Mobility in science. Bibliography of scientific career migration, field mobility, international academic circulation and brain drain. Scientometrics, 1, 201–228.Google Scholar
  108. Weingart, P. (2003). Evaluation of research performance: the danger if numbers. In: Bibliometric analysis in science and research. Applications, Benefits and Limitations. Second Conference of the Central Library, Forschungszentrum Jülich (pp. 7–19). (ISBN 3-89336-334-3).Google Scholar
  109. White, H.D., Griffith, B.C. (1981). Author cocitation— a literature measure of intellectual structure. Journal of the American Society for Information Science, 32, 163–171.Google Scholar
  110. White, H.D., McCain, K.W. (1998). Visualizing a discipline: An author co-citation analysis of information science, 1972—1995. Journal of the American Society for Information Science, 49, 327–355.Google Scholar
  111. Wouters, P.F. (1999), The Citation Culture, PhD thesis, University of Amsterdam.Google Scholar
  112. Ziman, J. (1978). From Parameters to Portents-and Back. In: Lederberg, J., Merton, R.K., Thackray, A., Zuckerman, H., (Eds.). Toward a metric of science: the advent of science indicators. New York: John Wiley Elkana et al., op.cit.Google Scholar

Copyright information

© Kluwer Academic Publishers 2004

Authors and Affiliations

  • Anthony F.J. van Raan
    • 1
  1. 1.Centre for Science and Technology Studies (CWTS)Leiden Universitythe Netherlands

Personalised recommendations