Abstract
This article investigates to what extent researchers’ experience and the solidity of their academic environment influence the quality of their research. The hypotheses are derived from the assumptions that experience matters for quality research and that there are great intellectual synergies to be obtained from interacting with many colleagues who are active researchers. All articles published between 2000 and 2006 in five leading transportation journals are included in the analysis, and their research quality is measured by the number of times each article is cited by August 2016. Controlling for other factors influencing citations, such as article age and the number of references, the most important finding is that both experience and academic environment matter for performing quality research. When the authors’ experience, measured by the number of previous publications, increases by 1% from its average level, their published articles are expected to garner 0.31% more citations. Moreover, when the research activity at the unit to which the authors are affiliated, measured by the unit’s total number of publications, increases by 1% from its average level, the number of times their articles are cited will increase by 0.19%. This signals that, relatively speaking, the researchers’ own experience and merits mean more than the academic environment with regard to producing high-quality research. The above results enable us to discuss how researchers’ experience can compensate for working in less active academic communities holding research quality constant.
Similar content being viewed by others
Notes
A thorough review on citing motives is provided by Bornmann and Daniel (2008).
From SCImago (2015), we deduced the following values of the weights: \(v_{\text{PA}} = 1.131,v_{\text{TR}} = 1.261, v_{\text{TS}} = 1.405\) and \(v_{\text{PB}} = 1.556.\) A similar method has been used in Norway to measure the research activity (publication points) at different universities and colleges; see Aagaard et al. (2014).
It is obvious that the value of AN has nothing to do with an article’s quality. This reveals one weakness of using citation counts as a quality indicator.
The five journals from which the articles analysed in this study were drawn are all in the top quartile of 119 transportation journals as ranked by SCImago (2015). This confirms that our selected journals are of good quality.
The \({\text{MRT }}\) defined here is equivalent with the concept “marginal rate of technical substitution.” It is an important concept when it comes to describing the characteristics of different production functions. See, for example, Nicholson (1998).
Problems of ranking researchers’ merits across different fields are discussed in Perry and Reny (2016).
Daniel McFadden and James Heckman were awarded the Nobel Prize in economics in 2000. An extensive review of the developments in the economics of transportation is given in Winston (1985).
References
Aagaard, K., Bloch, C., Schneider, J. W., Henriksen, D., Ryan, T. K., & Lauridsen, P. S. (2014). Evaluation of the Norwegian publication indicator—English summary. Aarhus: Danish Centre for Studies in Research and Research Policy, Aarhus University.
Abramo, G., & D’Angelo, C. A. (2015). The relationship between the number of authors of a publication, its citations and the impact factor of the publishing journal: Evidence from Italy. Journal of Informetrics, 9(4), 746–761. https://doi.org/10.1016/j.joi.2015.07.003.
Aksnes, D. W. (2003). Characteristics of highly cited papers. Research Evaluation, 12(3), 159–170. https://doi.org/10.3152/147154403781776645.
Beaver, D. D. (2004). Does collaborative research have greater epistemic authority? Scientometrics, 60(3), 399–408.
Becker, G. S. (1964). Human capital: A theoretical and empirical analysis with special reference to education. Chicago: The University of Chicago Press.
Bergh, D. D., Perry, J., & Hanke, R. (2006). Some predictors of SMJ article impact. Strategic Management Journal, 27(1), 81–100.
Bertocchi, G., Gambardella, A., Jappelli, T., Nappi, C. A., & Peracchi, F. (2015). Bibliometric evaluation vs. informed peer review: Evidence from Italy. Research Policy, 44(2), 451–466. https://doi.org/10.1016/j.respol.2014.08.004.
Bonaccorsi, A., & Daraio, C. (2003). Age effects in scientific productivity: The case of the Italian National Research Council (CNR). Scientometrics, 58(1), 49–90.
Bornmann, L., & Daniel, H. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45–80.
Diamond, A. M. (1986). What is a citation worth? The Journal of Human Resources, 21(2), 200–215. https://doi.org/10.2307/145797.
Didegah, F., & Thelwall, M. (2013). Determinants of research citation impact in nanoscience and nanotechnology. Journal of the American Society for Information Science and Technology, 64(5), 1055–1064.
Ebadi, A., & Schiffauerova, A. (2015). How to become an important player in scientific collaboration networks? Journal of Informetrics, 9(2015), 809–825.
Fox, J. (2016). Applied regression analysis and generalized linear models. Beverly Hills: Sage.
Garfield, E. (1962). Can citation indexing be automated? Essays of an Information Scientist, 1, 84–90.
Garfield, E. (1998). Random thoughts on citationology its theory and practice. Scientometrics, 43(1), 69–76. https://doi.org/10.1007/bf02458396.
Gazni, A., & Didegah, F. (2011). Investigating different types of research collaboration and citation impact: A case study of Harvard University’s publications. Scientometrics, 87(2), 251–265. https://doi.org/10.1007/s11192-011-0343-8.
Habibzadeh, F., & Yadollahie, M. (2010). Are shorter article titles more attractive for citations? Crosssectional study of 22 scientific journals. Croatian Medical Journal, 51(2), 165–170.
Hair, J. F. (1998). Multivariate data analysis. Upper Saddle River, NJ: Prentice Hall.
Hanssen, T.-E. S., & Jørgensen, F. (2014). Citation counts in transportation research. European Transport Research Review, 6(2), 205–212.
Hanssen, T.-E. S., & Jørgensen, F. (2015). The value of experience in research. Journal of Informetrics, 9(1), 16–24.
Hensher, D. A., & Greene, W. H. (2003). The mixed logit model: The state of practice. Transportation, 30(2), 133–176. https://doi.org/10.1023/a:1022558715350.
Horta, H., Cattaneo, M., & Meoli, M. (2016). PhD funding as a determinant of PhD and career research performance. Studies in Higher Education. https://doi.org/10.1080/03075079.2016.1185406.
Jamali, H. R., & Nikzad, M. (2011). Article title type and its relation with the number of downloads and citations. Scientometrics, 88(2), 653–661.
Judge, T. A., Cable, D. M., Colbert, A. E., & Rynes, S. L. (2007). What causes a management article to be cited-article author, or journal? Academy of Management Journal, 50(3), 491–506.
Kenna, R., & Berche, B. (2011). Critical mass and the dependency of research quality on group size. Scientometrics, 86(2), 527–540.
Kunnskapsdepartementet. (2014). Langtidsplan for forskning og høyere utdanning 2015–2024. Meld. St. 7 (2015–2024) Oslo.
Kyvik, S. (1995). Are big university departments better than small ones? Higher Education, 30(3), 295–304.
Lariviere, V., Gingras, Y., Sugimoto, C. R., & Tsou, A. (2015). Team size matters: Collaboration and scientific impact since 1900. Journal of the Association for Information Science and Technology, 66(7), 1323–1332.
Lokker, C., McKibbon, K. A., McKinlay, R. J., Wilczynski, N. L., & Haynes, R. B. (2008). Prediction of citation counts for clinical articles at two years using data available within three weeks of publication: Retrospective cohort study. BMJ, 336(7645), 655–657.
Martin, B. R., & Irvine, J. (1983). Assessing basic research. Research Policy, 12(2), 61–90. https://doi.org/10.1016/0048-7333(83)90005-7.
Neuhaus, C., & Daniel, H. (2008). Data sources for performing citation analysis: An overview. Journal of Documentation, 64(2), 193–210. https://doi.org/10.1108/00220410810858010.
Nicholson, W. (1998). Microeconomic theory: Basic principles and extensions. Forth Worth: Dryden Press.
NOU: 2016: 3. (2016). Ved et vendepunkt: Fra ressursøkonomi til kunnskapsøkonomi. Produktivitetskommisjonens andre rapport, Oslo.
Onodera, N., & Yoshikane, F. (2015). Factors affecting citation rates of research articles. Journal of the Association for Information Science and Technology, 66(4), 739–764. https://doi.org/10.1002/asi.23209.
Paiva, C. E., Lima, J. P. S. N., & Paiva, B. S. R. (2012). Articles with short titles describing the results are cited more often. Clinics, 67(5), 509–513.
Patterson, M. S., & Harris, S. (2009). The relationship between reviewers’ quality-scores and number of citations for papers published in the journal physics in medicine and biology from 2003–2005. Scientometrics, 80(2), 343–349. https://doi.org/10.1007/s11192-008-2064-1.
Perry, M., & Reny, P. J. (2016). How to count citations if you must. American Economic Review, 106(9), 2722–2741.
Ringelmann, M. (1913). Recherches sur les moteurs animés: Travails de l`homme. Annales del Institut National Agronomique, 2, 2–39.
Ronda-Pupo, G. A., & Katz, J. S. (2016). The power-law relationship between citation-based performance and collaboration in management journals: A scale-independent approach. Journal of the Association for Information Science and Technology, 67(10), 2565–2572.
SCImago. (2015). SJR—SCImago Journal & Country Rank. http://www.scimagojr.com. Accessed January 20, 2015.
Smithson, M., & Merkle, E. C. (2013). Generalized linear models for categorical and continuous limited dependent variables. Boca Raton, FL: CRC Press, Taylor & Francis Group.
Stankiewicz, R. (1979). The size and age of Swedish academic research groups and their scientific performance. In F. M. Andrews (Ed.), Scientific productivity. The effectiveness of research groups in six countries. Cambridge/Paris: Cambridge University Press/UNESCO.
Steiner, I. D. (1972). Group process and productivity (social psychology). New York: Academic Press.
Stremersch, S., Verniers, I., & Verhoef, P. C. (2007). The quest for citations: Drivers of article impact. Journal of Marketing, 71(3), 171–193.
Subotic, S., & Mukherjee, B. (2014). Short and amusing: The relationship between title characteristics, downloads, and citations in psychology articles. Journal of Information Science, 40(1), 115–124.
Tregenza, T. (1997). Darwin a better name than Wallace? Nature, 385(6616), 480.
Vabø, A., & Kårstein, A. (2014). Robuste fagmiljøer. En litteraturgjennomgang. Oslo: Nordic Institute for Studies in Innovation, Research and Education.
Vanclay, J. K. (2013). Factors affecting citation rates in environmental science. Journal of Informetrics, 7(2), 265–271.
Vintzileos, A. M., & Ananth, C. V. (2010). How to write and publish an original research article. American Journal of Obstetrics and Gynecology, 202(4), 344.e341–344.e346.
Vygotsky, L. (1962). Thought and language. Cambridge, MA: MIT Press.
Wallmark, J. T., Eckerstein, S., Langered, B., & Holmqvist, H. E. S. (1973). Increase in efficiency with size of research teams. IEEE Transactions on Engineering Management, EM-20(3), 80–86.
Walters, G. D. (2006). Predicting subsequent citations to articles published in twelve crime-psychology journals: Author impact versus journal impact. Scientometrics, 69(3), 499–510.
Webster, G. D., Jonason, P. K., & Schember, T. O. (2009). Hot topics and popular papers in evolutionary psychology: Analyses of title words and citation counts in evolution and human behavior, 1979–2008. Evolutionary Psychology, 7(3), 348–362.
Winston, C. (1985). Conceptual developments in the economics of transportation: An interpretive survey. Journal of Economic Literature, 23(1), 57–94.
Woelert, P. (2013). The ‘economy of memory’: Publications, citations, and the paradox of effective research governance. Minerva, 51(3), 341–362. https://doi.org/10.1007/s11024-013-9232-z.
Wuchty, S., Jones, B. F., & Uzzi, B. (2007). The increasing dominance of teams in production of knowledge. Science, 316(5827), 1036–1039. https://doi.org/10.1126/science.1136099.
Yitzhaki, M. (2002). Relation of the title length of a journal article to the length of the article. Scientometrics, 54(3), 435–447.
Author information
Authors and Affiliations
Corresponding author
Appendix: Deducing some mathematical properties of the model used
Appendix: Deducing some mathematical properties of the model used
It follows from Eq. (2) that the expressions for the first and second derivatives are \(\frac{{\partial {\text{CI}}}}{\partial i} = \beta_{i} \cdot {\text{CI}} \cdot i^{{\tau_{i} - 1}}\), \(\frac{{\partial {\text{CI}}}}{\partial j} = \beta_{j} {\text{CI}}\), \(\frac{{\partial^{2} {\text{CI}}}}{{\partial i^{2} }} = \beta_{i} \cdot {\text{CI}} \cdot i^{{\tau_{i} - 2}} \left[ {\beta_{i} \cdot i^{{\tau_{i} }} + \tau_{i} - 1} \right]\) and \(\frac{{\partial^{2} {\text{CI}}}}{{\partial j^{2} }} = \beta_{j}^{2} {\text{CI}}\), \(i = \left\{ {{\text{PI}},{\text{PP}},{\text{RF}}} \right\},\;j = \left\{ {{\text{AN}},{\text{YP}},{\text{WT}}} \right\}\). When \(0 < \tau_{i} < 1\) it implies that \(\frac{{\partial {\text{CI}}}}{\partial i},\frac{{\partial {\text{CI}}}}{\partial j} \ge \left( < \right) 0\) when \(\beta_{i} ,\beta_{j} \ge \left( < \right)0\), \(\frac{{\partial^{2} {\text{CI}}}}{{\partial i^{2} }} > 0\) when \(\beta_{i} < {0\,{\text{or}}\,\beta_{i } } > 0\;{\text{and}}\,i > \left[ {\frac{{1 - \tau_{i} }}{{\beta_{i} }}} \right]^{{1/\tau_{i} }} , \frac{{\partial^{2} {\text{CI}}}}{{\partial i^{2} }} < 0\) otherwise. When \(\tau_{\text{PI}}\) and \(\tau_{\text{PP}}\) are set to the stated values in Eq. (2). \(\frac{{\partial^{2} {\text{CI}}}}{{\partial {\text{PI}}^{2} }},\frac{{\partial^{2} {\text{CI}}}}{{\partial {\text{PP}}^{2} }} < 0\) when PI < 278 and PP < 690, whilst \(\frac{{\partial^{2} {\text{CI}}}}{{\partial {\text{RF}}^{2} }} < 0\) when RF < 2.54. Moreover, \(\frac{{\partial^{2} {\text{CI}}}}{{\partial j^{2} }} > 0\) for all values of \(\beta_{j}\). The derivatives show that neither are constant but rather vary with the values of \(i\) and j. The absolute change in \({\text{CI}}\) when one variable changes by one unit is thus dependent on the values of all explanatory variables in the model.
To obtain a good interpretation of the \(\beta\)-values, we can focus on the elasticity values. The above derivatives imply that \({\text{EL}}_{i} {\text{CI}} = \frac{{\partial {\text{CI}}}}{\partial i}\frac{i}{\text{CI}} = \beta_{i} \cdot i^{{\tau_{i} }}\) and \({\text{EL}}_{j} {\text{CI}} = \frac{{\partial {\text{CI}}}}{\partial j}\frac{j}{\text{CI}} = \beta_{j} \cdot j\) where \({\text{EL}}_{i} {\text{CI}}\) and \({\text{EL}}_{j} {\text{CI}}\) denote the elasticity of \({\text{CI}}\) with respect to \(i\) and \(j\), respectively. Hence, the absolute values of the elasticities of \({\text{CI}}\) with respect to \(i, i = \left\{ {{\text{PI}},{\text{PP}},{\text{RF}}} \right\}\) increase concavely with i, whereas the elasticities of \({\text{CI}}\) with respect to \(j, j = \left\{ {{\text{AN}},{\text{YP}}, \ldots } \right\}\) increase proportionally with \(j\).
From the above derivatives it also follows that \(\left( {\frac{{\frac{{\partial {\text{CI}}}}{\partial i}}}{\text{CI}}} \right) = \beta \cdot i^{{\tau_{i} - 1}}\) and \(\left( {\frac{{\frac{{\partial {\text{CI}}}}{\partial j}}}{\text{CI}}} \right) = \beta_{j} .\) Thus, an increase in i by one unit will always increase \({\text{CI}}\) by \(\left( {100 \cdot i^{\tau_{i} - 1} \cdot \beta_{i} } \right) \%\), while an increase in \(j\) by one unit will always increase \({\text{CI}}\) by approximately (\(100 \cdot \beta_{j} )\)%. Our model specification implies that the relative changes in \({\text{CI}}\) when \(i\) and j change by one unit, are independent of the values of the other explanatory variables. Moreover, the relative changes in j, \(j = \left\{ {{\text{AN}},{\text{YP}},{\text{WT}}} \right\}\) are independent of their own values whereas the relative changes in i, \(i = \left\{ {{\text{PI}},{\text{PP}},{\text{RF}}} \right\}\) decrease with their own values.
Equation (2) implicitly defines \({\text{PP}}\) as a function of \({\text{PI}}\) for given values of \({\text{CI}} ({\text{CI}}^{*} )\) and of the other explanatory variables (\({\text{AN}}^{*} ,{\text{RF}}^{*} ,{\text{WT}}^{*} ,{\text{YP}}^{*} ,{\text{TR}}^{*} ,{\text{TA}}^{*} ,{\text{TB}}^{*} ,{\text{TS}}^{*} )\); that is
in which q = \(\beta_{0} + \beta_{\text{RF}} {\text{RF}}^{0.37*} + \beta_{\text{AN}} {\text{AN}}^{*} + \beta_{\text{WT}} {\text{WT}}^{*} + \beta_{\text{YP}} {\text{YP}}^{*} + \beta_{\text{TR}} {\text{TR}}^{*} + \beta_{\text{PA}} {\text{PA}}^{*} +\, \beta_{\text{PB}} {\text{PB}}^{*} + \beta_{\text{TS}} {\text{TS}}^{*}\)
From the F(PI) relationship above we can deduce the marginal rate of substitution \(\left( {\text{MRT}} \right)\) between \({\text{PP}}\) and \({\text{PI}}\):
Rights and permissions
About this article
Cite this article
Hanssen, TE.S., Jørgensen, F. & Larsen, B. The relation between the quality of research, researchers’ experience, and their academic environment. Scientometrics 114, 933–950 (2018). https://doi.org/10.1007/s11192-017-2580-y
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-017-2580-y