Introduction

Academic science is big business and big money. Billions of US-dollars (USD), Euros and other currencies are channeled into academic science every year. As a matter of fact, the decision makers—politicians and career officials—want to know about the fate of funding: Did it work? What was done? How much was done? Who did it? Who did the most?

In order to answer these questions, the field of scientometrics and bibliometric offer convenient but debated benchmarking parameters including the impact factor of a scientific journal. Since the journal impact factor is a very superficial measure with no direct relation to the quality of a single scientific work (Carey 2016; Casadevall and Fang 2014) but only providing information about the performance of a specific journal over a relatively short term, other more sophisticated parameters were developed such as the Hirsch (H) index (Hirsch 2005, 2007). They also include a count of the individual citations that a scientific article receives. However, also the H-index is debatable and should not to be used for any purpose (Bertoli-Barsotti and Lando 2017a, b; Bornmann and Leydesdorff 2018). In this respect, experts in the field have coined the expression of amateur bibliometricians describing the uncritical use of bibliometric tools (Bornmann and Leydesdorff 2014).

With these benchmarking options at hand, other questions arise for decision makers: Can we allocate the funding towards a direction that those who did most—get more funding in order to increase their productivity? To which extend can we do so? Is there a ceiling effect? I.e. by which extend is the total (not relative) productivity increased, if we allocate 2 staff positions to a research group which consist of 2 scientists (making a total of 4 scientists then) in comparison to the allocation of 2 staff positions to a group consisting of 20 scientists or of 50 scientists (making it 22 or 52, respectively). Measured in citations? Or in accumulated impact factors or whatsoever? These questions are linked to the so called Matthew effect (Merton 1968): Those who have most get even more. When counting, measuring and benchmarking are done by the use of superficial parameters in an uncritical way by amateur bibliometricians, and (intramural and extramural) funding is allocated on the basis of who performs best in those superficial counts (i.e. total accumulated impact factor count) these questions critically target freedom of research: Scientists or fields who do not produce measurable amounts of superficial parameters such as accumulated journal impact factors will suffer (Lowy 1997).

Further to the question of funding allocation, also career opportunities are critically dependent on bibliometric benchmarking processes and it is common (but critically debated) law: publish or perish (in high impact factor journals) (Jokstad 2016; Publish or perish 2015; Bergquist et al. 2018). Taking these aspects into account, it is obvious that scientometric markers need to be used only with great caution. They should not be easily used to compare scientists of different ages and different fields, institutions or areas with the purpose to cut off funding since research should only be interpreted for quality on the individual level of a published piece of work.

Still, bibliometric parameters can be used to assess gross information contents and evolution of scientific fields over longer periods of time.

It was exactly this purpose when in the years 2005–2009 a new project was started at the Charité in Berlin (Borger et al. 2008; Groneberg-Kloft et al. 2008a, b, 2009a, d, e): A platform termed NewQIS (1.0) was constructed to establish a new approach to visualize research quantity and quality indices (Groneberg-Kloft et al. 2009b, c). NewQIS 1.0 should be used to assess research activities for (1) distinct areas of science, for (2) single institutions, for (3) single countries, or for (4) single time periods (Fig. 1).

Fig. 1
figure 1

The NewQIS platform can visualize research parameters for a different areas of science, for b different institutions, for c different countries, or for d different periods of time. A multitude of parameters can be assessed

Full size image

The platform was intended to be a sound basis for future NewQIS studies in all areas of medicine and science. In the following, we (a) briefly summarize the technical basis, and (b) present an overview of the studies and MD theses which were performed on the basis of NewQIS.

Technical platform

One important aspect of NewQIS 1.0 was to establish a unified technical platform that enables researchers from different fields of science to be able to assess their area of interest. Therefore, a study panel was formed that decided upon the feasibility of the proposed area. The usual applicants were medical students who—in their duty to conduct an MD thesis—submitted search topics to the study panel. After review and affirmation, the NewQIS analysis were performed and raw data was transferred to the applicants for their purpose.

Data acquisition

In NewQIS studies, data is usually retrieved from the Web of Science (WoS) database, i.e. (Kusma et al. 2009). The reason to choose WoS was the ability to perform a citation analysis. This was not possible with PubMed data files.

Depending on the topic of the NewQIS study, the search terms that are entered in the search field consist of various terms which are linked together with Boolean operators such as “AND”, “OR”, “NOT”, i.e. (Glynn et al. 2010).

Depending on the date of the research, the amount of publications and the focus of the research, the evaluation time span covers periods from 1900 until today. Usually, the year in which the NewQIS project is performed, is left out because of incomplete data acquisitions for that given year, i.e. (Al-Mutawakel et al. 2010).

Parameters

The large majority of NewQIS projects focus on a single field of medicine such as a disease and put a focus on the global landscape of research on this particular disease. Thus, the following parameters are usually analyzed (Fig. 1).


Quantity parameters: Productivity

  • Total number of published items (i.e. Scutaru et al. 2010b)

  • Country specific number of publishes items (i.e. Vitzthum et al. 2010a)

(Semi-)qualitative parameters: Usually, high quality research is characterized by a high number of citations. Therefore, the following citation parameters were also analyzed in the NewQIS projects:

  • Total number of citations

  • Total number of citations per country

  • Country-specific h index

  • Country-specific average citation rate per article

Cooperation parameters: A key instrument of NewQIS is to visualize different levels of collaboration. This includes either collaborations between single scientists, countries or institutions. The field of RSV (respiratory syncytial virus) research can give an example how this is achieved: After having identified all relevant RSV-associated publications, the collaborative studies were related to their countries of origin. Publications with two or more authors affiliated to the same country were counted only once for the total number of collaborations of this particular country (Bruggmann et al. 2017c). If an author had two affiliations, these were counted for every country mentioned in the affiliations. Connecting vectors visualized these co-operations; their width and shade of grey reflected the number of joint publications (Bruggmann et al. 2017c). Figure 2 illustrates international collaborations for RSV research.

Fig. 2
figure 2

https://bmjopen.bmj.com/content/7/7/e013615.long, Data from Bruggmann et al. (2017c)

International collaborations for RSV research. International cooperations on RSV research (threshold > 2 cooperations). Numbers in brackets report the number of publications in total/collaborative publications.

Full size image

Visualization

The above listed parameters can also be found in other publications using other approaches (Burak Atci et al. 2019; Ekundayo and Okoh 2018). A specific purpose of NewQIS was to combine these bibliometric parameters with visualization techniques in order to provide a picture of the global landscape of different research aspects. Among different available techniques, density-equalizing map projections (DEMP) were chosen. As elegantly described by Gastner and Newman, map makers searched for a long a way to generate cartograms, in which the sizes of countries appear in proportion to a chosen parameter such as their population (Gastner and Newman 2004). For the purpose of NewQIS, these maps could be used to visualize research activities. As stated by Gastner and Newman, in order to scale countries and still have them properly fit together, they need to be distorted, causing difficulties to read them. In 2004, a new method was proposed which was integrated to the NewQIS platform. With DEMPs being a part of NewQIS, the territories of countries were re-sized according to a particular variable, i.e. in proportion to the countries´ total number of published items regarding to a specific disease. Figure 3 shows examples of DEMPs published within NewQIS studies over the past decade. The distorted global landscape is usually characterized by a dominating USA and an enlarged European area as depicted in Fig. 3a for pulmonary hypertension research output (Gotting et al. 2017). However, there are also research areas in which also countries from other continental regions appear enlarged. This can be seen for snakebite envenoming research for Brazil as shown in Fig. 3b (Groneberg et al. 2016c). China—a rising star in many areas of science—does also appear in some NewQIS assessments prominently as shown for ovarian cancer research in Fig. 3c (Bruggmann et al. 2017d). Concerning Asian countries, a previous assessment of 5527,558 articles has indicated that Asian countries have largely different research focuses in comparison to Western countries (Groneberg-Kloft et al. 2008b). In order to assess changes over the time, spatiotemporal analyses can also be performed by merging to a video consisting of different density-equalizing mapping (Groneberg-Kloft et al. 2009e).

Fig. 3
figure 3

Density equalizing map projections (DEMP). a DEMP for pulmonary hypertension research output. Data from Gotting et al. (2017). b DEMP for snakebite envenoming research output. Data from Groneberg et al. (2016c). c DEMP exemplifying prominent Chinese research activities in ovarian cancer research. Data from Bruggmann et al. (2017d)

Full size image

Topics of NewQIS

Structured MD thesis program

The original concept of NewQIS was a low budget intramural platform which was established without major external funding. In order to be able to assess numerous fields of medicine, medical students were enabled to conduct their MD thesis within the NewQIS platform. The highly structured boundaries of the platform also served as a quality control for the results of the thesis projects making scientific misconduct very difficult (since there was no possibility for the students to manipulate the algorithms applied by the platform).

Since 2009, nearly 80 theses were completed using the methodology of the platform making NewQIS one of the most successful structured thesis programs in Germany. As tutors/mentors of the theses, seven associate/full professors served so far. Also, two technical tutors were present to oversee calculations and data management. Table 1 lists the medical thesis topics.

Table 1 Medical thesis projects that applied the structured NewQIS program
Full size table

Scientific publications

Since 2008, more than 80 studies using the NewQIS platform were published after peer review. The majority of them based on medical thesis projects with the MD students being first authors in case of writing the manuscripts or co-authors of the scientific studies. The topics ranged from infectious diseases, infectious agents, to cancers, neurological or psychiatric disorders, lung diseases or other diseases. Apart from diseases, they also encompassed i.e. public health issues including tobacco control, medical procedures or techniques. In total, more than 1.6 million published articles related to specific search terms were analyzed for the above listed parameters. Table 2 provides an overview of the different NewQIS articles.

Table 2 Scientific publications applying the NewQIS platform
Full size table

Limitations of NewQIS

There are numerous limitations present in every NewQIS-based study:

  1. (1)

    As with every other bibliometric approach, also NewQIS is limited to the data base it uses. Although producing global landscapes of research, it should never be forgotten that these pictures only delineate the research output which can be found in a specific data base (i.e. Web of Science) with a specific search term. Thus, all research not listed in the WoS and all research excluded by the search term (no search term can be absolutely perfect) is not included in the global landscape. This needs to be taken into account carefully when NewQIS results are interpreted. Especially the language bias constitutes an important problem: journals published in English have a higher chance of getting included to the data bases (Nieminen and Isohanni 1999). Thus, non-English speaking countries are underrepresented concerning their research activities and important but regional data such as regional epidemiologic data is not identified (Pleger et al. 2014).

  2. (2)

    A further limitation that needs to be addressed is the above Matthew effect mentioned above: Communication systems in science are directed towards a reward of highly productive and renowned scientists and institutions. This leads to a pyramidal citation scheme (Merton 1968; Pleger et al. 2014).

  3. 3)

    The so-called (semi-)qualitative indicators that are used in NewQIS are parameters such as the total citations, citation rate, country-specific h-index. They need to be interpreted very carefully. As already earlier critically discussed, they are not real measures for the quality of individual research (Pleger et al. 2014). In this respect, a recent study addressed the question if methodological quality and completeness of reporting are associated with citation-based measures of publication impact (Mackinnon et al. 2018). The authors performed a secondary analysis of a systematic review of dementia biomarker studies. They reported that citation rates and 5-year journal impact factors appear to measure different dimensions. While citation rates were weakly associated with completeness of reporting, none of these metrics was related to methodological rigor. They suggested that high publication usage and journal outlet is not a guarantee of quality and readers should critically appraise all papers regardless of presumed impact (Mackinnon et al. 2018). Therefore, qualitative aspects are better addressed by advanced meta-analysis approaches using i.e. Cochrane systematics (Stovold et al. 2014).

Further issues

Scientometrics as research area is a niche within science. Funding is difficult to acquire for scientometric projects. However, it is the long term aim of NewQIS to analyze about 200 different areas within the next decade and to repeat assessments in 5- to 10-year intervals of important areas in order to assess changes in global research activities. When counting the raw data analyzed in the first 100 projects, we approximately invested about 50,000 work hours. Without extramural funding, this was only achievable by the workforce of medical students who performed their MD projects within NewQIS. In contrast to peer reviewed scientific reports which have been published for different NewQIS studies, a German medical thesis usually encompasses a much longer manuscript with 80–100 pages. This has been achieved by the medical students by writing comprehensive introductions about the field of research they analyze within their thesis. Thereby, they demonstrate that they possess an extensive knowledge about their thesis project. This is a prerequisite to obtain an MD degree. Also, the thesis students have to write detailed descriptions of their methodological approach (the NewQIS techniques) in the methods sections of the thesis and they have to discuss limits of the methodology in the discussion sections of their thesis.


This leads to two potential pitfalls:

  1. (1)

    In the case of the methods sections, the thesis students have to follow strictly the above described protocols of NewQIS. This technical overlap is important and a strength of the platform in order to facilitate the comparison of results between the different diseases studied. However, it can be anticipated, that the use of these stringent protocols in nearly 80 different thesis projects—all with different target areas, i.e. ranging from burnout syndrome (Fröhlich 2009) to bronchial asthma (Puk 2009)—brings the same problem as rewriting a passage on the methodology of other highly structured techniques such as RT-PCR (reverse transcriptase-polymerase chain reaction) which has now been published more than 250,000 times according to the PubMed. As with nearly identical descriptions of PCR and other molecular biology methods which can be found in peer reviewed scientific papers, an overlapping wording does not represent an act of plagiarism but rather exemplifies the impossibility to reword a similar methods section for more than 80 times without overlapping sentences. This does also apply for the part of the discussion in which the methodology and its limitations are discussed. Addressing these issues, the international Committee on Publication Ethics (COPE) points to a guideline of BioMed Central editors which outlines the following: “Use of similar or identical phrases in methods sections where there are limited ways to describe a common method, (…), is not uncommon. In such cases, an element of text recycling is likely to be unavoidable in further publications using the same method. Editors should use their discretion when deciding how much overlap of methods text is acceptable, considering factors such as whether authors have been transparent and stated that the methods have already been described in detail elsewhere and provided a citation” (COPE) (https://publicationethics.org/text-recycling-guidelines). Therefore, to overcome this pitfall, peer-reviewed NewQIS studies cite previous studies because of the methodological similarities—which are a strength of the platform. Also, thesis students are urged to cite every other NewQIS thesis which used the platform and to declare that the used methodology is part of NewQIS and therefore similar (apart from i.e. the different search terms).

  2. (2)

    The introductions of the respective thesis usually follow the guidelines of up-to-date reviews i.e. on the disease which is analyzed for the thesis. In this respect, numerous introductions from NewQIS related thesis projects were also published as CME (continuing medical education) articles or as narrative reviews. Unfortunately, a recent analysis showed that within one thesis project, almost all parts of the introduction were copied by the student from the Wikipedia—a case of severe plagiarism that led to the deprivation of the Dr. med. degree (MD Thesis) of the student (Sudik 2011). In order to prevent future cases of plagiarism, all medical thesis now need to be analyzed within a plagiarism check prior to the official submission of the thesis to the medical school.

Future of NewQIS

The NewQIS platform will be used as NewQIS 2.0 in a next decade of further scientometric studies. There will be the following issues:

Project of 200 As stated earlier, NewQIS 2.0 is intended to encompass about 200 different search projects with all areas of medicine, life sciences and also other areas of science in the next 10 years. Also, projects carried out 10 years ago and reported worldwide research activities (in the Web of Science) until 2005, should now be repeated in order to investigate the development of scientific activities.

New focuses Originally conceived as a tool to investigate publication activities in single areas of medicine, i.e. in different infectious diseases, NewQIS has also proven to be a valuable tool for other purposes, i.e. to analyze journals (Scutaru et al. 2010b; Groneberg et al. 2018). Also, it could be used for the analysis of cities with regard to research activities of affiliations in these cities. A recent example was the so-called NewQIS-Wroclaw project that assessed scientific activities in the Central European Polish city in three different areas: biomedical research, chemical research and social sciences (Groneberg 2018a, b, 2019) and demonstrated a strong increase over the past decades.

New parameters As introduced in the past years, NewQIS studies may also focus upon socio-economic features. In this respect, various economic key figures were used. I.e. two quotients were calculated to assess the scientific output of a specific country for RSV research (Bruggmann et al. 2017c):

  1. (1)

    in relation to the number of inhabitants (Q1)

  2. (2)

    in relation to its economic power (as measured by the gross domestic product, GDP, Q2) (Bruggmann et al. 2017c). Data regarding the population and GDP of investigated countries was obtained from 2012 The quotients were calculated as follows:

    1. 1.

      Articles/population index (Q1) = number of articles/population in million inhabitants

    2. 2.

      Articles/GDP index (Q2) = number of articles/GDP in 1000 billion US-Dollars

Within the RSV research NewQIS study, also, all countries were classified into high-income, upper-middle-income, lower-middle-income and low-income groups according to World Bank definitions (Bruggmann et al. 2017c). Then, the total number of RSV articles was related to the gross domestic expenditure on Research and Development (R&D in % of GDP) as well as to the number of researchers (per million inhabitants) affiliated to the investigated countries.

Conclusion

For over 10 years, the NewQIS platform has been used as a tool for peer reviewed scientific studies and for medical thesis in order to study numerous fields of science. As NewQIS 2.0 the project now heads into the next decade with a variety of new aspects in focus such as detailed socio-economic analysis or gender aspects. Using density equalizing mapping projections thousands of new pictures of global research landscapes will be generated. With numerous novel aspects that have been introduced to NewQIS within the past years, the platform will be a helpful tool for different aspects of scientometrics in the future.