Responsible Use of Bibliometrics Information

Bibliometrics offers valuable information to support research management and research evaluation. However, this information should be used responsibly. The following eight principles can serve as guidelines for the responsible use of bibliometrics information.

1. Be aware of the limited coverage of bibliometric data sources: Bibliometric data sources such as Web of Science and Scopus offer only limited coverage of the scientific and scholarly literature. International journals in the sciences are typically well covered, but coverage of national journals, journals in the social sciences and humanities, and conference proceedings and books are much more limited. One should be aware of these limitations when performing bibliometric analyses.

2. Acknowledge the importance of accurate data collection: Accurate data collection is essential for high-quality bibliometric analyses. Poor data collection may result in bibliometric analyses that provide incorrect or misleading conclusions. The efforts needed for accurate data collection should not be underestimated. In a research evaluation context, the units under evaluation should have the opportunity to verify the data collection.

3. Recognize that bibliometric analyses capture research performance only in a partial manner: It is essential to recognize that bibliometric analyses reflect only specific aspects of the performance of research units. For instance, citation statistics provide insight into the scientific impact of research units, but they do not capture the broader societal impact. Even scientific impact is captured only partly by citation statistics since these statistics consider the only scientific impact that results from publications. Scientific impact resulting from other activities, such as curation of data sets and development of software tools, is not taken into account. Furthermore, bibliometric analyses focus on the outputs of the research process and typically do not consider the inputs. This for instance means that bibliometric analyses provide no insight into the productivity of research units.

4. Account for differences between scientific fields in publication, authorship, and citation practices: Different scientific fields have different publication, authorship, and citation practices. Publications in the life sciences for instance tend to have more authors and tend to receive more citations than publications in the social sciences. In bibliometric analyses that cover multiple fields, differences between fields should be carefully accounted for. This can be done by normalizing bibliometric indicators for field differences or by comparing research units to relevant benchmarks active in the same field.

5. Find an appropriate balance between transparency and analytical sophistication: Transparency of a bibliometric analysis helps to ensure that the analysis is interpreted correctly and facilitates a well-informed discussion about the outcomes of the analysis. Analytical sophistication, for instance, the use of advanced field[1]normalized bibliometric indicators, has the potential to provide insights that are hard to obtain using more straightforward bibliometric approaches. However, an increase in analytical sophistication often causes a decrease in transparency. Bibliometric analyses, therefore, require a careful trade-off between transparency and analytical sophistication.

6. Embrace the value of multidimensional and contextualized bibliometrics: Bibliometrics is sometimes used as a tool for making one-dimensional performance measurements. When bibliometrics is used in this way, its value is limited. To take full advantage of bibliometric information, a multidimensional and contextualized approach to bibliometrics needs to be adopted. Such an approach recognizes that research management and research evaluation benefit from being supported by diverse types of bibliometric information. It also recognizes the necessity of contextualizing bibliometric information (e.g., by explicitly linking the information to the underlying data) in order to enable in-depth interpretation of the information.

7. Use bibliometrics as part of a broader range of information sources: Bibliometrics offers just one source of information to support research management and research evaluation. There are other sources of information as well. In addition to quantitative sources, such as data on research funding and research staff and altimetric data, this also includes peer review by scientific experts. The use of bibliometrics should be considered within this broader framework. The best way to support research management and research evaluation typically is to combine bibliometric information with other information sources.

8. Anticipate the effects of bibliometric analyses on the science system: The use of bibliometrics in research management and research evaluation is likely to influence the behavior of researchers and other actors in the science system. These actors may change their behavior both in intended and unintended ways. It is important to anticipate these effects of the use of bibliometrics and to be aware that a strong reliance on bibliometrics may have undesirable consequences (e.g., researchers trying to improve their citation statistics in questionable ways).

Reference: Hicks, D., Wouters, P., Waltman, L., De Rijcke, S., & Rafols, I. (2015). The Leiden Manifesto for research metrics. Nature, 520, 429–431.