arXiv subjects

Ludo Waltman

Publications and source records attributed to Ludo Waltman.

At least 19 recordsLinked to original sources

arXiv2025

A comparison of citation-based clustering and topic modeling for science mapping

Understanding the different ways in which different science mapping approaches capture the structure of scientific fields is critical. This paper presents a comparative analysis of two commonly used approaches, topic modeling (TM) and citation-based clustering (CC), to assess their respective strengths, weaknesses, and the characteristics of their results. We compare the two approaches using cluster-to-topic and topic-to-cluster mappings based on science maps of cardiovascular research generated by TM and CC. Our findings reveal that relations between topics and clusters are generally weak, with limited overlap between topics and clusters. Only in a few exceptional cases do more than one-third of the documents in a topic belong to the same cluster, or vice versa. For TM the presence of highly similar topics is a considerable challenge. A strength of TM is its ability to represent societal needs related to cardiovascular disease, potentially offering valuable insights for policymakers. In contrast, CC excels in depicting the intellectual structure of cardiovascular diseases, with a strong capability to reflect scientific micro-communities. This study deepens the understanding of the use of TM and CC for science mapping, providing insights for users on how to apply these approaches based on their needs.

cs.DL↗

arXiv2024

Which topics are best represented by science maps? An analysis of clustering effectiveness for citation and text similarity networks

A science map of topics is a visualization that shows topics identified algorithmically based on the bibliographic metadata of scientific publications. In practice not all topics are well represented in a science map. We analyzed how effectively different topics are represented in science maps created by clustering biomedical publications. To achieve this, we investigated which topic categories, obtained from MeSH terms, are better represented in science maps based on citation or text similarity networks. To evaluate the clustering effectiveness of topics, we determined the extent to which documents belonging to the same topic are grouped together in the same cluster. We found that the best and worst represented topic categories are the same for citation and text similarity networks. The best represented topic categories are diseases, psychology, anatomy, organisms and the techniques and equipment used for diagnostics and therapy, while the worst represented topic categories are natural science fields, geographical entities, information sciences and health care and occupations. Furthermore, for the diseases and organisms topic categories and for science maps with smaller clusters, we found that topics tend to be better represented in citation similarity networks than in text similarity networks.

cs.DL↗

arXiv2023

Academic information retrieval using citation clusters: In-depth evaluation based on systematic reviews

The field of scientometrics has shown the power of citation-based clusters for literature analysis, yet this technique has barely been used for information retrieval tasks. This work evaluates the performance of citation based-clusters for information retrieval tasks. We simulated a search process using these clusters with a tree hierarchy of clusters and a cluster selection algorithm. We evaluated the task of finding the relevant documents for 25 systematic reviews. Our evaluation considered several trade-offs between recall and precision for the cluster selection, and we also replicated the Boolean queries self-reported by the systematic review to serve as a reference. We found that citation-based clusters search performance is highly variable and unpredictable, that it works best for users that prefer recall over precision at a ratio between 2 and 8, and that when used along with query-based search they complement each other, including finding new relevant documents.

cs.DL↗

arXiv2022

Funding Covid-19 research: Insights from an exploratory analysis using open data infrastructures

To analyse the outcomes of the funding they provide, it is essential for funding agencies to be able to trace the publications resulting from their funding. We study the open availability of funding data in Crossref, focusing on funding data for publications that report research related to Covid-19. We also present a comparison with the funding data available in two proprietary bibliometric databases: Scopus and Web of Science. Our analysis reveals a limited coverage of funding data in Crossref. It also shows problems related to the quality of funding data, especially in Scopus. We offer recommendations for improving the open availability of funding data in Crossref.

cs.DL↗

arXiv2021

Investigating Disagreement in the Scientific Literature

Disagreement is essential to scientific progress. However, the extent of disagreement in science, its evolution over time, and the fields in which it happens, remains poorly understood. Leveraging a massive collection of English-language scientific texts, we develop a cue-phrase based approach to identify instances of disagreement citations across more than four million scientific articles. Using this method, we construct an indicator of disagreement across scientific fields over the 2000-2015 period. In contrast with black-box text classification methods, our framework is transparent and easily interpretable. We reveal a disciplinary spectrum of disagreement, with higher disagreement in the social sciences and lower disagreement in physics and mathematics. However, detailed disciplinary analysis demonstrates heterogeneity across sub-fields, revealing the importance of local disciplinary cultures and epistemic characteristics of disagreement. Paper-level analysis reveals notable episodes of disagreement in science, and illustrates how methodological artifacts can confound analyses of scientific texts. These findings contribute to a broader understanding of disagreement and establish a foundation for future research to understanding key processes underlying scientific progress.

cs.DL↗

arXiv2021

A large-scale bibliometric analysis of global climate change research between 2001 and 2018

Global climate change is attracting widespread scientific, political, and public attention owing to the involvement of international initiatives such as the Paris Agreement and the Intergovernmental Panel on Climate Change. We present a large-scale bibliometric analysis based on approximately 120,000 climate change publications between 2001 and 2018 to examine how climate change is studied in scientific research. Our analysis provides an overview of scientific knowledge, shifts of research hotspots, global geographical distribution of research, and focus of individual countries. In our analysis, we identify five key fields in climate change research: physical sciences, paleoclimatology, climate-change ecology, climate technology, and climate policy. We draw the following key conclusions: (1) Over the investigated time period, the focus of climate change research has shifted from understanding the climate system toward climate technologies and policies, such as efficient energy use and legislation. (2) There is an imbalance in scientific production between developed and developing countries. (3) Geography, national demands, and national strategies have been important drivers that influence the research interests and concerns of researchers in different countries. Our study can be used by researchers and policy makers to reflect on the directions in which climate change research is developing and discuss priorities for future research.

cs.DL↗

arXiv2021

Gender differences in scientific careers: A large-scale bibliometric analysis

We present a large-scale bibliometric analysis of gender differences in scientific careers, covering all scientific disciplines and a large number of countries worldwide. We take a longitudinal perspective in which we trace the publication careers of almost six million male and female researchers in the period 1996-2018. Our analysis reveals an increasing trend in the percentage of women starting a career as publishing researcher, from 33% in 2000 to about 40% in recent years. Looking at cohorts of male and female researchers that started their publication career in the same year, we find that women seem to be somewhat less likely to continue their career as publishing researcher than men, but the difference is small. We also observe that men produce on average between 15% and 20% more publications than women. Moreover, in biomedical disciplines, men are about 25% more likely than women to be last author of a publication, suggesting that men tend to have more senior roles than women. Compared with cross-sectional studies, our longitudinal analysis has the advantage of providing a more in-depth understanding of gender imbalances among authors of scientific publications.

cs.DL↗

arXiv2021

Use of the journal impact factor for assessing individual articles: Statistically flawed or not?

Most scientometricians reject the use of the journal impact factor for assessing individual articles and their authors. The well-known San Francisco Declaration on Research Assessment also strongly objects against this way of using the impact factor. Arguments against the use of the impact factor at the level of individual articles are often based on statistical considerations. The skewness of journal citation distributions typically plays a central role in these arguments. We present a theoretical analysis of statistical arguments against the use of the impact factor at the level of individual articles. Our analysis shows that these arguments do not support the conclusion that the impact factor should not be used for assessing individual articles. Using computer simulations, we demonstrate that under certain conditions the number of citations an article has received is a more accurate indicator of the value of the article than the impact factor. However, under other conditions, the impact factor is a more accurate indicator. It is important to critically discuss the dominant role of the impact factor in research evaluations, but the discussion should not be based on misplaced statistical arguments. Instead, the primary focus should be on the socio-technical implications of the use of the impact factor.

cs.DL↗

arXiv2021

Large-scale comparison of bibliographic data sources: Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic

We present a large-scale comparison of five multidisciplinary bibliographic data sources: Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic. The comparison considers scientific documents from the period 2008-2017 covered by these data sources. Scopus is compared in a pairwise manner with each of the other data sources. We first analyze differences between the data sources in the coverage of documents, focusing for instance on differences over time, differences per document type, and differences per discipline. We then study differences in the completeness and accuracy of citation links. Based on our analysis, we discuss strengths and weaknesses of the different data sources. We emphasize the importance of combining a comprehensive coverage of the scientific literature with a flexible set of filters for making selections of the literature.

cs.DL↗

arXiv2020

Algorithmic labeling in hierarchical classifications of publications: Evaluation of bibliographic fields and term weighting approaches

Algorithmic classifications of research publications can be used to study many different aspects of the science system, such as the organization of science into fields, the growth of fields, interdisciplinarity, and emerging topics. How to label the classes in these classifications is a problem that has not been thoroughly addressed in the literature. In this study we evaluate different approaches to label the classes in algorithmically constructed classifications of research publications. We focus on two important choices: the choice of (1) different bibliographic fields and (2) different approaches to weight the relevance of terms. To evaluate the different choices, we created two baselines: one based on the Medical Subject Headings in MEDLINE and another based on the Science-Metrix journal classification. We tested to what extent different approaches yield the desired labels for the classes in the two baselines. Based on our results we recommend extracting terms from titles and keywords to label classes at high levels of granularity (e.g. topics). At low levels of granularity (e.g. disciplines) we recommend extracting terms from journal names and author addresses. We recommend the use of a new approach, term frequency to specificity ratio, to calculate the relevance of terms.

cs.DL↗

arXiv2020

A multi-dimensional framework for characterizing the citation impact of scientific publications

The citation impact of a scientific publication is usually seen as a one-dimensional concept. We introduce a multi-dimensional framework for characterizing the citation impact of a publication. In addition to the level of citation impact, quantified by the number of citations received by a publication, we also conceptualize and operationalize the depth and breadth and the dependence and independence of the citation impact of a publication. The proposed framework distinguishes between publications that have a deep citation impact, typically in a relatively narrow research area, and publications that have a broad citation impact, probably covering a wider area of research. It also makes a distinction between publications that are strongly dependent on earlier work and publications that make a more independent scientific contribution. We use our multi-dimensional citation impact framework to report basic descriptive statistics on the citation impact of highly cited publications in all scientific disciplines. In addition, we present a detailed case study focusing on the field of scientometrics. The proposed citation impact framework provides a more in-depth understanding of the citation impact of a publication than a traditional one-dimensional perspective.

cs.DL↗

arXiv2019

Intermediacy of publications

Citation networks of scientific publications offer fundamental insights into the structure and development of scientific knowledge. We propose a new measure, called intermediacy, for tracing the historical development of scientific knowledge. Given two publications, an older and a more recent one, intermediacy identifies publications that seem to play a major role in the historical development from the older to the more recent publication. The identified publications are important in connecting the older and the more recent publication in the citation network. After providing a formal definition of intermediacy, we study its mathematical properties. We then present two empirical case studies, one tracing historical developments at the interface between the community detection literature and the scientometric literature and one examining the development of the literature on peer review. We show both conceptually and empirically how intermediacy differs from main path analysis, which is the most popular approach for tracing historical developments in citation networks. Main path analysis tends to favor longer paths over shorter ones, whereas intermediacy has the opposite tendency. Compared to main path analysis, we conclude that intermediacy offers a more principled approach for tracing the historical development of scientific knowledge.

cs.DL↗

arXiv2019

From Louvain to Leiden: guaranteeing well-connected communities

Community detection is often used to understand the structure of large and complex networks. One of the most popular algorithms for uncovering community structure is the so-called Louvain algorithm. We show that this algorithm has a major defect that largely went unnoticed until now: the Louvain algorithm may yield arbitrarily badly connected communities. In the worst case, communities may even be disconnected, especially when running the algorithm iteratively. In our experimental analysis, we observe that up to 25% of the communities are badly connected and up to 16% are disconnected. To address this problem, we introduce the Leiden algorithm. We prove that the Leiden algorithm yields communities that are guaranteed to be connected. In addition, we prove that, when the Leiden algorithm is applied iteratively, it converges to a partition in which all subsets of all communities are locally optimally assigned. Furthermore, by relying on a fast local move approach, the Leiden algorithm runs faster than the Louvain algorithm. We demonstrate the performance of the Leiden algorithm for several benchmark and real-world networks. We find that the Leiden algorithm is faster than the Louvain algorithm and uncovers better partitions, in addition to providing explicit guarantees.

cs.SI↗

arXiv2019

A principled methodology for comparing relatedness measures for clustering publications

There are many different relatedness measures, based for instance on citation relations or textual similarity, that can be used to cluster scientific publications. We propose a principled methodology for evaluating the accuracy of clustering solutions obtained using these relatedness measures. We formally show that the proposed methodology has an important consistency property. The empirical analyses that we present are based on publications in the fields of cell biology, condensed matter physics, and economics. Using the BM25 text-based relatedness measure as evaluation criterion, we find that bibliographic coupling relations yield more accurate clustering solutions than direct citation relations and co-citation relations. The so-called extended direct citation approach performs similarly to or slightly better than bibliographic coupling in terms of the accuracy of the resulting clustering solutions. The other way around, using a citation-based relatedness measure as evaluation criterion, BM25 turns out to yield more accurate clustering solutions than other text-based relatedness measures.

cs.DL↗

arXiv2019

Accuracy of citation data in Web of Science and Scopus

We present a large-scale analysis of the accuracy of citation data in the Web of Science and Scopus databases. The analysis is based on citations given in publications in Elsevier journals. We reveal significant data quality problems for both databases. Missing and incorrect references are important problems in Web of Science. Duplicate publications are a serious problem in Scopus.

cs.DL↗

arXiv2019

Systematic analysis of agreement between metrics and peer review in the UK REF

When performing a national research assessment, some countries rely on citation metrics whereas others, such as the UK, primarily use peer review. In the influential Metric Tide report, a low agreement between metrics and peer review in the UK Research Excellence Framework (REF) was found. However, earlier studies observed much higher agreement between metrics and peer review in the REF and argued in favour of using metrics. This shows that there is considerable ambiguity in the discussion on agreement between metrics and peer review. We provide clarity in this discussion by considering four important points: (1) the level of aggregation of the analysis; (2) the use of either a size-dependent or a size-independent perspective; (3) the suitability of different measures of agreement; and (4) the uncertainty in peer review. In the context of the REF, we argue that agreement between metrics and peer review should be assessed at the institutional level rather than at the publication level. Both a size-dependent and a size-independent perspective are relevant in the REF. The interpretation of correlations may be problematic and as an alternative we therefore use measures of agreement that are based on the absolute or relative differences between metrics and peer review. To get an idea of the uncertainty in peer review, we rely on a model to bootstrap peer review outcomes. We conclude that particularly in Physics, Clinical Medicine, and Public Health, metrics agree quite well with peer review and may offer an alternative to peer review.

cs.DL↗

arXiv2018

Analyzing the activities of visitors of the Leiden Ranking website

To provide a better understanding of the way in which university rankings are used, we present a detailed analysis of the activities of visitors of a university ranking website. We use the website of the CWTS Leiden Ranking for this purpose. We for instance study the countries from which visitors originate, the specific pages on the Leiden Ranking website that they visit, the countries or the universities that they find of special interest, and the indicators that they focus on. In addition, we also discuss two experiments that were carried out on the Leiden Ranking website. Our analysis does not only provide new insights into the use of university rankings, but it also suggests possible ways in which these rankings can be improved.

cs.DL↗

arXiv2018

Exploration of reproducibility issues in scientometric research Part 1: Direct reproducibility

This is the first part of a small-scale explorative study in an effort to start assessing reproducibility issues specific to scientometrics research. This effort is motivated by the desire to generate empirical data to inform debates about reproducibility in scientometrics. Rather than attempt to reproduce studies, we explore how we might assess "in principle" reproducibility based on a critical review of the content of published papers. The first part of the study focuses on direct reproducibility - that is the ability to reproduce the specific evidence produced by an original study using the same data, methods, and procedures. The second part (Velden et al. 2018) is dedicated to conceptual reproducibility - that is the robustness of knowledge claims towards verification by an alternative approach using different data, methods and procedures. The study is exploratory: it investigates only a very limited number of publications and serves us to develop instruments for identifying potential reproducibility issues of published studies: These are a categorization of study types and a taxonomy of threats to reproducibility. We work with a select sample of five publications in scientometrics covering a variation of study types of theoretical, methodological, and empirical nature. Based on observations made during our exploratory review, we conclude this paper with open questions on how to approach and assess the status of direct reproducibility in scientometrics, intended for discussion at the special track on "Reproducibility in Scientometrics" at STI2018 in Leiden.

cs.DL↗