Searcharxiv⌕ Search

arXiv subjects

Lutz Bornmann

Publications and source records attributed to Lutz Bornmann.

At least 55 records · Page 3Linked to original sources
arXiv2020

Are papers addressing certain diseases perceived where these diseases are prevalent? The proposal to use Twitter data as social-spatial sensors

We propose to use Twitter data as social-spatial sensors. This study deals with the question whether research papers on certain diseases are perceived by people in regions (worldwide) that are especially concerned by the diseases. Since (some) Twitter data contain location information, it is possible to spatially map the activity of Twitter users referring to certain papers (e.g., dealing with tuberculosis). The resulting maps reveal whether heavy activity on Twitter is correlated with large numbers of people having certain diseases. In this study, we focus on tuberculosis, human immunodeficiency virus (HIV), and malaria, since the World Health Organization ranks these diseases as the top three causes of death worldwide by a single infectious agent. The results of the social-spatial Twitter maps (and additionally performed regression models) reveal the usefulness of the proposed sensor approach. One receives an impression of how research papers on the diseases have been perceived by people in regions that are especially concerned by the diseases. Our study demonstrates a promising approach for using Twitter data for research evaluation purposes beyond simple counting of tweets.

cs.DL↗
arXiv2020

Which papers cited which tweets? An empirical analysis based on Scopus data

Many altmetric studies analyze which papers were mentioned how often in specific altmetrics sources. In order to study the potential policy relevance of tweets from another perspective, we investigate which tweets were cited in papers. If many tweets were cited in publications, this might demonstrate that tweets have substantial and useful content. Overall, a rather low number of tweets (n=5506) were cited by less than 3000 papers. Most tweets do not seem to be cited because of any cognitive influence they might have had on studies; they rather were study objects. Most of the papers citing tweets are from the subject areas Social Sciences, Arts and Humanities, and Computer Sciences. Most of the papers cited only one tweet. Up to 55 tweets cited in a single paper were found. This research-in-progress does not support a high policy-relevance of tweets. However, a content analysis of the tweets and/or papers might lead to a more detailed conclusion.

cs.DL↗
arXiv2019

Do disruption index indicators measure what they propose to measure? The comparison of several indicator variants with assessments by peers

Recently, Wu, Wang, and Evans (2019) and Bu, Waltman, and Huang (2019) proposed a new family of indicators, which measure whether a scientific publication is disruptive to a field or tradition of research. Such disruptive influences are characterized by citations to a focal paper, but not its cited references. In this study, we are interested in the question of convergent validity, i.e., whether these indicators of disruption are able to measure what they propose to measure ('disruptiveness'). We used external criteria of newness to examine convergent validity: in the post-publication peer review system of F1000Prime, experts assess papers whether the reported research fulfills these criteria (e.g., reports new findings). This study is based on 120,179 papers from F1000Prime published between 2000 and 2016. In the first part of the study we discuss the indicators. Based on the insights from the discussion, we propose alternate variants of disruption indicators. In the second part, we investigate the convergent validity of the indicators and the (possibly) improved variants. Although the results of a factor analysis show that the different variants measure similar dimensions, the results of regression analyses reveal that one variant (DI5) performs slightly better than the others.

cs.DL↗
arXiv2019

Do we measure novelty when we analyze unusual combinations of cited references? A validation study of bibliometric novelty indicators based on F1000Prime data

Lee, Walsh, and Wang (2015) - based on Uzzi, Mukherjee, Stringer, and Jones (2013) - and Wang, Veugelers, and Stephan (2017) proposed scores based on cited references (cited journals) data which can be used to measure the novelty of papers (named as novelty scores U and W in this study). Although previous research has used novelty scores in various empirical analyses, no study has been published up to now - to the best of our knowledge - which quantitatively tested the convergent validity of novelty scores: do these scores measure what they propose to measure? Using novelty assessments by faculty members (FMs) at F1000Prime for comparison, we tested the convergent validity of the two novelty scores (U and W). FMs' assessments not only refer to the quality of biomedical papers, but also to their characteristics (by assigning certain tags to the papers): for example, are the presented findings or formulated hypotheses novel (tags "new findings" and "hypothesis")? We used these and other tags to investigate the convergent validity of both novelty scores. Our study reveals different results for the novelty scores: the results for novelty score U are mostly in agreement with previously formulated expectations. We found, for instance, that for a standard deviation (one unit) increase in novelty score U, the expected number of assignments of the "new finding" tag increase by 7.47%. The results for novelty score W, however, do not reflect convergent validity with the FMs' assessments: only the results for some tags are in agreement with the expectations. Thus, we propose - based on our results - the use of novelty score U for measuring novelty quantitatively, but question the use of novelty score W.

cs.DL↗
arXiv2019

What Do Citation Counts Measure? An Updated Review of Studies on Citations in Scientific Documents Published between 2006 and 2018

The purpose of this paper is to update the review of Bornmann and Daniel (2008) presenting a narrative review of studies on citations in scientific documents. The current review covers 41 studies published between 2006 and 2018. Bornmann and Daniel (2008) focused on earlier years. The current review describes the (new) studies on citation content and context analyses as well as the studies that explore the citation motivation of scholars through surveys or interviews. One focus in this paper is on the technical developments in the last decade, such as the richer meta-data available and machine-readable formats of scientific papers. These developments have resulted in citation context analyses of large datasets in comprehensive studies (which was not possible previously). Many studies in recent years have used computational and machine learning techniques to determine citation functions and polarities, some of which have attempted to overcome the methodological weaknesses of previous studies. The automated recognition of citation functions seems to have the potential to greatly enhance citation indices and information retrieval capabilities. Our review of the empirical studies demonstrates that a paper may be cited for very different scientific and non-scientific reasons. This result accords with the finding by Bornmann and Daniel (2008). The current review also shows that to better understand the relationship between citing and cited documents, a variety of features should be analyzed, primarily the citation context, the semantics and linguistic patterns in citations, citation locations within the citing document, and citation polarity (negative, neutral, positive).

cs.DL↗
arXiv2019

Heuristics as conceptual lens for understanding and studying the usage of bibliometrics in research evaluation

While bibliometrics are widely used for research evaluation purposes, a common theoretical framework for conceptually understanding, empirically studying, and effectively teaching its usage is lacking. In this paper, we outline such a framework: the fast-and-frugal heuristics research program, proposed originally in the context of the cognitive and decision sciences, lends itself particularly well for understanding and investigating the usage of bibliometrics in research evaluations. Such evaluations represent judgments under uncertainty in which typically not all possible options, their consequences, and those consequences' probabilities of occurring may be known. In these situations of incomplete information, candidate descriptive and prescriptive models of human behavior are heuristics. Heuristics are simple strategies that, by exploiting the structure of environments, can aid people to make smart decisions. Relying on heuristics does not mean trading off accuracy against effort: while reducing complexity, heuristics can yield better decisions than more information-greedy procedures in many decision environments. The prescriptive power of heuristics is documented in a cross-disciplinary literature, cutting across medicine, crime, business, sports, and other domains. We outline the fast-and-frugal heuristics research program, provide examples of past empirical work on heuristics outside the field of bibliometrics, explain why heuristics may be especially suitable for studying the usage of bibliometrics, and propose a corresponding conceptual framework.

cs.DL↗
arXiv2019

Does the $h_α$ index reinforce the Matthew effect in science? Agent-based simulations using Stata and R

Recently, Hirsch (2019a) proposed a new variant of the h index called the $h_α$ index. He formulated as follows: "we define the $h_α$ index of a scientist as the number of papers in the h-core of the scientist (i.e. the set of papers that contribute to the h-index of the scientist) where this scientist is the $α$-author" (p. 673). The $h_α$ index was criticized by Leydesdorff, Bornmann, and Opthof (2019). One of their most important points is that the index reinforces the Matthew effect in science. We address this point in the current study using a recently developed Stata command (h_index) and R package (hindex), which can be used to simulate h index and $h_α$index applications in research evaluation. The user can investigate under which conditions $h_α$ reinforces the Matthew effect. The results of our study confirm what Leydesdorff et al. (2019) expected: the $h_α$ index reinforces the Matthew effect. This effect can be intensified if strategic behavior of the publishing scientists and cumulative advantage effects are additionally considered in the simulation.

cs.DL↗
arXiv2019

Citation concept analysis (CCA) - A new form of citation analysis revealing the usefulness of concepts for other researchers illustrated by two exemplary case studies including classic books by Thomas S. Kuhn and Karl R. Popper

In recent years, the full text of papers are increasingly available electronically which opens up the possibility of quantitatively investigating citation contexts in more detail. In this study, we introduce a new form of citation analysis, which we call citation concept analysis (CCA). CCA is intended to reveal the cognitive impact certain concepts -- published in a document -- have on the citing authors. It counts the number of times the concepts are mentioned (cited) in the citation context of citing publications. We demonstrate the method using three classical examples: (1) The structure of scientific revolutions by Thomas S. Kuhn, (2) The logic of scientific discovery - Logik der Forschung: Zur Erkenntnistheorie der modernen Naturwissenschaft in German -, and (3) Conjectures and refutations: the growth of scientific knowledge by Karl R. Popper. It is not surprising -- as our results show -- that Kuhn's "paradigm" concept has had a significant impact. What is surprising is that it has had such a disproportionately larger impact than Kuhn's other concepts, e.g., "scientific revolution". The paradigm concept accounts for over 80% of the concept-related citations to Kuhn's work, and its impact is resilient across all disciplines and over time. With respect to Popper, "falsification" is the most used concept derived from his books. Falsification, after all, is the cornerstone of Popper's critical rationalism.

cs.DL↗
arXiv2019

Author name disambiguation of bibliometric data: A comparison of several unsupervised approaches

Adequately disambiguating author names in bibliometric databases is a precondition for conducting reliable analyses at the author level. In the case of bibliometric studies that include many researchers, it is not possible to disambiguate each single researcher manually. Several approaches have been proposed for author name disambiguation but there has not yet been a comparison of them under controlled conditions. In this study, we compare a set of unsupervised disambiguation approaches. Unsupervised approaches specify a model to assess the similarity of author mentions a priori instead of training a model with labelled data. In order to evaluate the approaches, we applied them to a set of author mentions annotated with a ResearcherID, this being an author identifier maintained by the researchers themselves. Apart from comparing the overall performance, we take a more detailed look at the role of the parametrization of the approaches and analyse the dependence of the results on the complexity of the disambiguation task. It could be shown that all of the evaluated approaches produce better results than those that can be obtained by using only author names. In the context of this study, the approach proposed by Caron and van Eck (2014) produced the best results.

cs.DL↗
arXiv2019

The Integrated Impact Indicator (I3) Revisited: A Non-Parametric Alternative to the Journal Impact Factor

We propose the I3* indicator as a non-parametric alternative to the Journal Impact Factor (JIF) and h-index. We apply I3* to more than 10,000 journals. The results can be compared with other journal metrics. I3* is a promising variant within the general scheme of non-parametric indicators I3 introduced previously: it provides a single metric which correlates with both impact in terms of citations (c) and output in terms of publications (p). We argue for weighting using four percentile classes: the top-1% and top-10% as excellence indicators; the top-50% and bottom-50% as output indicators. Like the h-index, which also incorporates both c and p, I3*-values are size-dependent; however, division of I3* by the number of publications (I3*/N) provides a size-independent indicator which correlates strongly with the two- and five-year Journal Impact Factors (JIF2 and JIF5). Unlike the h-index, I3* correlates significantly with both the total number of citations and publications. The values of I3* and I3*/N can be statistically tested against the expectation or against one another using chi-square tests or effect sizes. A template (in Excel) is provided online for relevant tests.

cs.DL↗
arXiv2019

Does the public discuss other topics on climate change than researchers? A comparison of explorative networks based on author keywords and hashtags

Twitter accounts have already been used in many scientometric studies, but the meaningfulness of the data for societal impact measurements in research evaluation has been questioned. Earlier research focused on social media counts and neglected the interactive nature of the data. We explore a new network approach based on Twitter data in which we compare author keywords to hashtags as indicators of topics. We analyze the topics of tweeted publications and compare them with the topics of all publications (tweeted and not tweeted). Our exploratory study is based on a comprehensive publication set of climate change research. We are interested in whether Twitter data are able to reveal topics of public discussions which can be separated from research-focused topics. We find that the most tweeted topics regarding climate change research focus on the consequences of climate change for humans. Twitter users are interested in climate change publications which forecast effects of a changing climate on the environment and to adaptation, mitigation and management issues rather than in the methodology of climate-change research and causes of climate change. Our results indicate that publications using scientific jargon are less likely to be tweeted than publications using more general keywords. Twitter networks seem to be able to visualize public discussions about specific topics.

cs.DL↗
arXiv2019

How well does I3 perform for impact measurement compared to other bibliometric indicators? The convergent validity of several (field-normalized) indicators

Recently, the integrated impact indicator (I3) indicator was introduced where citations are weighted in accordance with the percentile rank class of each publication in a set of publications. I3 can also be used as a field-normalized indicator. Field-normalization is common practice in bibliometrics, especially when institutions and countries are compared. Publication and citation practices are so different among fields that citation impact is normalized for cross-field comparisons. In this study, we test the ability of the indicator to discriminate between quality levels of papers as defined by Faculty members at F1000Prime. F1000Prime is a post-publication peer review system for assessing papers in the biomedical area. Thus, we test the convergent validity of I3 (in this study, we test I3/N - the size-independent variant of I3 where I3 is divided by the number of papers) using assessments by peers as baseline and compare its validity with several other (field-normalized) indicators: the mean-normalized citation score (MNCS), relative-citation ratio (RCR), citation score normalized by cited references (CSNCR), characteristic scores and scales (CSS), source-normalized citation score (SNCS), citation percentile, and proportion of papers which belong to the x% most frequently cited papers (PPtop x%). The results show that the PPtop 1% indicator discriminates best among different quality levels. I3 performs similar as (slightly better than) most of the other field-normalized indicators. Thus, the results point out that the indicator could be a valuable alternative to other indicators in bibliometrics.

cs.DL↗
arXiv2019

Which are the influential publications in the Web of Science subject categories over a long period of time? CRExplorer software used for big-data analyses in bibliometrics

What are the landmark papers in scientific disciplines? On whose shoulders does research in these fields stand? Which papers are indispensable for scientific progress? These are typical questions which are not only of interest for researchers (who frequently know the answers - or guess to know them), but also for the interested general public. Citation counts can be used to identify very useful papers, since they reflect the wisdom of the crowd; in this case, the scientists using the published results for their own research. In this study, we identified with recently developed methods for the program CRExplorer landmark publications in nearly all Web of Science subject categories (WoSSCs). These are publications which belong more frequently than other publications across the citing years to the top-per mill in their subject category. The results for three subject categories "Information Science and Library Science", "Computer Science, Information Systems", and "Computer Science, Software Engineering" are exemplarily discussed in more detail. The results for the other WoSSCs can be found online at http://crexplorer.net.

cs.DL↗
arXiv2018

hα: The Scientist as Chimpanzee or Bonobo

In a recent paper, Hirsch (2018) proposes to attribute the credit for a co-authored paper to the α-author--the author with the highest h-index--regardless of his or her actual contribution, effectively reducing the role of the other co-authors to zero. The indicator hα inherits most of the disadvantages of the h-index from which it is derived, but adds the normative element of reinforcing the Matthew effect in science. Using an example, we show that hα can be extremely unstable. The empirical attribution of credit among co-authors is not captured by abstract models such as h, h_bar , or hα.

cs.DL↗
arXiv2018

Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK Research Excellence Framework (REF)

Altmetrics have been proposed as a way to assess the societal impact of research. Although altmetrics are already in use as impact or attention metrics in different contexts, it is still not clear whether they really capture or reflect societal impact. This study is based on altmetrics, citation counts, research output and case study data from the UK Research Excellence Framework (REF), and peers' REF assessments of research output and societal impact. We investigated the convergent validity of altmetrics by using two REF datasets: publications submitted as research output (PRO) to the REF and publications referenced in case studies (PCS). Case studies, which are intended to demonstrate societal impact, should cite the most relevant research papers. We used the MHq' indicator for assessing impact - an indicator which has been introduced for count data with many zeros. The results of the first part of the analysis show that news media as well as mentions on Facebook, in blogs, in Wikipedia, and in policy-related documents have higher MHq' values for PCS than for PRO. Thus, the altmetric indicators seem to have convergent validity for these data. In the second part of the analysis, altmetrics have been correlated with REF reviewers' average scores on PCS. The negative or close to zero correlations question the convergent validity of altmetrics in that context. We suggest that they may capture a different aspect of societal impact (which can be called unknown attention) to that seen by reviewers (who are interested in the causal link between research and action in society).

cs.DL↗
arXiv2018

Statistical Significance and Effect Sizes of Differences among Research Universities at the Level of Nations and Worldwide based on the Leiden Rankings

The Leiden Rankings can be used for grouping research universities by considering universities which are not statistically significantly different as homogeneous sets. The groups and intergroup relations can be analyzed and visualized using tools from network analysis. Using the so-called "excellence indicator" PPtop-10%--the proportion of the top-10% most-highly-cited papers assigned to a university--we pursue a classification using (i) overlapping stability intervals, (ii) statistical-significance tests, and (iii) effect sizes of differences among 902 universities in 54 countries; we focus on the UK, Germany, Brazil, and the USA as national examples. Although the groupings remain largely the same using different statistical significance levels or overlapping stability intervals, these classifications are uncorrelated with those based on effect sizes. Effect sizes for the differences between universities are small (w <.2). The more detailed analysis of universities at the country level suggests that distinctions beyond three or perhaps four groups of universities (high, middle, low) may not be meaningful. Given similar institutional incentives, isomorphism within each eco-system of universities should not be underestimated. Our results suggest that networks based on overlapping stability intervals can provide a first impression of the relevant groupings among universities. However, the clusters are not well-defined divisions between groups of universities.

cs.DL↗
arXiv2018

Interdisciplinarity as Diversity in Citation Patterns among Journals: Rao-Stirling Diversity, Relative Variety, and the Gini coefficient

Questions of definition and measurement continue to constrain a consensus on the measurement of interdisciplinarity. Using Rao-Stirling (RS) Diversity produces sometimes anomalous results. We argue that these unexpected outcomes can be related to the use of "dual-concept diversity" which combines "variety" and "balance" in the definitions (ex ante). We propose to modify RS Diversity into a new indicator (DIV) which operationalizes variety, balance, and disparity independently and then combines them ex post. "Balance" can be measured using the Gini coefficient. We apply DIV to the aggregated citation patterns of 11,487 journals covered by the Journal Citation Reports 2016 of the Science Citation Index and the Social Sciences Citation Index as an empirical domain and, in more detail, to the citation patterns of 85 journals assigned to the Web-of-Science category "information science & library science" in both the cited and citing directions. We compare the results of the indicators and show that DIV provides improved results in terms of distinguishing between interdisciplinary knowledge integration (citing) versus knowledge diffusion (cited). The new diversity indicator and RS diversity measure different features. A routine for the measurement of the various operationalizations of diversity (in any data matrix) is made available online.

cs.DL↗