Searcharxiv⌕ Search

arXiv subjects

Lutz Bornmann

Publications and source records attributed to Lutz Bornmann.

At least 163 records · Page 9Linked to original sources
arXiv2013

Field-normalized Impact Factors: A Comparison of Rescaling versus Fractionally Counted IFs

Two methods for comparing impact factors and citation rates across fields of science are tested against each other using citations to the 3,705 journals in the Science Citation Index 2010 (CD-Rom version of SCI) and the 13 field categories used for the Science and Engineering Indicators of the US National Science Board. We compare (i) normalization by counting citations in proportion to the length of the reference list (1/N of references) with (ii) rescaling by dividing citation scores by the arithmetic mean of the citation rate of the cluster. Rescaling is analytical and therefore independent of the quality of the attribution to the sets, whereas fractional counting provides an empirical strategy for normalization among sets (by evaluating the between-group variance). By the fairness test of Radicchi & Castellano (2012a), rescaling outperforms fractional counting of citations for reasons that we consider.

cs.DL↗
arXiv2013

How to calculate the practical significance of citation impact differences? An empirical example from evaluative institutional bibliometrics using adjusted predictions and marginal effects

Evaluative bibliometrics is concerned with comparing research units by using statistical procedures. According to Williams (2012) an empirical study should be concerned with the substantive and practical significance of the findings as well as the sign and statistical significance of effects. In this study we will explain what adjusted predictions and marginal effects are and how useful they are for institutional evaluative bibliometrics. As an illustration, we will calculate a regression model using publications (and citation data) produced by four universities in German-speaking countries from 1980 to 2010. We will show how these predictions and effects can be estimated and plotted, and how this makes it far easier to get a practical feel for the substantive meaning of results in evaluative bibliometric studies. We will focus particularly on Average Adjusted Predictions (AAPs), Average Marginal Effects (AMEs), Adjusted Predictions at Representative Values (APRVs) and Marginal Effects at Representative Values (MERVs).

cs.DL↗
arXiv2012

The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000

The data of F1000 provide us with the unique opportunity to investigate the relationship between peers' ratings and bibliometric metrics on a broad and comprehensive data set with high-quality ratings. F1000 is a post-publication peer review system of the biomedical literature. The comparison of metrics with peer evaluation has been widely acknowledged as a way of validating metrics. Based on the seven indicators offered by InCites, we analyzed the validity of raw citation counts (Times Cited, 2nd Generation Citations, and 2nd Generation Citations per Citing Document), normalized indicators (Journal Actual/Expected Citations, Category Actual/Expected Citations, and Percentile in Subject Area), and a journal based indicator (Journal Impact Factor). The data set consists of 125 papers published in 2008 and belonging to the subject category cell biology or immunology. As the results show, Percentile in Subject Area achieves the highest correlation with F1000 ratings; we can assert that for further three other indicators (Times Cited, 2nd Generation Citations, and Category Actual/Expected Citations) the 'true' correlation with the ratings reaches at least a medium effect size.

cs.DL↗
arXiv2012

The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits

Percentiles have been established in bibliometrics as an important alternative to mean-based indicators for obtaining a normalized citation impact of publications. Percentiles have a number of advantages over standard bibliometric indicators used frequently: for example, their calculation is not based on the arithmetic mean which should not be used for skewed bibliometric data. This study describes the opportunities and limits and the advantages and disadvantages of using percentiles in bibliometrics. We also address problems in the calculation of percentiles and percentile rank classes for which there is not (yet) a satisfactory solution. It will be hard to compare the results of different percentile-based studies with each other unless it is clear that the studies were done with the same choices for percentile calculation and rank assignment.

cs.DL↗
arXiv2012

Statistical Tests and Research Assessments: A comment on Schneider (2012)

In a recent presentation at the 17th International Conference on Science and Technology Indicators, Schneider (2012) criticised the proposal of Bornmann, de Moya Anegon, and Leydesdorff (2012) and Leydesdorff and Bornmann (2012) to use statistical tests in order to evaluate research assessments and university rankings. We agree with Schneider's proposal to add statistical power analysis and effect size measures to research evaluations, but disagree that these procedures would replace significance testing. Accordingly, effect size measures were added to the Excel sheets that we bring online for testing performance differences between institutions in the Leiden Ranking and the SCImago Institutions Ranking.

stat.AP↗
arXiv2012

How Can Journal Impact Factors be Normalized across Fields of Science? An Assessment in terms of Percentile Ranks and Fractional Counts

Using the CD-ROM version of the Science Citation Index 2010 (N = 3,705 journals), we study the (combined) effects of (i) fractional counting on the impact factor (IF) and (ii) transformation of the skewed citation distributions into a distribution of 100 percentiles and six percentile rank classes (top-1%, top-5%, etc.). Do these approaches lead to field-normalized impact measures for journals? In addition to the two-year IF (IF2), we consider the five-year IF (IF5), the respective numerators of these IFs, and the number of Total Cites, counted both as integers and fractionally. These various indicators are tested against the hypothesis that the classification of journals into 11 broad fields by PatentBoard/National Science Foundation provides statistically significant between-field effects. Using fractional counting the between-field variance is reduced by 91.7% in the case of IF5, and by 79.2% in the case of IF2. However, the differences in citation counts are not significantly affected by fractional counting. These results accord with previous studies, but the longer citation window of a fractionally counted IF5 can lead to significant improvement in the normalization across fields.

cs.DL↗
arXiv2012

How to analyse percentile impact data meaningfully in bibliometrics: The statistical analysis of distributions, percentile rank classes and top-cited papers

According to current research in bibliometrics, percentiles (or percentile rank classes) are the most suitable method for normalising the citation counts of individual publications in terms of the subject area, the document type and the publication year. Up to now, bibliometric research has concerned itself primarily with the calculation of percentiles. This study suggests how percentiles can be analysed meaningfully for an evaluation study. Publication sets from four universities are compared with each other to provide sample data. These suggestions take into account on the one hand the distribution of percentiles over the publications in the sets (here: universities) and on the other hand concentrate on the range of publications with the highest citation impact - that is, the range which is usually of most interest in the evaluation of scientific performance.

stat.AP↗
arXiv2012

Citation impact of papers published from six prolific countries: A national comparison based on InCites data

Using the InCites tool of Thomson Reuters, this study compares normalized citation impact values calculated for China, Japan, France, Germany, United States, and the UK throughout the time period from 1981 to 2010. The citation impact values are normalized to four subject areas: natural sciences; engineering and technology; medical and health sciences; and agricultural sciences. The results show an increasing trend in citation impact values for France, the UK and especially for Germany across the last thirty years in all subject areas. The citation impact of papers from China is still at a relatively low level (mostly below the world average), but the country follows an increasing trend line. The USA exhibits a relatively stable pattern of high citation impact values across the years. With small impact differences between the publication years, the US trend is increasing in engineering and technology but decreasing in medical and health sciences as well as in agricultural sciences. Similar to the USA, Japan follows increasing as well as decreasing trends in different subject areas, but the variability across the years is small. In most of the years, papers from Japan perform below or approximately at the world average in each subject area.

cs.DL↗
arXiv2012

Mapping (USPTO) Patent Data using Overlays to Google Maps

A technique is developed using patent information available online (at the US Patent and Trademark Office) for the generation of Google Maps. The overlays indicate both the quantity and quality of patents at the city level. This information is relevant for research questions in technology analysis, innovation studies and evolutionary economics, as well as economic geography. The resulting maps can also be relevant for technological innovation policies and R&D management, because the US market can be considered the leading market for patenting and patent competition. In addition to the maps, the routines provide quantitative data about the patents for statistical analysis. The cities on the map are colored according to the results of significance tests. The overlays are explored for the Netherlands as a "national system of innovations," and further elaborated in two cases of emerging technologies: "RNA interference" and "nanotechnology."

cs.CY↗
arXiv2012

Metrics to evaluate research performance in academic institutions: A critique of ERA 2010 as applied in forestry and the indirect H2 index as a possible alternative

Excellence for Research in Australia (ERA) is an attempt by the Australian Research Council to rate Australian universities on a 5-point scale within 180 Fields of Research using metrics and peer evaluation by an evaluation committee. Some of the bibliometric data contributing to this ranking suffer statistical issues associated with skewed distributions. Other data are standardised year-by-year, placing undue emphasis on the most recent publications which may not yet have reliable citation patterns. The bibliometric data offered to the evaluation committees is extensive, but lacks effective syntheses such as the h-index and its variants. The indirect H2 index is objective, can be computed automatically and efficiently, is resistant to manipulation, and a good indicator of impact to assist the ERA evaluation committees and to similar evaluations internationally.

cs.DL↗
arXiv2011

Percentile Ranks and the Integrated Impact Indicator (I3)

We tested Rousseau's (in press) recent proposal to define percentile classes in the case of the Integrated Impact Indicator (I3) so that the largest number in a set always belongs to the highest (100th) percentile rank class. In the case a set of nine uncited papers and one with citation, however, the uncited papers would all be placed in the 90th percentile rank. A lowly-cited document set would thus be advantaged when compared with a highly-cited one. Notwithstanding our reservations, we extended the program for computing I3 in Web-of-Science data (at http://www.leydesdorff.net/software/i3) with this option; the quantiles without a correction are now the default. As Rousseau mentions, excellence indicators (e.g., the top-10%) can be considered as special cases of I3: only two percentile rank classes are distinguished for the evaluation. Both excellence and impact indicators can be tested statistically using the z-test for independent proportions.

cs.CY↗
arXiv2011

The new Excellence Indicator in the World Report of the SCImago Institutions Rankings 2011

The new excellence indicator in the World Report of the SCImago Institutions Rankings (SIR) makes it possible to test differences in the ranking in terms of statistical significance. For example, at the 17th position of these rankings, UCLA has an output of 37,994 papers with an excellence indicator of 28.9. Stanford University follows at the 19th position with 37,885 papers and 29.1 excellence, and z = - 0.607. The difference between these two institution thus is not statistically significant. We provide a calculator at http://www.leydesdorff.net/scimago11/scimago11.xls in which one can fill out this test for any two institutions and also for each institution on whether its score is significantly above or below expectation (assuming that 10% of the papers are for stochastic reasons in the top-10% set).

cs.DL↗
arXiv2011

Which cities' paper output and citation impact are above expectation in information science? Some improvements of our previous mapping approaches

Bornmann and Leydesdorff (in press) proposed methods based on Web-of-Science data to identify field-specific excellence in cities where highly-cited papers were published more frequently than can be expected. Top performers in output are cities in which authors are located who publish a number of highly-cited papers that is statistically significantly higher than can be expected for these cities. Using papers published between 1989 and 2009 in information science improvements to the methods of Bornmann and Leydesdorff (in press) are presented and an alternative mapping approach based on the indicator I3 is introduced here. The I3 indicator was introduced by Leydesdorff and Bornmann (in press).

cs.DL↗
arXiv2011

The Anna Karenina principle: A concept for the explanation of success in science

The first sentence of Leo Tolstoy's novel Anna Karenina is: "Happy families are all alike; every unhappy family is unhappy in its own way." Here Tolstoy means that for a family to be happy, several key aspects must be given (such as good health of all family members, acceptable financial security, and mutual affection). If there is a deficiency in any one or more of these key aspects, the family will be unhappy. In this paper we introduce the Anna Karenina principle as a concept that can explain success in science. Here we will refer to three central areas in modern science in which scarce resources will most usually lead to failure: (1) peer review of research grant proposals and manuscripts (money and journal space as scarce resources), (2) citation of publications (reception as a scarce resource), and (3) new scientific discoveries (recognition as a scarce resource). If resources are scarce (journal space, funds, reception, and recognition), there can be success only when several key prerequisites for the allocation of the resources are fulfilled. If any one of these prerequisites is not fulfilled, the grant proposal, manuscript submission, the published paper, or the discovery will not be successful.

cs.DL↗
arXiv2011

Which are the best cities for psychology research worldwide? A map visualizing city ratios of observed and expected numbers of highly-cited papers

We present scientometric results about world-wide centers of excellence in psychology. Based on Web of Science data, domain-specific excellence can be identified for cities where highly cited papers are published. Data refer to all psychology articles published in 2007 which are documented in the Social Science Citation Index and to their citation frequencies from 2007 to May 2011. Visualized are 214 cities with an article output of at least 50 in 2007. Statistical z tests are used for the evaluation of the degree to which an observed number of top-cited papers (top-10%) for a city differs from the number expected on the basis of randomness in the selection of papers. Map visualizing city ratios on significant differences between observed and expected numbers of highly-cited papers point at excellence centers in cities at the East and West Coast of the United States as well as in Great Britain, Germany, the Netherlands, Ireland, Belgium, Sweden, Finland, Australia, and Taiwan. Furthermore, positive but non-significant differences in favor of high citation rates are documented for some cities in the United States, Great Britain, the Netherlands, the Scandinavian and the German-speaking countries, Belgium, France, Spain, Israel, South Korea, and China. Scientometric results show convincingly that highly-cited psychological research articles come from the Anglo-American countries and some of the non-English European countries in which the number of English-language publications has increased during the last decades.

stat.OT↗
arXiv2011

Which cities produce excellent papers worldwide more than can be expected? A new mapping approach--using Google Maps--based on statistical significance testing

The methods presented in this paper allow for a statistical analysis revealing centers of excellence around the world using programs that are freely available. Based on Web of Science data, field-specific excellence can be identified in cities where highly-cited papers were published significantly. Compared to the mapping approaches published hitherto, our approach is more analytically oriented by allowing the assessment of an observed number of excellent papers for a city (in the sample) against the expected number. Using this test, the approach cannot only identify the top performers in output but the "true jewels." These are cities locating authors who publish significantly more top cited papers than can be expected. As the examples in this paper show for physics, chemistry, and psychology, these cities do not necessarily have a high output of excellent papers.

cs.DL↗
arXiv2011

Turning the tables in citation analysis one more time: Principles for comparing sets of documents

We submit newly developed citation impact indicators based not on arithmetic averages of citations but on percentile ranks. Citation distributions are-as a rule-highly skewed and should not be arithmetically averaged. With percentile ranks, the citation of each paper is rated in terms of its percentile in the citation distribution. The percentile ranks approach allows for the formulation of a more abstract indicator scheme that can be used to organize and/or schematize different impact indicators according to three degrees of freedom: the selection of the reference sets, the evaluation criteria, and the choice of whether or not to define the publication sets as independent. Bibliometric data of seven principal investigators (PIs) of the Academic Medical Center of the University of Amsterdam is used as an exemplary data set. We demonstrate that the proposed indicators [R(6), R(100), R(6,k), R(100,k)] are an improvement of averages-based indicators because one can account for the shape of the distributions of citations over papers.

cs.DL↗