arXiv subjects

Ciriaco Andrea D'Angelo

Publications and source records attributed to Ciriaco Andrea D'Angelo.

At least 91 records · Page 5Linked to original sources

arXiv2018

National research assessment exercises: a comparison of peer review and bibliometrics rankings

Development of bibliometric techniques has reached such a level as to suggest their integration or total substitution for classic peer review in the national research assessment exercises, as far as the hard sciences are concerned. In this work we compare rankings lists of universities captured by the first Italian evaluation exercise, through peer review, with the results of bibliometric simulations. The comparison shows the great differences between peer review and bibliometric rankings for excellence and productivity.

cs.DL↗

arXiv2018

Are researchers that collaborate more at the international level top performers? An investigation on the Italian university system

The practice of collaboration, and particularly international collaboration, is becoming ever more widespread in scientific research, and is likewise receiving greater interest and stimulus from policy-makers. However, the relation between research performance and degree of internationalization at the level of single researchers still presents unresolved questions. The present work, through a bibliometric analysis of the entire Italian university population working in the hard sciences over the period 2001-2005, seeks to answer some of these questions. The results show that the researchers with top performance with respect to their national colleagues are also those who collaborate more abroad, but that the reverse is not always true. Also, interesting differences emerge at the sectorial level. Finally, the effect of the nation involved in the international partnership plays a role that should not be ignored.

cs.DL↗

arXiv2018

A field-standardized application of DEA to national-scale research assessment of universities

The current work proposes an application of DEA methodology for measurement of technical and allocative efficiency of university research activity. The analysis is based on bibliometric data from the Italian university system for the five year period 2004-2008. Technical and allocative efficiency is measured with input being considered as a university's research staff, classified according to academic rank, and with output considered as the field-standardized impact of the research product realized by these staff. The analysis is applied to all scientific disciplines of the so-called hard sciences, and conducted at subfield level, thus at a greater level of detail than ever before achieved in national-scale research assessments.

cs.DL↗

arXiv2018

Assessing the varying level of impact measurement accuracy as a function of the citation window length

With the passage of more time from the original date of publication, the measure of the impact of scientific works using subsequent citation counts becomes more accurate. However the measurement of individual and organizational research productivity should ideally refer to a period with closing date just prior to the evaluation exercise. Therefore it is necessary to compromise between accuracy and timeliness. This work attempts to provide an order of magnitude for the error in measurement that occurs with decreasing the time lapse between date of publication and citation count. The analysis is conducted by scientific discipline on the basis of publications indexed in the Thomson Reuters Italian National Citation Report.

cs.DL↗

arXiv2018

A sensitivity analysis of research institutions' productivity rankings to the time of citation observation

One of the critical issues in bibliometric research assessments is the time required to achieve maturity in citations. Citation counts can be considered a reliable proxy of the real impact of a work only if they are observed after sufficient time has passed from publication date. In the present work the authors investigate the effect of varying the time of citation observation on accuracy of productivity rankings for research institutions. Research productivity measures are calculated for all Italian universities active in the hard sciences in the 2001-2003 period, by individual field and discipline, with the time of the citation observation varying from 2004 to 2008. The objective is to support policy-makers in choosing a citation window that optimizes the tradeoff between accuracy of rankings and timeliness of the exercise.

cs.DL↗

arXiv2018

The dispersion of research performance within and between universities as a potential indicator of the competitive intensity in higher education systems

Higher education systems in competitive environments generally present top universities, that are able to attract top scientists, top students and public and private financing, with notable socio-economic benefits in their region. The same does not hold true for non-competitive systems. In this study we will measure the dispersion of research performance within and between universities in the Italian university system, typically non-competitive. We will also investigate the level of correlation that occurs between performance in research and its dispersion in universities. The findings may represent a first benchmark for similar studies in other nations. Furthermore, they lead to policy indications, questioning the effectiveness of selective funding of universities based on national research assessment exercises. The field of observation is composed of all Italian universities active in the hard sciences. Research performance will be evaluated using a bibliometric approach, through publications indexed in the Web of Science between 2004 and 2008.

cs.DL↗

arXiv2018

A sensitivity analysis of researchers' productivity rankings to the time of citation observation

In this work we investigate the sensitivity of individual researchers' productivity rankings to the time of citation observation. The analysis is based on observation of research products for the 2001-2003 triennium for all research staff of Italian universities in the hard sciences, with the year of citation observation varying from 2004 to 2008. The 2008 rankings list is assumed the most accurate, as citations have had the longest time to accumulate and thus represent the best possible proxy of impact. By comparing the rankings lists from each year against the 2008 benchmark we provide policy-makers and research organization managers a measure of trade-off between timeliness of evaluation execution and accuracy of performance rankings. The results show that with variation in the evaluation citation window there are variable rates of inaccuracy across the disciplines of researchers. The inaccuracy results negligible for Physics, Biology and Medicine.

cs.DL↗

arXiv2018

When research assessment exercises leave room for opportunistic behavior by the subjects under evaluation

This study inserts in the stream of research on the perverse effects that PBRF systems can induce in the subjects evaluated. The authors' opinion is that more often than not, it is the doubtful scientific basis of the evaluation criteria that leave room for opportunistic behaviors. The work examines the 2004-2010 Italian national research assessment (VQR) to verify possible opportunistic behavior by universities in order to limit the penalization of their performance (and funding) due to the presence of scientifically unproductive professors in faculty. In particular, institutions may have favored "gift authorship" practices. The analysis thus focuses on the output of professors who were unproductive in the VQR publication window, but became productive ("new productives") in the following five years: a number of universities show a remarkably higher than average share of publications by new productives that are in co-authorship exclusively with colleagues from the same university.

cs.DL↗

arXiv2018

A new approach to measure the scientific strengths of territories

The current work applies a methodology for mapping the supply of new knowledge from public research organizations, in this case from Italian institutions at the level of regions and provinces (NUTS2 and NUTS3). Through the analysis of scientific production indexed in the Web of Science for the period 2006-2010, the new knowledge is classified in subject categories and mapped according to an algorithm for the reconciliation of authors' affiliations. Unlike other studies in the literature based on simple counting of publications, the present study adopts an indicator, Scientific Strength, which takes account of both the quantity of scientific production and its impact on the advancement of knowledge. The differences in the results that arise from the two approaches are examined. The results of works of this kind can inform public research policies, at national and local levels, as well as the localization strategies of research-based companies.

cs.DL↗

arXiv2018

The VQR, Italy's second national research assessment: Methodological failures and ranking distortions

The 2004-2010 VQR, completed in July 2013, was Italy's second national research assessment exercise. The VQR performance evaluation followed a pattern also seen in other nations, in being based on a selected subset of products. In this work we identify the exercise's methodological weaknesses and measure the distortions that result from them in the university performance rankings. First we create a scenario in which we assume the efficient selection of the products to be submitted by the universities and from this simulate a set of rankings applying the precise VQR rating criteria. Next we compare these "VQR rankings" with those that would derive from application of more appropriate bibliometrics. Finally we extend the comparison to university rankings based on the entire scientific production for the period, as indexed in the Web of Science.

cs.DL↗

arXiv2018

A bibliometric tool to assess the regional dimension of university-industry research collaborations

The present work proposes a bibliometric methodology for measuring the grade of correspondence between regional industry's demand for research collaboration and supply from public laboratories. The methodology also permits measurement of the intensity and direction of the regional flows of knowledge in public-private collaborations. The aim is to provide a diagnostic instrument for regional and national policy makers, which could add to existing ones to plan interventions for re-balancing sectorial public supply of knowledge with industrial absorptive capacity, and maximizing appropriability of knowledge spillovers. The methodology is applied to university-industry collaborations in the hard sciences in all Italian administrative regions.

cs.DL↗

arXiv2018

What is the appropriate length of the publication period over which to assess research performance?

National research assessment exercises are conducted in different nations over varying periods. The choice of the publication period to be observed has to address often contrasting needs: it has to ensure the reliability of the results issuing from the evaluation, but also reach the achievement of frequent assessments. In this work we attempt to identify which is the most appropriate or optimal publication period to be observed. For this, we analyze the variation of individual researchers' productivity rankings with the length of the publication period within the period 2003-2008, by the over 30,000 Italian university scientists in the hard sciences. First we analyze the variation in rankings referring to pairs of contiguous and overlapping publication periods, and show that the variations reduce markedly with periods above three years. Then we will show the strong randomness of performance rankings over publication periods under three years. We conclude that the choice of a three year publication period would seem reliable, particularly for physics, chemistry, biology and medicine.

cs.DL↗

arXiv2018

National peer-review research assessment exercises for the hard sciences can be a complete waste of money: the Italian case

There has been ample demonstration that bibliometrics is superior to peer-review for national research assessment exercises in the hard sciences. In this paper we examine the Italian case, taking the 2001-2003 university performance rankings list based on bibliometrics as benchmark. We compare the accuracy of the first national evaluation exercise, conducted entirely by peer-review, to other rankings lists prepared at zero cost, based on indicators indirectly linked to performance or available on the Internet. The results show that, for the hard sciences, the costs of conducting the Italian evaluation of research institutions could have been completely avoided.

cs.DL↗

arXiv2018

Variation in research collaboration patterns across academic ranks

The ability to activate and manage effective collaborations is becoming an increasingly important criteria in policies on academic career advancement. The rise of such policies leads to development of indicators that permit measurement of the propensity to collaborate for academics of different ranks, and to examine the role of several variables in collaboration, first among these being the researchers' disciplines. In this work we apply an innovative bibliometric approach based on individual propensity for collaboration to measure the differences in propensity across academic ranks, by discipline and for choice of collaboration forms - intramural, extramural domestic and international. The analysis is based on the scientific production of Italian academics for the period 2006 to 2010, totaling over 200,000 publications indexed in Web of Science. It shows that assistant professors register a propensity for intramural collaboration that is clearly greater than for professors of higher ranks. Vice versa, the higher ranks, but not quite so clearly, register greater propensity to collaborate at the international level.

cs.DL↗

arXiv2018

Revisiting the scaling of citations for research assessment

Over the past decade, national research evaluation exercises, traditionally conducted using the peer review method, have begun opening to bibliometric indicators. The citations received by a publication are assumed as proxy for its quality, but they require standardization prior to use in comparative evaluation of organizations or individual scientists: the citation data must be standardized, due to the varying citation behavior across research fields. The objective of this paper is to compare the effectiveness of the different methods of normalizing citations, in order to provide useful indications to research assessment practitioners. Simulating a typical national research assessment exercise, he analysis is conducted for all subject categories in the hard sciences and is based on the Thomson Reuters Science Citation Index-Expanded. Comparisons show that the citations average is the most effective scaling parameter, when the average is based only on the publications actually cited.

cs.DL↗

arXiv2018

How important is choice of the scaling factor in standardizing citations?

Because of the variations in citation behavior across research fields, appropriate standardization must be applied as part of any bibliometric analysis of the productivity of individual scientists and research organizations. Such standardization involves scaling by some factor that characterizes the distribution of the citations of articles from the same year and subject category. In this work we conduct an analysis of the sensitivity of researchers' productivity rankings to the scaling factor chosen to standardize their citations. To do this we first prepare the productivity rankings for all researchers (more than 30,000) operating in the hard sciences in Italy, over the period 2004-2008. We then measure the shifts in rankings caused by adopting scaling factors other than the particular factor that seems more effective for comparing the impact of publications in different fields: the citation average of the distribution of cited-only publications.

cs.DL↗

arXiv2018

The impact of unproductive and top researchers on overall university research performance

Unlike competitive higher education systems, non-competitive systems show relatively uniform distributions of top professors and low performers among universities. In this study, we examine the impact of unproductive and top faculty members on overall research performance of the university they belong to. Furthermore, we analyze the potential relationship between research productivity of a university and the indexes of concentration of unproductive and top professors. Research performance is evaluated using a bibliometric approach, through publications indexed on the Web of Science between 2004 and 2008. The set analyzed consists of all Italian universities active in the hard sciences.

cs.DL↗

arXiv2018

The importance of accounting for the number of co-authors and their order when assessing research performance at the individual level in the life sciences

Accurate measurement of research productivity should take account of both the number of co-authors of every scientific work and of the different contributions of the individuals. For researchers in the life sciences, common practice is to indicate such contributions through position in the authors list. In this work, we measure the distortion introduced to bibliometric ranking lists for scientific productivity when the number of co-authors or their position in the list is ignored. The field of observation consists of all Italian university professors working in the life sciences, with scientific production examined over the period 2004-2008. The outcomes of the study lead to a recommendation against using indicators or evaluation methods that ignore the different authors' contributions to the research results.

cs.DL↗