Research Evaluation
|
Research in the Web 2.0 era is still evaluated mainly trough peer review,
scientists seem not to exploit the new opportunities and lessons
learned by the Web and Open Source paradigm, as agile development to
develop concepts, models, metrics, and tools for an efficient (for people), effective (for science), and sustainable (for publishers and the community) way of creating, disseminating, evaluating, and consuming
scientific knowledge. In this scenario our research aim at these different objectives follows three different paths:
(1) how to improve current peer review process; (2) alternative ways of
evaluating scientific contributions relying on the filtering power of
the community; (3) ways to assess impact of research (papers, people,
projects).
|
Documents
Reviewing peer review: a quantitative analysis of peer review
Fabio Casati, Maurizio Marchese, Katsiaryna Mirylenka, Azzurra Ragone
In this paper we focus on the analysis of peer reviews and reviewers behavior in a number of different review processes. More specifically, we report on the development, definition and rationale of a theoretical model for peer review processes to support the identification of appropriate metrics to assess the processes main properties. We then apply the proposed model and analysis framework to data sets from conference evaluation processes and we discuss the results implications and their eventual use toward improving the analyzed peer review processes. A number of unexpected results were found, in particular: (1) the low correlation between peer review outcome and impact in time of the accepted contributions and (2) the presence of an high level of randomness in the analyzed peer review processes.
Is peer review any good? A quantitative analysis of peer review (Preliminary Draft)
Fabio Casati, Maurizio Marchese, Azzurra Ragone, Matteo Turrini
(This is a short and limited version of D3.1v1)
In
this paper we focus on the analysis of peer reviews and reviewers behavior in conference review processes. We report on the development,
definition and rationale of a theoretical model for peer review
processes to support the identification of appropriate metrics to
assess the processes main properties. We then apply the proposed model
and analysis framework to data sets about reviews of conference papers.
We discuss in details results, implications and their eventual use
toward improving the analyzed peer review processes. Conclusions and
plans for future work close the paper.
Is peer review any good? An analysis framework and large-scale experiments. Fabio Casati, Maurizio Marchese, Azzurra Ragone, Matteo Turrini
A quantitative analysis of Peer Review. Azzurra Ragone, Katsiaryna Mirylenka, Fabio Casati, Maurizio Marchese. In proceedings of 13th International Society of Scientometrics and Informetrics Conference - 2011
On the correlation between bibliometric indicators and rankings of conferences and researchers. Cristhian Parra, Marlon Dumas,Luciano Garcia-Banuelos, Karina Kisselite, Peep Küngas
In Liquid Proceedings of the Interdisciplinary Workshop on Trust and Reputation.
This paper aims at finding correlation between bibliometric indicators, that are traditionally used in research evaluation (e.g. citations count, h-index, g-index, etc.), with perceived reputation of researchers and conferences. The empirical results show that while the mentioned indicators are the essential features, among other objective criteria, in reputation of conferences, they are not the discriminant features in reputation of individual researchers.
Homophily-Weighted Citation Metrics. Marlon Dumas, Luciano Garcia-Banuelos, Karina Kisselite, Peep Küngas, Cristhian Parra
In Liquid Proceedings of the Interdisciplinary Workshop on Trust and Reputation.
The concept of homophily describes relationships that are based on some measures of similarity or closeness. Despite the fact that no single definition exists for similarity, in the scope of this work, we consider two types of relationships to define similarity: citation and co-authorship. Using these relationships, we can measure similarity to be inversely proportional to the distance that separates researchers in the co-authorship network and papers in the citation network. Following this basic intuition, the rationale behind our homophily-weighted metrics consists on using the distance on these graphs to weight traditional metrics, which in this particular paper is applied to citation count.
Investigating the nature of scientific reputation. Cristhian Parra, Fabio Casati, Florian Daniel, Maurizio Marchese, Luca Cernuzzi, Marlon Dumas, Peep Küngas, Luciano García-Bañuelos, Karina Kisselite. 13th Conference of the International Society for Scientometrics and Informetrics.
Excellence or quality are often regarded as the holy grail of science, and it is the main goal driving scientist to pursue research that will influence the direction of their fields. This excellence, however, has no standard definition and varies across disciplines, and even from person to person, making it difficult to evaluate research and reputation of researchers. In this work, we introduce the general problem of studying the nature of reputation in the context of computer science, providing preliminary results on its relation with bibliometric indicators and hints for future experiments that will foster a better understanding of reputation in the scientific domain.
A Mashup Platform for Research Evaluation. Muhammad Imran, Florian Daniel, Fabio Casati, Maurizio Marchese. Proceedings of ECSS 2010, 2010, Prague, Czech Republic.
Bibliometrics has changed out the way the research evaluation conducted, and it is widely used to evaluate research groups, individual research's, department and many more. However establishing fair criteria to evaluate the scienti c community, as well as individual publications and researcher, is a tough task and constitutes a challenge that has not been achieved yet. This paper addresses the problem of research evaluation and introduces ResEval, a mashup platform that enables the creation of customize metrics and their computation in order to make the scienti c evaluation easier. This platform addresses various problems with current approaches such as data incompleteness, exibility in de fining new metrics, xed UI restrictions for the customization of metrics and to apply filters.
Tools
Reseval is a tool for evaluating research contributions and people by using citation-based metrics. At the moment the only source used by Reseval to compute metrics is Google Scholar. However, in the near future, there will be the possibility to choose which source to look for (so not only Scholar) or to directly provide the source where to look for, and also the possibility to choose a particular algorithm for computing given metrics, as, for instance, h-index without self-citations, h-index without top citers, etc.
Group comparison is a tool that allow you to create groups of researchers and then evaluate and compare them using several metrics, like h-index, g-index, number of publications and citations, average number of citations, etc. You can create a group browsing universities, sectors, departments, faculties or simply adding your co-authors or your research team to your personal group. Once groups are created you can compare researchers within a particular group, comparing their h-index, g-index, number of citations or publications, etc.
It is also possible to do comparison across groups, selecting two or more groups. You can compare global indexes among groups to discover the more productive ones or the highly cited ones.
OpinioNet: the LP Reputation Module
The RepModule is a tool used to calculate the reputation of both research and researchers. The calculation is based on the opinions that fellow researchers may form about existing research. The tool essentially focuses on the propagation of opinions in structured SKO graphs and the aggregation of these opinions for the calculation of the final group opinion. A brief overview is presented here.
Homophily Weighted Citation Count (HWCC)
The concept of homophily describes relationships that are based on some measures of similarity or closeness. HWCC defines a family of bibliometric indicators, namely the homophily weighted citation count that takes similarity to be inversely proportional to the distance that separates researchers in the co-authorship network and papers in the citation network. The rational behind this is to provide an indicator that favors publications having an impact in broader communities. Similar to the conventional citation count, this metric can be used for assessing the impact of Venues.
Information on the application's API is available here.
Further information on the concept and equations is available here.
Presentations
"LiquidPub: Services at Service of Science" invited talk by Fabio Casati at ECOW 2009 (European Conference on Web Services)
Presentation "Is peer review any good? A quantitative analysis of peer review " by Fabio Casati at EUROPEAN COMPUTER SCIENCE SUMMIT - ECSS 2009 - 5th Annual INFORMATICS-Europe Meeting - 8-9 October 2009, Paris -
Related Work
Publish and perish: why the current publication and review model is killing research and wasting your money Fabio Casati, Fausto Giunchiglia, Maurizio Marchese (ACM Ubiquity 8(3), Feb 2007)
Exploring and Understanding Scientific
Metrics in Citation Networks. MikalaiKrapivin,
Maurizio Marchese, Fabio Casati. Complex 2009
Positional effect on citation and readership in arXiv, by Haque and Ginsparg
Stochastic modeling of citation slips, by M. V. SIMKIN, V. P. ROYCHOWDHURY
