Research Evaluation
|
Research in the Web 2.0 era is still evaluated mainly trough peer review,
scientists seem not to exploit the new opportunities and lessons
learned by the Web and Open Source paradigm, as agile development to
develop concepts, models, metrics, and tools for an efficient (for people), effective (for science), and sustainable (for publishers and the community) way of creating, disseminating, evaluating, and consuming
scientific knowledge. In this scenario our research aim at these different objectives follows three different paths:
(1) how to improve current peer review process; (2) alternative ways of
evaluating scientific contributions relying on the filtering power of
the community; (3) ways to assess impact of research (papers, people,
projects).
|
Documents
Reviewing peer review: a quantitative analysis of peer review
Fabio Casati, Maurizio Marchese, Katsiaryna Mirylenka, Azzurra Ragone
In this paper we focus on the analysis of peer reviews and reviewers behavior in a number of different review processes. More specifically, we report on the development, definition and rationale of a theoretical model for peer review processes to support the identification of appropriate metrics to assess the processes main properties. We then apply the proposed model and analysis framework to data sets from conference evaluation processes and we discuss the results implications and their eventual use toward improving the analyzed peer review processes. A number of unexpected results were found, in particular: (1) the low correlation between peer review outcome and impact in time of the accepted contributions and (2) the presence of an high level of randomness in the analyzed peer review processes.
Is peer review any good? A quantitative analysis of peer review (Preliminary Draft)
Fabio Casati, Maurizio Marchese, Azzurra Ragone, Matteo Turrini
(This is a short and limited version of D3.1v1)
In
this paper we focus on the analysis of peer reviews and reviewers behavior in conference review processes. We report on the development,
definition and rationale of a theoretical model for peer review
processes to support the identification of appropriate metrics to
assess the processes main properties. We then apply the proposed model
and analysis framework to data sets about reviews of conference papers.
We discuss in details results, implications and their eventual use
toward improving the analyzed peer review processes. Conclusions and
plans for future work close the paper.
Is peer review any good? An analysis framework and large-scale experiments. Fabio Casati, Maurizio Marchese, Azzurra Ragone, Matteo Turrini
Tools
Reseval is a tool for evaluating research contributions and people by using citation-based metrics. At the moment the only source used by Reseval to compute metrics is Google Scholar. However, in the near future, there will be the possibility to choose which source to look for (so not only Scholar) or to directly provide the source where to look for, and also the possibility to choose a particular algorithm for computing given metrics, as, for instance, h-index without self-citations, h-index without top citers, etc.
Group comparison is a tool that allow you to create groups of researchers and then evaluate and compare them using several metrics, like h-index, g-index, number of publications and citations, average number of citations, etc. You can create a group browsing universities, sectors, departments, faculties or simply adding your co-authors or your research team to your personal group. Once groups are created you can compare researchers within a particular group, comparing their h-index, g-index, number of citations or publications, etc.
It is also possible to do comparison across groups, selecting two or more groups. You can compare global indexes among groups to discover the more productive ones or the highly cited ones.
The RepModule is a tool used to calculate the reputation of both research and researchers. The calculation is based on the opinions that fellow researchers may form about existing research. The tool essentially focuses on the propagation of opinions in structured SKO graphs and the aggregation of these opinions for the calculation of the final group opinion. A brief overview is presented here.
Presentations
"LiquidPub: Services at Service of Science" invited talk by Fabio Casati at ECOW 2009 (European Conference on Web Services)
Presentation "Is peer review any good? A quantitative analysis of peer review " by Fabio Casati at EUROPEAN COMPUTER SCIENCE SUMMIT - ECSS 2009 - 5th Annual INFORMATICS-Europe Meeting - 8-9 October 2009, Paris -
Related Work
Publish and perish: why the current publication and review model is
killing research and wasting your money Fabio Casati, Fausto Giunchiglia, Maurizio Marchese (ACM Ubiquity 8(3), Feb 2007)
Exploring and Understanding Scientific
Metrics in Citation Networks. MikalaiKrapivin,
Maurizio Marchese, Fabio Casati. Complex 2009
Positional effect on citation and readership in arXiv, by Haque and Ginsparg
Stochastic modeling of citation slips, by M. V. SIMKIN, V. P. ROYCHOWDHURY
