A story in Nature discusses a recent paper (preprint) from CNetS members Jasleen Kaur, Filippo Radicchi and Fil Menczer on the universality of scholarly impact metrics. In the paper, we present a method to quantify the disciplinary bias of any scholarly impact metric. We use the method to evaluate a number of established scholarly impact metrics. We also introduce a simple universal metric that allows to compare the impact of scholars across scientific disciplines. Mohsen JafariAsbagh integrated this metric into Scholarometer, a crowdsourcing system developed by our group to collect and share scholarly impact data. The Nature story highlight how one can use normalized impact metrics to rank all scholars, as illustrated in the widget shown here.
Scholarometer is a social tool to facilitate citation analysis and help evaluate the impact of an author’s publications. One of the promises of Web Science is to leverage the wisdom of the crowds to give rise to emergent, bottom-up semantics, by making it easy for users to express relationships between arbitrary kinds of objects. Rather than starting with an ontology that determines the kinds of objects and relationships to be described and reasoned about, the idea is to give users the freedom to annotate arbitrary objects with arbitrary predicates, along with incentives for such annotations. Social tagging systems for images are one example, where the motivation can stem from the wish to organize and share one’s photos or from entertaining games to guess one another’s tags. Here we explore a similar approach in the domain of scholarly publications. We describe a system called Scholarometer, which provides a service to scholars by computing citation-based impact measures. This motivates users to provide disciplinary annotations for authors, which in turn can be used to compute for the first time measures that allow to compare authors’ impact across disciplinary boundaries. We show how this crowdsourcing approach can lead to emergent semantic networks to study interdisciplinary annotations and trends. To learn more please visit http://scholarometer.indiana.edu/about.html
- J. Kaur, E. Ferrara, F. Menczer, A. Flammini, F. Radicchi (2015) Quality versus quantity in scientific impact. Journal of Informetrics 9(4): 800-808, doi:10.1016/j.joi.2015.07.008
- Jasleen Kaur, Mohsen JafariAsbagh, Filippo Radicchi, Filippo Menczer (2014) Scholarometer: a system for crowdsourcing scholarly impact metrics. Proceedings of the 2014 ACM conference on Web Science, 285-286
- Jasleen Kaur, Mohsen JafariAsbagh, Filippo Radicchi, Filippo Menczer (2014) Crowdsourced disciplines and universal impact. Proc. ACM WebSci14 Altmetrics workshop.
- Jasleen Kaur, Filippo Radicchi, Filippo Menczer (2014) On the use of sampling statistics to advance bibliometrics. Journal of Informetrics 8(2): 419-420, doi:10.1016/j.joi.2014.01.010
- Jasleen Kaur, Filippo Radicchi, and Filippo Menczer (2013) Universality of scholarly impact metrics. Journal of Informetrics 7(4): 924-932, doi:10.1016/j.joi.2013.09.002
- Xiaoling Sun, Jasleen Kaur, Stasa Milojevic, Alessandro Flammini and Filippo Menczer (2013) Social Dynamics of Science. Nature Scientific Reports 3(1069). doi:10.1038/srep01069
- Presented at 2013 International Science of Team Science (SciTS) Conference
- Presented at ECCS’13 COVENANT workshop
- Xiaoling Sun, Jasleen Kaur, Lino Possamai and Filippo Menczer (2013) Ambiguous author query detection using crowdsourced digital library annotations. Information Processing & Management 49(2): 454-464. doi:10.1016/j.ipm.2012.09.001
- Jasleen Kaur, Diep Thi Hoang, XIaoling Sun, Lino Possamai, Mohsen JafariAsbagh, Snehal Patil and Filippo Menczer (2012) Scholarometer: A Social Framework for Analyzing Impact across Disciplines. PLoS ONE 7(9): e43235. doi:10.1371/journal.pone.0043235
- Xiaoling Sun, Jasleen Kaur, Lino Possamai and Filippo Menczer (2011). Detecting Ambiguous Author Names in Crowdsourced Scholarly Data. Proceedings of 3rd IEEE Conference on Social Computing
- Thi Hoang, Diep and Kaur, Jasleen and Menczer, Filippo (2010) Crowdsourcing Scholarly Data. Proceedings of the Web Science Conference (WebSci10): Extending the Frontiers of Society On-Line
Scholarometer is becoming a more mature tool. The idea behind scholarometer — crowdsourcing scholarly data — was presented at the Web Science 2010 Conference in Raleigh, North Carolina, along with some promising preliminary results. Recently acquired functionality includes a Chrome version, percentile calculations for all impact measures, export of bibliographic data in various standard formats, heuristics to determine reliable tags and detect ambiguous names, etc. Next up: an API to share annotation and impact data, and an interactive visualization for the interdisciplinary network.
CNetS graduate student Diep Thi Hoang and associate director Filippo Menczer have developed a tool (called Scholarometer, previously Tenurometer in beta version) for evaluating the impact of scholars in their field. Scholarometer uses the h-index, which combines the scholarly output with the influence of the work, but adds the universal h-index proposed by Radicchi et al. to compare the impact of research in different disciplines. This is enabled by a social mechanism in which users of the tool collaborate to tag the disciplines of the scholars. “We have computer scientists, physicists, social scientists, people from many different backgrounds, who publish in lots of different areas,” says Menczer. However, the various communities have different citation methods and different publishing traditions, making it difficult to compare the influence of a sociologist and a computer scientist, for example. The universal h-index controls for differences in the publishing traditions, as well as the amount of research scholars in various fields have to produce to make an impact. Menczer is especially excited about the potential to help show how the disciplines are merging into one another. More from Inside Higher Ed… (Also picked up by ACM TechNews and CACM.)