Summary

I attended the Impact Factor and Research Metrics online workshop at the GC library. The host Jill Cirasella is the Scholarly Communication, Librarian and University Liaison. Cirasella presented an informative discussion on how to assess the importance of research articles, authors, and journals using several distinct kinds of metrics. Moreover, while Ms. Cirasella pointed out the advantages and pitfalls of this journey it was nonetheless entertaining and quite dramatic for a novice like me. In assessing the impact of a particular research article, author, or journal one is using a quantitative metric to determine something that may be qualitative and subjective in nature. Nevertheless, these metrics can be a guide we can use when we decide on articles to include in our research endeavors.
When we analyze the impact of a particular research article, we might want to ask ourselves these four questions. How important is the article? How prominent is the author? How important is the author’s work? How good is the journal? Citation metrics is one way to tell how important an article, an author and a journal are. In its basic form, citation metric is a count of how many times an article, author or journal has been cited. They are tools that we can use to give us citation counts. The popular ones are Google Scholar, Web of Science and Scopus. Google Scholar is by far the most extensive. But is citation count an effective way to determine importance and impact? The answer to that is not clear cut. Articles are cited for varied reasons. Some articles are cited to further contribute to research in a discipline. Other articles can be cited because the current researcher is aiming to disprove the findings of a previous researcher. Still others are cited because it is newsy in the sense that it has been retracted or surrounded in controversy. An example of this, is an article that has since been retracted which purported to find a link between vaccines and autism. Also, when looking at citation counts it is important to situate it in the context of its academic discipline. For example, a discipline such as Biology will have more citations than a discipline such as Philosophy. In Biology it is standard practice when reporting on the results of an experiment to cite all prior experiments. Whereas in philosophy it is common to just cite few main texts.
To determine how important a researcher and their body of work is we can calculate a metric referred to as the h-index. Google Scholar popularized the h-index, and it says how much articles that an author has published that have been cited h or more times. So, an h-index of 3 means that an author has 3 articles that have been cited 3 or more times. This ignores the total number of articles that the author has published. An author could have published 100 articles but only have an h-index of 3. Some problems with the h-index are it can vary across discipline and it can be manipulated using self-citation. There are variants of the h-index that can be applied to journals. These include the Journal h-index and the Journal h5-index.
To assess the importance of a journal we can also calculate the “Impact Factor” of the journal. To calculate the impact factor of 2023 for Journal A =# of citations in 2023 for articles from 2021 to 2022/ # of articles from 2021 and 2022. This tells us the average number of times articles published in 2021 and 2022 were cited in 2023. There are some problems with the impact factor some of the main ones are it prioritizes faster citations. It is based on citation counts and not all citations are good ones. It can be manipulated because researchers can collude with journals to cite other articles in that journal.
The are alternative ways to assess impact that looks beyond the scholarly journals and articles. Some examples are Altmetric.com and PlumX which assesses impact by looking at social media, blog post, news articles and Wikipedia. Drawbacks to these metrics are that they reward buzzworthy clickbait topics, reward those active on social media and these metrics can also be manipulated like their traditional counterparts. A promising alternative metric is HuMetricsHSSS which is Humane Metrics in the Humanities and Social Sciences. This metric includes a completely different value set. It values things like Equity, Openness, Collegiality, Soundness and Community. Finally, DORA which refers to declaration on Research Assessment is an international movement of researchers and organizations who recognize the need for eliminating the use of journal base metrics in assessing the impact of research.

The final issue touched upon in this workshop was the inequities in citation metrics in relation to gender and race. The idea is that citations tend to privilege articles written by male researchers over female researchers. This is called the Matthew and Matilda effect. The Matthew effect refers to the idea that often it is the old boys club. The same people who have contributed gets elevated as being prominent in the field. This privileges male researchers. The Matilda effect suppresses the contributions of female researchers crediting and citing their works less. Studies have been done to assess the disparities in gender citations. Researchers chose a sample of articles using the authors name, they count the citations in the articles and control for factors such as tenure to see if there is a disparity in citation metrics for men and women. These studies found a disparity, but they are not without their critique. One of the problems with such studies is that it is hard to know the person’s gender simply by the name. Names can be both male and females. Further still authors sometimes just use initials. One interesting study looked at self- hyping and the gender gap. They look at words that male researchers use to describe their article verses female researchers. Male researchers tend to words such as Novel, Unique, Promising, Favorable, and Robust. Women tend to use words such as Supportive.
Black women tend to be underrepresented in citation counts relative to their numbers in the field. For example, in Anthropology black women account for 2.8% of Anthropologist but have a citation count of 0.8%.