Forum dyskusyjne • Discussion forum
Lies, damned lies, and impact factor
Ana Marušić
Invited commentary
The original quote that I paraphrased in the title of my comment is “Lies, damned lies, and statistics” and is a frequent phrase to emphasize the power of numbers when they are used to support weak arguments.
And impact factor is indeed a number generated from a formula; it is a number with great power in academic community and the subject of heated debates and misuse, including fake impact factor and fake impact factor companies [1]. However, its primary purpose has been distorted over time from a useful tool for librarians to the major currency of academic advancement. It was developed in 1955 by Eugene Garfield as a bibliographical tool to help the selection of journals into a bibliographical database [2]. Garfield’s intention was to use citations to journal articles to identify journals with the highest impact on research; his idea was to create a database with ten percent of the most influential scientific journals and thus provide guidance for librarians how to use limited funds to get most important journals to their libraries and their researchers. The database — Science Citation Index (SCI) remains the most selective and prestigious bibliographical and citation database. As a journal editor, I thus find the impact factor a useful tool to follow my own journal and other journals in the field and generally in science, careful to take into account well known differences between different disciplines and different journals. I also do not look only for the impact factor, but also for other measures of impact (again being aware that they are based on article citations and thus biased to some extent, just as the impact factor) [3–5].
As a scientist, I fully agree with proposals that have been around for a long time that the impact factor should not be the sole element for the evaluation of individual researchers and/or articles [6]. It may be useful: I know from my own and my colleagues’ experience that it is more difficult to publish a paper in high-impact factor journals — if I publish there I know that, on average (!), this journal has selected my manuscript in a highly competitive, selective and critical evaluation.
When we use the impact factor for journals or researchers, such as a “single researcher impact factor” [7], we have to keep in mind a few simple facts about the mathematical equation that generates the value for the impact factor. By definition, it is the ratio between the citations a journal’s articles from two previous years receive in the current year and the number of articles published in the two previous years in that journal [8]. And there is a catch in this equation: the numerator in the formula includes all citations, regardless whether they are to original research work or non-research items, such as letters, comments, editorials and other non-research item. On the other hand, the denominator includes only the journal items that are considered citable: this means those published items categorized as “Article” or “Review” by the indexing experts at the Thompson Reuters. This means that journals publishing many non-research items and which are regarded as important and cited by the research community will always have higher impact factors that journals publishing solely research articles. Because of this, big commercial journals with large editorial staff to cover news and with influence to attract leading experts to write their commentaries or correspondence will by default have larger impact factors that smaller scholarly journal that are mostly run by volunteers and publishing only research articles (and frankly, the former journals are a more interesting read).
To provide evidence for the influence of research and non-research articles on the impact factor equation, which is very much needed in the current debates about impact factor, my colleagues and I performed a study of what constituted the impact factor of “small” and “big” journals [8]. We read over 9000 items in published 4 journals during two years: New England Journal of Medicine (the first in the Thompson Reuters’ Journal Citation Reports’ (JCR) category of “Medicine, General and Internal”; Croatian Medical Journal (the journal I was editing at that time, somewhere in the middle of the same JCR category); Nature (the first-ranking journal in JCR category “Multidisciplinary sciences”; and Anais da Academia Brasileira de Ciencias (a small scholarly journal at the same ranking place in the “Multidisciplinary sciences” category as the Croatian Medical Journal in its “Medicine, General and Internal” category. We classified whether the published items contained original material (such as figures, tables or numbers in the text that we not referenced to other sources), checked whether all published items were indexed in the Web of Science, and retrieved their citations. This was all done before the release of the relevant impact factor for that year, so we were not biased by the existing impact factor for the dataset that we analysed.
We found that high-impact journals published more than 60% of their content a non-research items (editorial material, letters, comments). These items were well cited — for example, the total number of citations to non-research items exceeded the number of citations to all reviews published in a journal. They also increased the impact factor of the journals. For my journal, which also published editorial material that was cited, these citations also contributed to the impact factor (small as it was). Only for the Anais da Academia Brasileira de Ciencias, which published a single editorial and an obituary in two years’ time, the number of citations reflected only the research content of the journal.
I told this story to illustrate that the question of impact factor is a complex one, that it has its good and bad sides, and that it should be regarded as one but not the only measure of journals, research, and researchers. We have to understand what the number means, and do not give it more statistical or real-world significance than it should have. Yes, it is easy to substitute a complex evaluation such as that for academic promotion with a single number — it is much more difficult to actually read the actual paper and evaluate its impact not only in citations but in other measures of impact. And citations are not always linked to individuals: for example, the quote from the beginning of the text on lies and statistics is well known (retrieves 387000 results on Google for the full citation and 21 articles on Medline with the quote in article title) and often cited, but we do not know who the actual author is, except that the origin could be traced to the end of the 19th century [9]. The unknown author would not be able to use these citations for academic promotion.
Professor Ana Marušić
Chair, Department of Research in Biomedicine and Health
University of Split School of Medicine
Editor, “Journal of Global Health”
President, European Association of Science Editors (EASE) (2015–)
Past President, World Association of Medical Editors (WAME)
Split, Croatia
e-mail: ana.marusic@mefst.hr
References
Jalalian M. The story of fake impact factor companies and how we detected them. Electron Physician 2015; 7: 1069–1072.
Garfield E. The history and meaning of the journal impact factor. JAMA 2006; 295: 90–93.
Gonzalez-Pereira B, Guerrero-Bote V, Moya-Anegon F. The SJR indicator: A new indicator of journals’ scientific prestige. arXiv:0912.4141 [cs.DL] Available from: http://arxiv.org/abs/0912.4141. Accessed: 25 September 2015.
Moed HF (ed). Measuring contextual citation impact of scientific journals. arXiv: arXiv:0911.2632 [cs.DL]. Available from: http://arxiv.org/abs/0911.2632. Accessed: 25 September 2015.
Yang ZG, Zhang CT.A proposal for a novel impact factor as an alternative to the JCR impact factor. Sci Rep 2013; 3: 3410.
Colledge L, James C. A “basket of metrics”— the best support for understanding journal merit. Eur Sci Edit 2015; 41: 61–65.
Castelnuovo G, Limonta D, Sarmiento L, Molinari E A more comprehensive index in the evaluation of scientific research: the single researcher impact factor proposal. Clin Pract Epidemiol Ment Health 2010; 6: 109–114.
Golubic R, Rudes M, Kovacic N, Marusic M, Marusic A. Calculating impact factor: how bibliographical classification of journal items affects the impact factor of large and small journals. Sci Eng Ethics 2008; 14: 41–49.
University of York Department of Mathematics. Lies, damned lies, and statistics. Available at: http://www.york.ac.uk/depts/maths/histstat/lies.htm. Accessed: 25 September 2015.