How are impact factors calculated?

The impact factor was invented by Eugene Garfield as a simple method for comparing journals, regardless of their size.¹ The impact factor of a journal is calculated as the number of citations in a certain year to papers in the same journal in the two years before (numerator), divided by the number of “source” (citable) items published in that journal in the same two years (denominator). To give an example, the impact factor for 2001 will be calculated from the total number of citations in 2001 to papers published in the years 2000 and 1999, divided by the total number of citable items in the years 2000 and 1999. “Citable” is important here, because not all items published in a journal are considered to be citable by ISI, The Institute for Scientific Information (www.isi.com), which is responsible for compiling the impact factors. Research papers and short reports will be counted as citable items, but editorials, reviews, and letters to the editor are usually not. However, this is not completely predictable because ISI is independent and known to have a mind of its own, and may therefore not follow the allocation of papers to certain categories by an individual journal. To give an example, we traditionally call review papers “leaders”,^2–8 which may be confusing because ISI considers most of them to be research papers, which is of course unfavourable for our impact factor. In 1999 and 1998, we published a total of 47 leaders, which were in fact reviews. Had these not been counted in the denominator, our impact factor for 2000 would have been 930/(483) = 1.93! Therefore, from this issue onwards, we have renamed our leaders “reviews” for the sake of clarity.

The impact of an individual paper on the impact factor may be assessed in the same way. For instance, this editorial which is published in 2001 appropriately cites 26 JCP papers from the years 2000 and 1999. Assuming a total number of citable items of 500 in 2000 and 2001, this editorial in itself will conveniently raise our impact factor by about 0.05.

For JCP and Molecular Pathology (MP), which have a combined impact factor, the total number of citations in the year 2000 to papers in the years 1998 and 1998 was 930, which when divided by the total of 530 published citable items in 1999 and 1998 results in an impact factor of 1.8 for the year 2000. We are pleased that over the past four years our impact factor has been gradually increasing, indicating that papers published in JCP and MP are not only well read and appreciated (we have known that for a long time), but are also increasingly cited. JCP ranks 18th out of 67 in the big league of “pathology” journals (ISI grouping), but within the smaller league of diagnostically oriented pathology journals, we more realistically rank sixth out of 26 (our grouping), which we consider to be quite satisfactory. In fact, JCP ranks second out of the European diagnostically oriented pathology journals, with Histopathology ranking first.

Because we know from our own citation analysis that JCP and MP papers have a similar citation frequency, the combined impact factor can also be used to compare MP with other molecular journals. MP ranks first of all six pathology journals featuring “molecular” in their title with regard to all bibliometric features, except that Diagnostic Molecular Pathology has a slightly better immediacy index.

Nevertheless, we would like to stress that we do not only care whether JCP and MP papers get cited, but also whether they are read. For example, we know that our Best Practice papers^9–13 are well read (many of them appear in our “top ten” of most accessed papers through our website), well appreciated, and probably often photocopied (and certainly their PDFs are downloaded many times), but they are not among our citation classics. Obviously, the extent to which papers are read is much more difficult to measure than how often they are cited. However, new developments in electronic publishing have opened new possibilities, as we will discuss later. We were pleased to note that JCP received up to 50 000 hits/month on its website (www.jclinpath.com) immediately after it opened, which compares well with some of the larger journals within the BMJ Publishing Group.

Where to find impact factors?

Many few journals advertise their impact factor on their websites (especially those that are proud of them!), but otherwise they are not so easy to find. The libraries of most of the bigger institutions will have a subscription to ISI (http://www.isinet.com/isi/), which provides access to the impact factors. If not, there are some websites that can be freely accessed to check on impact factors (http://www.biblioteca.cbpf.br/fator_e.html or http://www.med-rz.uni-sb.de/ubuklu/impact.html). However, these may not display the most recent impact factor data.

Are impact factors useful?

The impact factor is increasingly perceived as an important bibliometric feature to judge the quality of the output of researchers and their groups. Many researchers have to provide the impact factors of their best publications when submitting a grant proposal. For some, next year's budget will depend on the quality (as measured by the impact factor) and quantity of the research output. These facts help to explain the growing interest in (or obsession with) the impact factor for both authors and editors. Some librarians also use the impact factor to manage their journal collections. Is this really justified?

Impact factors tell us only how often articles in a certain journal are cited in the relatively short term. There is no doubt that this is a useful measure, especially for scientific journals. However, journals that are more clinically and diagnostically oriented also care about the extent to which they are read, and whether a paper continues to be read after some time, although this is more difficult to measure. To give an example, the total number of citations to JCP/MP papers in 2000 was no less than 7144, which means that we ranked seventh of all pathology journals for total number of citations. However, only 930 of these were to papers from the years 1998 and 1999, and therefore counted for the impact factor. The other 6214 citations did not help our impact factor, but these figures do indicate that these “older” papers still contain information that is worth citing. Therefore, other means of calculating the additional worth of papers need to be explored. In addition, the impact factor is itself imperfect, as explained below. Lastly, there should be more attention to the social impact of research,¹⁴ and to this end, the impact factor is of little help. A Dutch initiative is now exploring alternative ways to assess this factor.¹⁵

Can impact factors be “spun”?

Although one should not admit this (or should use the word “inflated” instead),¹⁶ the answer is yes, at least to some extent. Just imagine the effect of accepting only those papers with a certain number of citations to papers in that journal published in the past two years. Other options are to be liberal in designating short papers as letters,^17–20 hoping that they will not be counted. A very elegant way is to have many editorials,^21–25 which are not counted, but that cite as many papers as possible from the past two years in the same journal. In addition, letters to the editor relating to previously published papers are most welcome,^26–31 at least when they are published in the year after the original paper appeared. In fact, it has been noted before that such “uncitables” have an inordinate effect on the impact factor, and as a consequence, the impact factor ranking of a considerable number of journals, including the most esteemed journals, can be inflated by 30–40%.¹⁶

In addition, there are many other problems with impact factors as reviewed by Seglen,³² which we will not discuss in detail, but will only mention; namely: (1) impact factors are not statistically representative of individual journal articles; (2) impact factors correlate poorly with citations of individual articles; (3) authors use many criteria other than impact factors when submitting to journals; (4) self citations are not corrected for; (5) long articles collect many citations and give high journal impact factors; (6) a short publication time lag allows many short term journal self citations and gives a high journal impact factor; (7) citations in the national language of the journal are preferred by the journal's authors; (8) selective journal self citation: articles tend preferentially to cite other articles in the same journal; (9) coverage of the database is not complete; (10) the database has an English language bias and is dominated by American publications; (11) the journal set in the database may vary from year to year; (12) the impact factor is a function of the number of references/article in the research field; (13) research fields with literature that rapidly becomes obsolete are favoured; (14) the impact factor depends on dynamics (expansion or contraction) of the research field; (15) small research areas tend to lack journals with high impact; (16) relations between fields (for example, clinical versus basic research) strongly determine the journal impact factor; and (17) the citation rate of an article determines the journal impact, but not vice versa.

The impact factor is therefore clearly imperfect, and its meaning should not be overestimated. A more honest measure of the quality of a paper, rather than the impact factor of the journal it is published in, may be derived from the number of times it is cited. This information can also be retrieved from ISI through the website “Web of Science”, for which institutes need a subscription, which is unfortunate because it is very useful. This is a mandatory site for editors trying to learn from a citation analysis those papers that do well and those that do not, and for vain authors who want to know how often their papers are cited, and by whom. For authors, the mean number of citations to their paper each year (for example, calculated in the first five or 10 years after publication) might provide a fairly reliable “citation factor”. In the past, one could argue that papers in bigger journals with a wider circulation would be better read and more easily retrieved. These days, with huge literature databases freely available online (such as PubMed at http://www.ncbi.nlm.nih.gov/entrez/query.fcgi), this is probably no longer true. However, it does mean that journals need a website for electronic retrieval^22,33 to survive in the long term.

Are there alternatives?

As stated above, for individual papers the mean number of citations each year (calculated in the first five or 10 years after publication) may provide a fairly reliable citation factor, but for journals, other existing bibliometric features deserve more attention. The impact factor could be based on the previous year's articles alone, which would give an even greater weight to rapidly changing fields. This would probably not be favourable for JCP, so we will not advocate this. It would probably be better for us to go beyond two years retrospectively for the “source items” in the denominator, thereby taking longer periods into account. Nevertheless, when journals are studied within disciplinary categories, the rankings based on one, seven, or 15 year impact factors do not seem to differ much, as was recently reported in The Scientist.^34–36 The “citation density” (mean number of references cited for each article) should be considered because there are large differences between different disciplines. Citation studies should be normalised to take into account variables such as field, or discipline, and citation practices.³² The half life (number of years, going back from the current year, that cover 50% of the citations in the current year to the journal) is also quite important. For JCP/MP, the half life is high—8.4 in 2000—which ranks third in the first 25 pathology journals. Other features provided by ISI are the immediacy index (citations in a year to articles published in the same year divided by the number of source items in that year). For 2000, the immediacy index for JCP was 0.31, which ranks 15th when all pathology journals are considered, and seventh when only diagnostically oriented pathology journals are considered (table 1).

Perhaps the product of the impact factor and the half life would provide an interesting new measure. For such a combined factor, JCP/MP would rank 13th of all pathology journals and fifth of the diagnostically oriented pathology journals. However, information derived from websites could also be important. On our website (http://www.jclinpath.com), we monitor how many times individual papers are being accessed, from which we compose our top ten papers of most frequently accessed papers. It is also possible to monitor which papers are electronically ordered through PubMed or other electronic libraries such as Ovid.

These novel electronic data provide very useful information on the interest that individual papers raise, and thus the journals in which they are published. Electronic access information from journals' websites, PubMed, and Ovid could be used to compose an “access factor” for papers and thereby for journals. It is certainly true that this could at present be easily manipulated. It is not hard to imagine overzealous and frustrated editors accessing their own website all day. However, if the combined forces of different publishers, PubMed, and Ovid could produce a way of preventing this from being manipulated, the access factor could provide a new view on the “quality” of papers and journals.

Conclusion

The impact factor is a useful quality measure of scientific journals, but on its own provides a too limited idea on the extent to which papers in journals are being read and appreciated for daily practice. More attention should be paid to already existing alternative bibliometric scores, such as half life and total number of citations, and “electronic hit data” derived from websites should be used in an intelligent way to provide an electronic access factor. In addition, more attention should be given to the social impact of research.

More importantly, because the publication of a paper in a high impact factor journal does not generate more citations than the paper deserves (the “free ride” hypothesis³²), choosing to submit a paper to the journal with the proper audience should be one of the major concerns for authors. Because JCP has a wide circulation and a broad cross specialty audience, pathology papers are in good hands with us!