Article Text

PDF

Does activity in research correlate with visibility?
  1. T M Reynolds1,
  2. A S Wierzbicki2
  1. 1Queen’s Hospital, Belvedere Road, Burton on Trent, Staffordshire DE13 0RB, UK
  2. 2St Thomas’ Hospital, Lambeth Palace Rd, London SE1 7EH, UK
  1. Correspondence to:
 Professor T M Reynolds
 Clinical Chemistry Department, Queen’s Hospital, Belvedere Road, Burton on Trent, Staffordshire DE13 0RB, UK; tim.reynoldsqueens.burtonh-tr.wmids.nhs.uk

Abstract

Background: A previous survey has highlighted the fact that most individuals in chemical pathology identifiable from specialist society membership failed to publish material in Medline cited journals during a five year period. It could be considered that published research that is not cited in other work is not useful unless it has achieved visibility, as demonstrated by citation in another research publication.

Aims: To determine whether the frequency of research publication is associated with research visibility.

Methods: A random selection from the previous survey was investigated to determine whether the frequency of research publication is associated with research visibility.

Results: There was a logarithmic relation between the frequency of publication and visibility, with an increasing probability of citation as publication frequency increases.

Conclusions: If academic activity is to survive then individuals must stay active in research; this requires a continuing commitment to a tradition of support for individuals at all stages of their careers engaging in research.

  • cement worker
  • mortality
  • cancer incidence

Statistics from Altmetric.com

The formal assessment of research publications is used to grade individuals, departments, and universities in the UK. In general, individuals are ranked on the journal in which their work is published and many researchers aim to publish in journals of high impact factor, defined as total journal citations divided by articles published over the past two years. However, it is well known that citation rates for individual articles in journals show a wide degree of variation, and it has been suggested that citation rates might be more informative than impact factors for the assessment of research quality. However, research productivity is only one aspect of scientific endeavour. It could be argued that the most erudite piece of work is useless if no one reads it. Clearly, it is impossible to assess whether research papers have been read, but it is possible to identify when they have been cited. This study sought to compare citation rates with total publication production for medical and scientific researchers working in chemical pathology.

METHODS

We have previously reported a study in which 1399 individuals in a single specialty (chemical pathology) were literature searched for the period 1994–8 inclusive using the Medline database.1 Using the database from our initial study, a sample of randomly chosen individuals was identified comprising: 10% of those with one publication (14 individuals), 10% of people with two to five publications (19 individuals), 20% of people with six to 10 publications (17 individuals), 10% of people with 11–20 publications (13 individuals), and 20% of people with > 20 publications (10 individuals). Citations for each article by each author were identified using the Web of Science database held at the University of Manchester (http://wos.mimas.ac.uk). Data were exported to a spreadsheet and analysed by non-parametric statistics or by linear regression after log transformation to correct for the skewed nature of the data.

RESULTS

The retrospective nature of the survey (1994–8) in contrast to the citation search (2003) allowed correction for the lag in citation that follows publication. It became clear that in some large fields very high citation rates were possible (for example, one paper from the UK prevention of diabetes study had 350 citations), whereas in other fields with fewer researchers and a smaller body of literature (for example, Down’s syndrome screening, where the total body of publications numbers only approximately 900) much lower citation rates were just as significant. Therefore, we expressed “useful research productivity” as the probability that any paper would be cited by another author over the next five to 10 years to correct for this potential bias.

Figure 1 shows the probability that publications will be cited plotted against the number of publications/five years identified in our earlier survey.1 The distribution follows a logarithmic trend. Thus, there appears to be a threshold effect, such that at a low yearly publication rate (less than three publications/year), the usefulness of research or credibility of the researcher is viewed to be lower than when more frequent publication rates are achieved. The more productive individuals tended to be cited at least three quarters of the time. Similarly, within individuals, certain articles are cited far more frequently than others and individual citation distributions also follow a logarithmic distribution. Unfortunately, it is not yet possible to identify how much of a citation rate is self citation, although this will certainly contribute to the score.

Figure 1

The relation of the probability (P) of citation compared with research productivity in terms of papers published.

The original research productivity was highly skewed, with fewer than 50% of staff in chemical pathology publishing even once in a five year period.1 At least one letter was published by 49% of staff; 20% published one original piece of work in five years; and only 4% could be defined as research active when defined by the internationally accepted standard of one publication/year.1 Similarly, citation rates even in research active individuals were highly skewed, with citation rates varying up to 10-fold, both within the work of individual researchers and across the population. As might be expected, review articles were more frequently cited than original scientific papers, and these were cited more than letters commenting on research. This implies that more active researchers are more likely to produce a well cited paper, and that the nature of science is essentially statistical and probabilistic.

DISCUSSION

This work tends to confirm the suggestion that was advanced in the previous survey that research follows a “winner takes all” pattern, with disproportionate returns to those achieving initially only slightly better outcomes.1,2 The effect is probably exacerbated by the natural tendency among researchers to collaborate with established workers in the field and the network effects that this can generate.3 This law of positive returns also implies negative selection for those maintaining low rates of activity, suggesting that it may be very difficult for either individuals, departments, or disciplines to recover from falling research productivity and hence falling citations and lower impact factors for specialist journals in their field. This maintenance of research activity in a field such as chemical pathology requires a continuing commitment to a tradition of support for individuals at all stages of their careers engaging in research. If this does not occur then academic activity will rapidly disappear.

“Better ranking systems require individual review of papers by peer assessors with specialist knowledge in the field”

However, the common practice of using journal impact factors or citation counts as quality indicators takes competitive measures of journal performance including speed of review, electronic or paper access, and editorial decisions (such as newsworthiness or article reference limits) and applies them inappropriately to individuals.4–7 Better ranking systems require individual review of papers by peer assessors with specialist knowledge in the field.4–7 It is notable that even Nobel prize winning research may not be extensively cited because extremely important articles, such as those of Salk and Sabin, had only received 39 and 90 citations, respectively, by 1987.7

Our study confirms that the skewed distribution of citations in chemical pathology matches that of publication rates and journal impact factors, with more active researchers producing better known work (higher impact factor and citation rate) than relatively inactive researchers. The use of citation rates and journal impact factors is a common but inappropriate way of judging the quality of research. Yet, because grant funding bodies and university departments use these measures it is incumbent upon the profession to point out their limitations. Chemical pathologists must stay active in research or disappear; in addition, they need to devise mechanisms that can assess and review quality in research within the profession, because external assessors are not capable of performing this function accurately.

Take home messages

  • In chemical pathology, more active researchers produce work with a higher impact factor and citation rate than relatively inactive researchers

  • The use of citation rates and journal impact factors is a common but inappropriate way of judging the quality of research, although these measures are used by grant funding bodies and university departments

  • If academic activity is to survive then individuals must stay active in research; this requires a continuing commitment to a tradition of support for individuals at all stages of their careers engaging in research

COMPETING INTERESTS

Both authors were included in the database used in the first study and one was randomly selected to be in the second study.

REFERENCES

View Abstract

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.