The scientific literature comprises scholarly publications that report original empirical and theoretical work in the natural and social sciences.
It is the principal medium for communicating the results of scientific research and, as such, represents the permanent record of the collective achievements of the scientific community over time. The scope of the literature is hard to conceive. More than 80% of the scientific research literature is published as articles in subject-specific journals. Estimates suggest that more than 1.5 million such papers are published in over 25,000 peer-reviewed journals annually and that over 50 million have been published since the first scientific journal was published in 1665.
The contribution each of the constituent parts of the literature makes to our collective scientific knowledge is a subject of some debate. Value is often determined by the number of times an article is cited in other scientific articles. Over the last decade a new field of study has exploded onto the scientific landscape – that of bibliometrics (you can download a useful description of the field here). Qualitative analysis of the scientific literature is changing rapidly with the creation of new evaluation tools, parameters and normative data. These parameters can be categorised into author or journal focused metrics. Once you have an understanding of the field of bibliometrics it is immediately obvious that publication metrics fail to give a true assessment of research ‘quality’, whatever that may be; it is not yet possible to derive a simple indication of the value of a piece of work. You still have to determine that for yourself.
People have different ideas about how much each part of the literature adds to our overall science knowledge. A scientific article's value is often based on how many times it is cited in other scientific pieces. Bibliometrics is a new field of study that has exploded onto the scientific scene in the last ten years [1]. There are a lot of changes happening quickly in qualitative analysis of the scientific literature as new review tools, parameters, and normative data are made. You can divide these factors into metrics that focus on authors or metrics that focus on journals. Once you know about bibliometrics, it's clear that publication metrics don't really give a true picture of study "quality," whatever that might mean; it's still not possible to get a simple idea of how much a piece of work is worth. That's still something you need to decide for yourself.
A number of scientific works are never cited. This has been known for a while. Some people have thought about and talked about how big and useful the uncited literature is. Up to half of all papers have not yet been cited for 5 years after they were published, according to earlier figures. Even papers by Nobel Prize winners are not cited 10% of the time. More than half of all academic works are still not cited after 5 years. This is a fact that is often brought up. Researchers often worry that a lot of papers that aren't cited mean that the study is going in a direction that isn't useful or important.
In order to get a better idea on the extent of this undiscovered country of published research, Nature recently [2] performed its own analysis of just how many papers go uncited. In their article it is noted how difficult it can be to get a grasp on accurate numbers. In this case, the author reviewed data on a core group of 12,000 journals catalogued in the Web of Science. They found evidence that previous estimates are far too high. Looking at recent papers it appeared that fewer than 10% remain uncited. This may also be an over estimate as data recorded on Web of Science does not include citations in journals that are not included on the database they used or appear in books, patents, etc. As for the work of Nobelist’s, it seems that just 0.3% of their work goes uncited.
A closer look at the data suggests that the proportion of uncited papers has been higher in the past and uncited numbers falling steadily. For example, 20% of papers published in 1980 still don't have a single citation today. This could well reflect the revolution we have experienced in the way we undertake background research following the emergence of electronic literature search engines over the last 30 years in addition to the growth in the number of references cited in each paper over this same period.
The authors of the Nature paper note that we shouldn’t assume that an uncited paper is worthless – it is still possible for the work to affect researchers and practitioners. It can still influence readers who may change the way they do something or think but who may not necessarily make comment about the change within the literature. In itself this is worrisome. We have long known that a high proportion of clinical trials, for example, go unpublished – effectively excluding any findings, positive or negative, from consideration [3]. Here we enter a truly undiscovered country. Although it has been suggested that the pharmaceutical industry, in particular, is hesitant to publish negative results, much research suggests that this is not specific to industry-funded projects. In fact, there are serious concerns that in their pursuit for higher publishing metrics journals themselves are biasing the data available to researchers [1].
The idea that the literature is awash with uncited research goes back to a pair of articles published in Science in the early 1990s. The present evidence shows that concept these reports introduced is now an anachronism and misleading. The Nature article also points out that the original work might have been flawed in that the authors counted documents such as letters, corrections, meeting abstracts and other editorial material, which most scientists accept wouldn’t usually get cited. When these data are removed, leaving only research papers and review articles, rates of uncitedness are reduced.
The last 30 years have seen the way that we undertake our background research change beyond all recognition. Gone are the days of spending an afternoon leafing through the latest version of Index Medicus in the library, taking notes of relevant papers to review in the dusty journals kept in ‘the stacks’. They have been replaced elegant Boolean search strategies performed on multiple data platforms and generating comprehensive candidate articles to review and rank for relevance and importance. Younger researchers, born into these systems now adopt a learn-as-they-go approach, adapting their searching strategy ‘on the fly’. And this is perhaps where the true value of the whole body of the scientific literature comes in. Progress in science occurs through both evolution and revolution. The two are complementary and not mutually exclusive. The body of our research work as a whole, positive, negative and revolutionary, empowers progress. Although it may go uncited, even the most uninspiring paper can impact on our thinking and empower our next great discovery.
References
Congratulations class of 2024. That was all I needed to hear when I stood in your shoes almost 40 years ago. 40 years
Finding the best vehicle in which to publish your research findings is a perennial challenge.
In the past, Halloween has meant the end of the crop and the start of the dark, cold winter.
Investigator sites are very important to clinical studies because they make sure that new drugs are safe and work well.
The UK's world-class human challenge study gives us new information about SARS-CoV-2 cases that were mild in 36 healthy.
Have you seen our latest Insider's Insight on Case Reports? Once a staple of medical journals,
Investigator-Initiated Clinical Trials (IITs) are clinical studies led by academic researchers or clinicians
The proceedings of the Association for Human Pharmacology in the Pharmaceutical Industry (AHPPI)
The largest pharmaceutical companies not surprisingly rely on the largest contract research organizations
Paraskevidekatriaphobia is a phobia of Friday the 13th. Yes, they actually gave it a name.
Experience at Niche has taught us that clinical study protocols are born in the white-hot fire
I've been studying medicine for 40 years, and it's beginning to look like something out of science fiction
Our team are exploring different and novel ways for young people to enter the pharmaceutical industry.
Advisory board meetings are often used to address perceived knowledge gaps and build consensus.
Yes, it’s Star Wars Day again! The Star Wars films base their wow-factor on their adoption of science
Since 2013 we have been sharing our understanding on key issues with friends and clients via our Insider’s Insights.
Get our latest news and publications
Sign up to our news letterSocial
Contact us
Address
Niche Science & Technology
Unit 26 Falstaff House
Bardolph Road
Richmond TW9 2LH
United Kingdom
