Publishing scientific papers originally was meant as a form of communication with peers and perhaps a first step towards the application of new findings. Increasingly since the late 1980s, publishing papers has become a goal in itself.
HIV/AIDS research in the early 1980s was a new and exciting field of science. I had started working as a biomedical researcher in Amsterdam, a city with a large and visible gay community. The new disease was a threat to public health and was highly contagious. It was transmitted by sexual contact and in the developed world affected young healthy gay men and recipients of blood and blood products. It took some time to realise that a truly immense and devastating epidemic was going on in sub-Saharan Africa affecting men, women and children.
This disease attracted bright scientific minds all over the world, working feverishly to understand the origin and biology of the virus. We wanted to know how the virus moved through the population, entered and killed immune cells and how to counteract it. AIDS patients were dying in the hospitals and we were working as fast as we could towards better therapies for HIV-positive patients.
Or were we?
I was very proud when results of experiments from my laboratory were published in prestigious academic journals like Nature, Science and The Lancet. I felt I had made a significant contribution to understanding and battling HIV. As well as at scientific conferences, we presented our results to participants of the Amsterdam Cohort Studies which started in the late 1980s. These were mainly gay men who helped our research by donating blood samples and filling in lifestyle questionnaires. One evening I presented with my usual enthusiasm new results on how HIV destroyed white blood cells of the immune system.
Then a man came to the microphone. “Doctor Miedema, thank you for your interesting talk, but to be honest, it was a bit over my head, with apoptosis, virus particles and what have you. However, what I would like to know is whether we should practice safe sex even when my partner and I are already HIV-positive.”
I was flabbergasted. Here I was with my clever immunological experiments and detailed molecular understanding of the virus, but I couldn’t answer this real-world question. And the question made sense. Rephrased in viro-immunological terms: is it possible and, if so, is it bad to become co-infected with a different virus strain? Can mosaic viruses with increased pathogenicity emerge? We, the smart boys and girls in the lab, hadn’t thought of that question. Why not? Because it hadn’t come up in the lab. We had informed the patients but forgotten to talk to them, the people we were supposedly working so hard for.
Fast forward 30 years. Biomedical science has grown exponentially. Based on past successes in understanding diseases and developing cures, billions of taxpayers’ money flow into the biomedical scientific enterprise worldwide each year. The numbers of scientists and scientific papers have increased accordingly. Calculations differ, but over 1.5 million peer-reviewed scientific papers are published each year.
This tsunami of papers would suggest a vastly expanding reservoir of knowledge with enormous societal impact. Unfortunately, this is not the case. Many papers are never cited and probably never read. Even worse, many supposedly high-quality papers in the life sciences, describing breakthroughs or possible therapeutic targets, cannot be reproduced.
Why then, do academics churn out all these papers? In my view, the large size of the scientific community combined with increased calls for accountability, have paved the way for quantitative performance assessment, especially in the biomedical field. Assessing ‘scientific quality’ has been reduced to bean counting. Scientists are judged by the number of papers they publish and by the impact factors of the journals these papers are published in.
This has led to major goal displacement. Publishing scientific papers originally was meant as a form of communication with peers and perhaps a first step towards the application of new findings. Increasingly since the late 1980s, publishing papers has become a goal in itself.
This diagnosis is hardly new. The San Francisco Declaration on Research Assessment (DORA) in 2012 was a response to this tendency. The METRICS Center in Stanford aims steer away from bibliometrics to improve quality and relevance of biomedical research. And The Metric Tide report in the UK also pointed to goal displacement when using bibliometric indicators for scientific quality.
In 2013 three colleagues and I started a nationwide debate in the Netherlands about quality and relevance of research under the name Science in Transition. We explicitly linked the primacy of research output in the evaluation of academics to the diminished appreciation of teaching, outreach, translational research, engagement with knowledge users outside academia, peer review and other academic duties. We also pointed to the dominance of ‘science for science’ in current evaluation practices.
We believe that societal impact should be explicitly valued when designing and doing research, since one of the roles, if not the major aim, of science is to help confront grand societal challenges. There is a need for new indicators of scientific quality, but we also suggest the need to rethink how research agendas are formulated. As Helga Nowotny describes in her latest, very thoughtful book, science, alongside the social sciences and humanities, must confront the big challenges of the 21st century.
We debated these issues in March at the third Science in Transition conference at the Royal Academy of Sciences in Amsterdam. A young PhD researcher described vividly how PhD students are ‘used’ to produce peer-reviewed papers and how this threatens scientific integrity and hampers personal development of young researchers. We debated the ‘Dutch National Research Agenda’, an experiment in which Dutch citizens could submit questions to science, resulting in 12,000 questions, later clustered in 140 themes. This was symbolically important, producing a list of topics but also suggesting the need for further articulation. The Science in Transition agenda on the need to rethink the incentive and reward system of science will be part of the Amsterdam Call for Action on Open Science which will be published in the context of Netherlands’ Presidency of the EU.
Meanwhile in the University Medical Center Utrecht we are putting these ideas into practice. In our latest research evaluation we didn’t just quantify past performance, but looked forwards too. We ask for public involvement in the evaluation, with representatives from patient organizations, health charities, private parties and health insurers.
In our next evaluation we will be piloting a new framework developed in-house. These new indicators aim to stimulate dialogue about how to improve research structures and processes for societal impact. We believe the main questions reviewers should keep in mind are “Why are you doing this research?” and “How did you decide on that set of questions?” instead of “what have you measurably produced?”
To return to the HIV-positive man at the microphone. His question was translated into biological terms in subsequent grant proposals and has been adequately addressed since. The conclusion is that there is only a low risk for super-infection and when it happens there is no increased risk for disease progression. Since its beginning, AIDS research has been exemplary for the strength of patient advocacy. Stakeholders, patients and researchers found a shared working space during AIDS conferences that shaped science for the better. Let’s follow that example to battle the acute and chronic challenges that lie ahead.
On behalf of Science in Transition and the University Medical Center Utrecht, The Netherlands. Frank Miedema is Dean and Vice Chairman of the Board of UMC Utrecht, professor of immunology. With thanks to Rinze Benedictus for editorial advice.
guardian.co.uk © Guardian News & Media Limited 2010