What the Tamiflu saga tells us about drug trials and big pharma

Today we found out that Tamiflu doesn't work so well after all. Roche, the drug company behind it, withheld vital information on its clinical trials for half a decade, but the Cochrane Collaboration, a global not-for-profit organisation of 14,000 academics, finally obtained all the information. Putting the evidence together, it has found that Tamiflu has little or no impact on complications of flu infection, such as pneumonia.

That is a scandal because the UK government spent £0.5bn stockpiling this drug in the hope that it would help prevent serious side-effects from flu infection. But the bigger scandal is that Roche broke no law by withholding vital information on how well its drug works. In fact, the methods and results of clinical trials on the drugs we use today are still routinely and legally being withheld from doctors, researchers and patients. It is simple bad luck for Roche that Tamiflu became, arbitrarily, the poster child for the missing-data story.

And it is a great poster child. The battle over Tamiflu perfectly illustrates the need for full transparency around clinical trials, the importance of access to obscure documentation, and the failure of the regulatory system. Crucially, it is also an illustration of how science, at its best, is built on transparency and openness to criticism, because the saga of the Cochrane Tamiflu review began with a simple online comment.

In 2009, there was widespread concern about a new flu pandemic, and billions were being spent stockpiling Tamiflu around the world. Because of this, the UK and Australian governments specifically asked the Cochrane Collaboration to update its earlier reviews on the drug. Cochrane reviews are the gold-standard in medicine: they summarise all the data on a given treatment, and they are in a constant review cycle, because evidence changes over time as new trials are published. This should have been a pretty everyday piece of work: the previous review, in 2008, had found some evidence that Tamiflu does, indeed, reduce the rate of complications such as pneumonia. But then a Japanese paediatrician called Keiji Hayashi left a comment that would trigger a revolution in our understanding of how evidence-based medicine should work. This wasn't in a publication, or even a letter: it was a simple online comment, posted informally underneath the Tamiflu review on the Cochrane website, almost like a blog comment.

Cochrane had summarised the data from all the trials, explained Hayashi, but its positive conclusion was driven by data from just one of the papers it cited: an industry-funded summary of 10 previous trials, led by an author called Kaiser. From these 10 trials, only two had ever been published in the scientific literature. For the remaining eight, the only available information on the methods used came from the brief summary in this secondary source, created by industry. That's not reliable enough.

This is science at its best. The Cochrane review is readily accessible online; it explains transparently the methods by which it looked for trials, and then analysed them, so any informed reader can pull the review apart, and understand where the conclusions came from. Cochrane provides an easy way for readers to raise criticisms. And, crucially, these criticisms did not fall on deaf ears. Dr Tom Jefferson is the head of the Cochrane respiratory group, and the lead author on the 2008 review. He realised immediately that he had made a mistake in blindly trusting the Kaiser data. He said so, without defensiveness, and then set about getting the information needed.

First, the Cochrane researchers wrote to the authors of the Kaiser paper. By reply, they were told that this team no longer had the files: they should contact Roche. Here the problems began. Roche said it would hand over some information, but the Cochrane reviewers would need to sign a confidentiality agreement. This was tricky: Cochrane reviews are built around showing their working, but Roche's proposed contract would require them to keep the information behind their reasoning secret from readers. More than this, the contract said they were not allowed to discuss the terms of their secrecy agreement, or publicly acknowledge that it even existed. Roche was demanding a secret contract, with secret terms, requiring secrecy about the methods and results of trials, in a discussion about the safety and efficacy of a drug that has been taken by hundreds of thousands of people around the world, and on which governments had spent billions. Roche's demand, worryingly, is not unusual. At this point, many in medicine would either acquiesce, or give up. Jefferson asked Roche for clarification about why the contract was necessary. He never received a reply.

Then, in October 2009, the company changed tack. It would like to hand over the data, it explained, but another academic review on Tamiflu was being conducted elsewhere. Roche had given this other group the study reports, so Cochrane couldn't have them. This was a non-sequitur: there is no reason why many groups should not all work on the same question. In fact, since replication is the cornerstone of good science, this would be actively desirable.

Then, one week later, unannounced, Roche sent seven documents, each around a dozen pages long. These contained excerpts of internal company documents on each of the clinical trials in the Kaiser meta-analysis. It was a start, but nothing like the information Cochrane needed to assess the benefits, or the rate of adverse events, or fully to understand the design of the trials.

At the same time, it was rapidly becoming clear that there were odd inconsistencies in the information on this drug. Crucially, different organisations around the world had drawn vastly different conclusions about its effectiveness. The US Food and Drug Administration (FDA) said it gave no benefits on complications such as pneumonia, while the US Centers for Disease Control and Prevention said it did. The Japanese regulator made no claim for complications, but the European Medicines Agency (EMA) said there was a benefit. There are only two explanations for this, and both can only be resolved by full transparency. Either these organisations saw different data, in which case we need to build a collective list, add up all the trials, and work out the effects of the drug overall. Or this is a close call, and there is reasonable disagreement on how to interpret the trials, in which case we need full access to their methods and results, for an informed public debate in the medical academic community.

This is particularly important, since there can often be shortcomings in the design of a clinical trial, which mean it is no longer a fair test of which treatment is best. We now know this was the case in many of the Tamiflu trials, where, for example, participants were sometimes very unrepresentative of real-world patients. Similarly, in trials described as "double blinded" – where neither doctor nor patient should be able to tell whether they're getting a placebo or the real drug – the active and placebo pills were different colours. Even more oddly, in almost all Tamiflu trials, it seems a diagnosis of pneumonia was measured by patients' self-reporting: many researchers would have expected a clear diagnostic algorithm, perhaps a chest x-ray, at least.

Since the Cochrane team were still being denied the information needed to spot these flaws, they decided to exclude all this data from their analysis, leaving the review in limbo. It was published in December 2009, with a note explaining their reasoning, and a small flurry of activity followed. Roche posted their brief excerpts online, and committed to make full study reports available. For four years, they then failed to do so.

During this period, the global medical academic community began to realise that the brief, published academic papers on trials – which we have relied on for many years – can be incomplete, and even misleading. Much more detail is available in a clinical study report (CSR), the intermediate document that stands between the raw data and a journal article: the precise plan for analysing the data statistically, detailed descriptions of adverse events, and so on.

By 2009, Roche had shared just small portions of the CSRs, but even this was enough to see there were problems. For example, looking at the two papers out of 10 in the Kaiser review that were published, one said: "There were no drug-related serious adverse events", and the other doesn't mention adverse events. But in the CSR documents shared on these same two studies, 10 serious adverse events were listed, of which three are classified as being possibly related to Tamiflu.

By setting out all the known trials side by side, the researchers were able to identify peculiar discrepancies: for example, the largest "phase three" trial – one of the large trials that are done to get a drug on to the market – was never published, and is rarely mentioned in regulatory documents.

The chase continued, and it exemplifies the attitude of industry towards transparency. In June 2010, Roche told Cochrane it was sorry, but it had thought they already had what they wanted. In July, it announced that it was worried about patient confidentiality. By now, Roche had been refusing to publish the study reports for a year. Suddenly, it began to raise odd personal concerns. It claimed that some Cochrane researchers had made untrue statements about the drug, and about the company, but refused to say who, or what, or where. "Certain members of Cochrane Group," it said, "are unlikely to approach the review with the independence that is both necessary and justified." This is hard to credit, but even if true, it should be irrelevant: bad science is often published, and is shot down in public, in academic journals, by people with good arguments. This is how science works. No company or researcher should be allowed to choose who has access to trial data. Still Roche refused to hand over the study reports.

Then Roche complained that the Cochrane reviewers had begun to copy in journalists, including me, on their emails when responding to Roche staff. At the same time, the company was raising the broken arguments that are eerily familiar to anyone who has followed the campaign for greater trials transparency. Key among these was one that cuts to the core of the culture war between evidence-based medicine, and the older "eminence-based medicine" that we are supposed to have left behind. It is simply not the job of academics to make these decisions about benefit and risk, said Roche, it is the job of regulators.

This argument fails on two fronts. First, as with many other drugs, it now seems that not even the regulators had seen all the information on all the trials. But more than that, regulators miss things. Many of the most notable problems with medicines over the past few years – with the arthritis drug Vioxx; with the diabetes drug rosiglitazone, marketed as Avandia; and with the evidence base for Tamiflu – weren't spotted primarily by regulators, but rather by independent doctors and academics. Regulators don't miss things because they are corrupt, or incompetent. They miss things because detecting signals of risk and benefit in reviews of clinical trials is a difficult business and so, like all difficult questions in science, it benefits from having many eyes on the problem.

While the battle for access to Tamiflu trials has gone on, the world of medicine has begun to shift, albeit at a painful pace, with the European Ombudsman and several British select committees joining the push for transparency. The AllTrials campaign, which I co-founded last year, now has the support of almost all medical and academic professional bodies in the UK, and many more worldwide, as well as more than 100 patient groups, and the drug company GSK. We have seen new codes of conduct, and European legislation, proposing improvements in access: all riddled with loopholes, but improvements nonetheless. Crucially, withholding data has become a headline issue, and much less defensible.

Last year, in the context of this wider shift, under ceaseless questions from Cochrane and the British Medical Journal, after half a decade, Roche finally gave Cochrane the information it needed.

So does Tamiflu work? From the Cochrane analysis – fully public – Tamiflu does not reduce the number of hospitalisations. There wasn't enough data to see if it reduces the number of deaths. It does reduce the number of self-reported, unverified cases of pneumonia, but when you look at the five trials with a detailed diagnostic form for pneumonia, there is no significant benefit. It might help prevent flu symptoms, but not asymptomatic spread, and the evidence here is mixed. It will take a few hours off the duration of your flu symptoms. But all this comes at a significant cost of side-effects. Since percentages are hard to visualise, we can make those numbers more tangible by taking the figures from the Cochrane review, and applying them. For example, if a million people take Tamiflu in a pandemic, 45,000 will experience vomiting, 31,000 will experience headache and 11,000 will have psychiatric side-effects. Remember, though, that those figures all assume we are only giving Tamiflu to a million people: if things kick off, we have stockpiled enough for 80% of the population. That's quite a lot of vomit.

Roche has issued a press release saying it contests these conclusions, but giving no reasons: so now we can finally let science begin. It can shoot down the details of the Cochrane review – I hope it will – and we will edge towards the truth. This is what science looks like. Roche also denies being dragged to transparency, and says it simply didn't know how to respond to Cochrane. This, again, speaks to the pace of change. I have no idea why it was withholding information: but I rather suspect it was simply because that's what people have always done, and sharing it was a hassle, requiring new norms to be developed. That's reassuring and depressing at the same time.

Should we have spent half a billion on this drug? That's a tricky question. If you picture yourself in a bunker, watching a catastrophic pandemic unfold, confronting the end of human civilisation, you could probably persuade yourself that Tamiflu might be worth buying anyway, even knowing the risks and benefits. But that final clause is the key. We often choose to use treatments in medicine, knowing that they have limited benefit, and significant side-effects: but we make an informed decision, balancing the risks and benefits for ourselves.

And in any case, that £500m is the tip of the iceberg. Tamiflu is a side show, the one place where a single team of dogged academics said "enough" and the company caved in. But the results of clinical trials are still being routinely and legally withheld on the medicines we use today and nothing about a final answer on Tamiflu will help plug this gaping hole.

More importantly, for all that there is progress, so far we have only sentiment, and half measures. None of the changes to European legislation or codes of conduct get us access to the information we need, because they all refer only to new trials, so they share a loophole that excludes – remarkably – all the trials on all the medicines we use today, and will continue to use for decades. To take one concrete and topical example: they wouldn't have made a blind bit of difference on Tamiflu. We have seen voluntary pledges for greater transparency from many individual companies – Johnson & Johnson, Roche, GSK, now Roche, and more – which are welcome, but similar promises have been given before, and then reversed a few years later.

This is a pivotal moment in the history of medicine. Trials transparency is finally on the agenda, and this may be our only opportunity to fix it in a decade. We cannot make informed decisions about which treatment is best while information about clinical trials is routinely and legally withheld from doctors, researchers, and patients. Anyone who stands in the way of transparency is exposing patients to avoidable harm. We need regulators, legislators, and professional bodies to demand full transparency. We need clear audit on what information is missing, and who is withholding it.

Finally, more than anything – because culture shift will be as powerful as legislation – we need to do something even more difficult. We need to praise, encourage, and support the companies and individuals who are beginning to do the right thing. This now includes Roche. And so, paradoxically, after everything you have read above, with the outrage fresh in your mind, on the day when it feels harder than any other, I hope you will join me in saying: Bravo, Roche. Now let's do better.