The FDA’s Epic Statistical Fail & Worse: Has Informed Consent Been Made Next To Impossible?

Enthusiasm for an experimental intervention often gets the better of doctors’ critical thinking. I think that happened with the advocates for Covid-19 convalescent plasma – and inside the FDA as well. It was inevitable that an uncritical take from that agency would be grabbed and run with for political ends.

There has been a lot of discussion about the harm all that did to recruitment for randomized trials for the unproven treatment, from the point the FDA decided to expand access instead of super-charging a trial for a reliable answer. But what was all this doing to patients’ right to informed consent? Watching the Covid-19 convalescent plasma drama unfold across August got me thinking about this a lot. That epic statistical fail by the FDA Commissioner, and the publicity campaign around it, goes straight to the heart of this, so let’s start there.

I had to watch it twice to make sure this was a thing that really happened. It was an FDA Commissioner, at a particularly high-profile press conference. He was emphasizing a claim the President had just made about convalescent plasma – from the donated blood of people who have recovered from Covid-19. It was based on a 35% relative risk reduction from a post-hoc subgroup analysis in an observational study:

So let me just put this into perspective. Many of you know I was a cancer doctor before I became FDA Commissioner and a 35% improvement in survival is a pretty substantial clinical benefit. What that means is – and if the data continue to pan out – 100 people who are sick with Covid-19, 35 would have been saved because of the administration of plasma.

Stephen Hahn, FDA Commissioner, August 24

Everything about this is wrong. I’ve explained exactly why, and why the FDA’s whole assessment was a biased accounting of an unreliable body of historical and recent evidence in an op-ed at WIRED. And I’ve posted a critical comment on the preprint of the observational study that goes into its problems more deeply. But let’s focus here on the egregious wrong-ness of the “35 would have been saved” claim, and its implications.

The 35% risk difference in the data came from a comparison of the 2 smallest groups within a small subgroup of a big study of people who were severely ill with Covid-19 – and it was an observational study riddled with problems, so it couldn’t prove something like this. It was obviously ridiculous, too. Only about 20% of people with Covid-19 symptoms get severely ill – so nowhere near 35 out of 100 people sick with Covid-19 might die. So the 35% obviously isn’t what Hahn said it was.

The statistic he was riffing on was a relative risk reduction, but the explanation he gave was for an absolute risk reduction. A relative risk reduction is what I call risk communication’s magnifying glass. It can inflate a modest absolute reduction – which is the drop in percentage points – into a big one by using a proportion. Say my risk of something is 1%. If it drops to 0.5%, the absolute risk reduction is 0.5% (from 1% to 0.5%), but the relative risk reduction is 50% (because 0.5% is half of 1%).

In this case, a 35% difference obviously sounded way more dramatic than the 4.8% absolute risk reduction it came from. That can matter a lot. Giving people information in only relative risk format can mean they say “yes” to a treatment more often than people who are told the absolute difference.

You wouldn’t expect an FDA Commissioner to do this by mistake, because it would mean he had completely misunderstood the size of the difference convalescent plasma might be able to make. But the day after, Hahn made one of those “I mis-spoke” statements.

One day later, though, he said this:

I was trying to do what I do with patients, because patients often understand things in absolute terms versus relative terms. And I should’ve been more careful, there’s no question about it… [T]hese are really difficult concepts to explain to the American people. At the end of the day, I know my patients never cared about relative versus absolute benefits. They just wanted to understand if the juice is worth the squeeze.

That’s a model of paternalistic medical care, isn’t it? You get the impression that once he’s decided the “juice” is worth the “squeeze”, it’s up to him to sell it, not support informed decision-making. Even leaving the question of politicians’ motivations aside, it makes it easy to see why Hahn said, at a launch of the $8 million campaign to get more people who have recovered from Covid-19 to donate plasma, “You can literally save lives”.

That campaign began on July 30. It was PSAs, billboards – the works. With the nation’s community and doctors being bombarded with the message that convalescent plasma can save lives, it’s not only the uncertainty that’s the ethical basis for randomized trials that’s lost. The basis for informed decision making in hospitals is probably lost, too, as the message of enthusiasm for the treatment spreads through the community.

This is what the FDA’s information for doctors says: “Based on the totality of scientific evidence available at this time, the known and potential benefits of COVID-19 convalescent plasma outweigh the known and potential risks”. It provides no numbers at all, just vague adjectives. It says that plasma with high levels of antibodies “may be effective”, but plasma with low levels are authorized, too, and, “Health care providers can decide whether to use the units [labeled as low] based on an individualized assessment of patient benefit:risk”. And how on earth are they supposed to do that, when there’s no information to base that kind of assessment on?

The FDA’s patients’/caregivers’ information sheet strongly implies convalescent plasma has been shown to work for other respiratory diseases (it hasn’t), and on Covid-19, “This treatment might be effective in improving the likelihood of you recovering from the disease”. It also might not, or it might make very little difference, while exposing people to risk.

From the access program, to the donation publicity drive, to the campaigning around the FDA’s Emergency Use Authorization, and the formal product fact sheets for patients/caregivers and doctors, I think the FDA has created conditions that make informed consent for convalescent plasma next to impossible.

In the Netherlands, there was a randomized trial for convalescent plasma for Covid-19 that was stopped when the investigators realized it was impossible for the trial as designed to answer the question of whether it worked. Even though the mortality rate at that time for the hospitalized people eligible to participate was around 30%, 1 in 4 people declined – and the authors said that was typically because of fear of adverse effects. I think there are 2 lessons there. Firstly, if properly informed, enough people would have agreed to trial participation to get an answer in the U.S. very quickly. And secondly, it’s safe to assume that reliable data on the benefit to risk ratio for them would be critical to many people’s decisions. But with one action after another, the FDA and the full force of the U.S. government’s communication apparatus have propelled the community towards transfusing an unproven blood product.

People keep saying the FDA’s decision was politicized. However it was already clear at the beginning of August that Peter Marks, the doctor that leads the section of the FDA responsible for these decisions, wasn’t being critical enough about the shaky historical and recent evidence. But where are the lines between garden-variety medical over-enthusiasm, and government propaganda?

In 2018, Vlan Bakir and colleagues published a conceptual framework for research on public relations, propaganda, and promotional culture that I found helpful. They categorize it all as “organized persuasive communication”, and it is, they wrote, “central to the exercise of power across all social spheres”:

Scholars of PR (and related fields), as well as promotional culture, focus on what they perceive to be non-manipulative forms of OPC occurring within contemporary liberal democracies, while scholars of propaganda focus on its manipulative forms and on either historical cases or non-democratic societies. Across these fields, there is minimal conceptual development with regard to manipulative modes of OPC involving deception, incentivization and coercion. As a consequence, manipulative OPC within liberal democracies is a blind spot, rarely recognized let alone researched, and with the result that our understanding and grasp of these activities is profoundly curtailed.

Vlan Bakir et al, 2018

Bakir and colleagues described propaganda as “non-consensual organized persuasive communication”, and it requires deception. Deception includes “lying, distortion, omission or misdirection”. By that definition, the FDA and/or its Commissioner has engaged in deception at different points in this drama, although it may have been at times self-deception, too, by virtue of being uncritical. Consider their stance on the historical evidence in other diseases.

When they rely principally, as they did, on a review that only included evidence only up to July 2013, and therefore no randomized trials, it meant that they weren’t considering the NIH’s randomized trials which did not show benefit from convalescent plasma for influenza. Those trials aren’t obscure – they were published in a Lancet journal. So the best you can say is, they were so biased towards a presumption of support from historical evidence, that they didn’t look properly. That’s by no means an intention to deceive, even though the public been given a false perception of benefit. But it’s not good enough. We need agencies like the FDA to remain critical and thorough.

What I called Hahn’s epic statistical fail definitely falls under Bakir & co’s description of deception by distortion, though: “presenting a statement in a deliberately misleading way to support a viewpoint”. It may have been a mistake, but his later statements suggest he meant to use that inherently misleading 35% statistic from a very small unrepresentative subgroup – knowing how that would be perceived in the community. Whether it’s the FDA Commissioner doing it, or an individual doctor, it’s the opposite of enabling informed consent. That’s the essence of propaganda: It’s been described as “consent-engineering”. Patients and their families deserve better.

~~~~

The cartoons are my own (CC BY-NC-ND license). (More cartoons at Statistically Funny and on Tumblr.)