A Reader’s Guide to Safety & Adverse Event Data From Vaccine Trials

The competition for the “honor” of being the most confusing part of clinical trial results is fierce. I don’t think technical terms are even the worst part of it. It’s apparently simple words and concepts that mean something quite different in everyday language that trip people up the most.

I think information about adverse events – safety and tolerability – “wins” this crown, hands down, especially with vaccines. Here’s a tour of a landscape that may sound obvious, but often isn’t what you might think.

Events vs effects and reactions

This is critical. Just because an unwanted thing happened – an adverse event – doesn’t mean it was an effect or reaction. Almost always, the data you will see are for adverse events – regardless of what words are used to describe them. (Both adverse events and effects are abbreviated as AEs.)

Take headaches, for example. They’re common in everyday life. Say 14% of the people who got a vaccine had a mild headache in the next few days. If 11% of the people in the control group had one too, the vaccine didn’t cause mild headaches in 14% of people – it did so in perhaps a few percent.

There are 2 ways to tell in clinical trials what adverse outcomes were, or might have been, caused by vaccines, and both involve serious investigation:

1. Routinely assess how often particular events happen in the vaccinated and control groups: if it’s definitely higher in the vaccinated group, it’s likely to be an effect; and
2. Investigate each potentially important event, to try to pin down if it was vaccine-related, and if it’s happening at a normal rate.

The monitoring in the first point above could involve routine testing, or testing a subgroup of people. In a vaccine trial, that could be running blood counts, for example. But the main type we hear about comes from vaccinated people monitoring themselves for possible reactions – so-called solicited adverse events.

Solicited vs unsolicited or spontaneously reported adverse events

Solicited adverse events are a list of events/symptoms that participants are specifically asked to record. If that’s not done in a consistent, structured way, the rate of adverse events is likely to be under-estimated. The less thoroughly researchers look, the fewer they’ll find – and the rate of adverse events they report could be artificially low.

A vaccine trial should have a structured method for soliciting adverse events for the week after vaccination, either for everyone, or for a large enough representative subset of people. That’s because vaccines stimulate the immune system, and that stimulation can cause a well-known set of adverse reactions (called reactogenicity). Normal immune reactions are transient, which means they go away on their own, usually within a day or two. People ideally keep a diary for the first week, filling in details according to clear instructions about what to look for and record.

Here’s a tweet from a participant in the phase 3 trial of the Novavax Covid vaccine in the UK, to give you an idea of what a thorough approach to soliciting adverse events is like:

Covid vaccine trial swag pile: three home test kits, thermometer, ruler, info pack, ginger nuts pic.twitter.com/J8lCafrJT6

— Chris Keating (@chriskeating) October 9, 2020

The thermometer and ruler are for accurately measuring 2 of the solicited reactions if they happen: fever/feverishness and how big it was if there’s a red patch around the injection point. That’s because solicited adverse events are graded for how severe they are.

People often dismiss all these reactions as “minor”, but they aren’t necessarily. They can be severe enough to interfere with daily life, and some can become serious – “serious”, in adverse-event-speak, is worse than “severe”.

The most common grading is probably the one from the US FDA, and these terms and grades are widely used (even if they might be defined somewhat differently):

• Mild (or grade 1): This equates to what pretty well everyone would agree is “minor”. It’s a symptom or discomfort that doesn’t interfere with your usual activities, like feeling feverish, or some soreness where you were injected.
• Moderate (or grade 2): This is more uncomfortable, but still generally manageable – you might use a mild pain reliever, for example, but still get on with your day. A mild fever ranks here, or a few bouts of nausea.
• Severe (or grade 3): This can be so uncomfortable, it would be a battle to continue your day as normal – people generally take time off work if they can. You might take narcotic-level pain relief, or vomiting could get bad enough for you to need an IV – but nothing so bad that you would need to go to hospital. A fever of 39º to 40ºC ranks here (102.1º – 104º F).
• Grade 4 is potentially life-threatening: This is an emergency room visit, or hospitalization. A fever over 40ºC ranks here.

Some of these would be clearly objective – temperature, for example. But the lines between these categories are often blurry, too. Instructions for how to grade events aren’t always identical from trial to trial either. All this makes comparison of trial results very tough, and sometimes downright perilous.

The issue of how likely it is that you’ll just have some minor discomfort, or whether you might be pretty much out of commission for a day or so, is a pretty critical issue for many people. Yet it’s the consistently worst-communicated topic I see in health information and media reports.

Typically, you’ll see severity of adverse reactions described as either, “mostly mild” or “mostly mild or moderate”. That tells you precisely nothing: severe adverse events aren’t ever going to outnumber mild to moderate ones in a vaccine that’s proposed for community-wide prevention. The easiest to tolerate vaccines have mostly mild to moderate reactions – and so do the ones that get a name for being pretty rough.

If you really want to understand where a vaccine fits along the tolerability spectrum, you need to know how common moderate and severe reactions were. And you really need this separated for local and systemic symptoms. Mild, local adverse events swamp all other reactions – that’s symptoms directly related to the spot you were injected: pain, redness, soreness, and so on. Systemic reactions are everything else – fever, fatigue, general aches, headaches, etc.

The solicited symptoms can also occur after the diary-keeping period – as well as unexpected adverse events that aren’t on the solicited list, or any safety issues that people are asked to watch out for and notify. Those are grouped into unsolicited adverse events (also called spontaneously reported events) – that may be time-limited, for example, to the month after vaccination.

If something unexpected appears in this group during trials of a vaccine, it could get added to the list of “adverse events of special interest” that drug regulators will monitor closely, both in later trials, and after the vaccine is in use (pharmacovigilance). Grade 4 events, and any others requiring unexpected medical care (MAAEs, or medically attended adverse events), will get further careful attention, too, to try to establish whether or not the event was vaccine-related. There can be a wide range of severity here, too.

Adverse events of special interest (AESIs) and serious adverse events (SAEs)

Along with grade 4 solicited events & any other MAAEs, it’s the AESIs and serious adverse events (SAEs) that are the “safety” end of the spectrum, and there’s no time limit for assessing those. These are events that either pose immediate or potentially ongoing health concerns, either themselves, or because they could be indicators of something serious. Serious adverse events include death, potentially life-threatening situations, or the potential for serious disease or disability. It doesn’t mean people necessarily sustained long-term harm: it could be resolved by treatment.

These events are examined by people involved with the trial, and then at drug regulatory agencies, to try to determine if any are, or could be, vaccine-related, and if so, whether the risk is too high for community use. A part of this process also involves considering how often you would expect that event to occur in a similar group of people in everyday life. People are looking for any signal that indicates even a slight chance of serious harm. And any questions will be folded into the plan to monitor the vaccine after it goes into community use.

The number of people vs the number of events

People who experience adverse events don’t necessarily just experience one. For example, one person could have none, or just a sore arm, while another could have a fever, a headache, and a sore arm. Seeing lists of percentages of AEs doesn’t help you know what your chance of experiencing adverse events at all are. It’s something to keep in mind, because it seems as though people jump to conclusions about this easily – and it’s not always reported well, even in detailed clinical trial publications.

Getting the data into perspective

Even if you are not particularly fond of numbers, this is one of those times when you really need to see some, at least. When people use subjective descriptions like common, rare, and minor, they could be pegging those at very different levels, and you might see it differently if you saw the numbers or objective descriptions. As far as we know, using words without numbers tilts people towards over-estimating the risk of adverse events.

Drug regulators in the US and Europe have some rules for pharmaceutical companies about communicating adverse event information in product leaflets. So at least in those leaflets, certain words will always mean the same thing. It’s an objective standard I try to adhere to when I write about adverse events, and I find it useful to help me put numbers into perspective:

• FDA: Common events are “e.g., all adverse reactions occurring at a rate of 10 percent or greater in the treatment group and at a rate at least twice the placebo rate”.
• European Medicines Agency (EMA): An adverse event is –
  • “very common” if 1 in 10 or more (10%);
  • “common” if 1 in 100 or more (1%) and less than 10%;
  • “uncommon” if 1 in 1,000 or more and less than 1%;
  • “rare” if 1 in 10,000 or more and less than 1 in 1,100; and
  • “very rare” if less than 1 in 10,000 people.

Ideally, reports on vaccine outcomes include clarity on the moderate and severe end of the tolerability spectrum – and at least enough numbers for you to get an idea of how the writer is framing adverse events. From one direction, there will be exaggeration of potential harms – and because of that, there will be a lot of sources minimizing or dismissing adverse events in a way they wouldn’t if it was any other type of drug. Neither helps us prepare us for the experience, and both will be a barrier to informed decision-making for people who will be lucky enough to have a choice of vaccines.

~~~~

You can keep up with my work at my newsletter, Living With Evidence. And I’m active on Mastodon: @hildabast@mastodon.online 

Adverse events can also combined into composite outcomes – see my backgrounder on these in general (whether for benefits or harms) at Statistically Funny.

To read more about adverse events in clinical trials and systematic reviews, check out the chapter in The Cochrane Handbook.

Disclosure: I was a member of the Cochrane Collaboration’s Methods Group on Adverse Effects many years ago, including at the time a paper I “cited” in this post was produced through the group.

The cartoons are my own (CC BY-NC-ND license). (More cartoons at Statistically Funny.)