It Will Likely Be Years Before We Really Know The Extent of COVID-19's Badness

Meta-note: This article is about covid-19 and some difficulties I think exist in quantifying exactly how bad it is, while acknowledging that regardless of those difficulties it’s still clearly very bad.

With that said, it’s still another covid-19 article; I can certainly understand if you are burned out on these by now. If you can’t take another, I give you full permission to skip this one; I’ll talk about something different in my next post, I’m sure.

Counting things can be hard. Counting lots and lots of things that happen at a distance is very hard. Counting lots and lots of things that happen at a distance in an imperfect information environment can be damn near impossible, even before we start talking about bias and one’s preferred worldview muddying the waters. Even something simple-sounding that we do all the time and should already be good at - like counting overall deaths in the US - is fraught with delays and inaccuracies that take real work to get around.

Covid-19 is not an exception to this at all, and it’s a shame. As with anything that drives massive, society-shaking restrictions we would hope to get really rich, beautiful data on exactly how bad things are. Everyone from the masks-are-an-authoritarian-plot team all the way over to the endless-quarantine-at-any-cost-is-fine group would at least claim to want better data, even if they might not like what that data ended up indicating.

But we can’t have a terribly precise estimate of just how bad covid is or isn’t yet, no matter how badly the previous paragraph’s weakmen might want it. Oddly, this isn’t necessarily because some secret cabal of scientists is off sinfully obfuscating and confounding our data; it’s most likely just the reality of things that this data is fundamentally hard to gather and interpret.

As an initial example of why, think about how COVID deaths are counted. You probably already know that counts are vulnerable to political pressures; Russia only counts people confirmed with an autopsy to have been sick with covid, intentionally making their numbers look smaller:

A wide discrepancy between Russian cities’ accurate counts of deaths from all causes and the official national covid-19 mortality figures has been evident since the beginning of the pandemic.2 The low official figure was generated by only reporting deaths in which novel coronavirus infection was identified on autopsy.

From the beginning the official numbers drew widespread scorn from Russian doctors, who on social media painted a far grimmer picture, often mourning deaths in their own ranks from lack of personal protective equipment. A privately maintained list of Russian medical staff who lost their lives fighting the pandemic now stands at over 1000 names.

In contrast, the US counts a huge amount more:

For more details on the snip above go here, but the short story is that the first bullet-point applies to people with covid-like symptoms who have been around someone with covid and the second bullet is folks who have a positive covid antibody test. The third bullet is the broadest, pulling into the count every individual whose death certificate listed they had covid at their time of death. This is a “Physician’s best guess” standard, in the absence of better information:

Most States require that the death certificate be completed and filed within a specified time period. Physicians are expected to use medical training, knowledge of medicine, available medical history, symptoms, diagnostic tests, and autopsy results, if available, to determine the cause of death.

While the CDC attributing deaths to covid-guesses probably added a lot of false positives, it’s very likely better than the alternative of leaving out cases that looked probable to the doctors but weren’t proven otherwise. Antibody tests weren’t approved until April and weren’t widespread until later, so that’s more deaths than we want to give up. And look at this mess:

They found that, for the 249,167 total excess deaths in these 1,021 counties, there were 183,686 deaths directly assigned to COVID on death certificates, and another 65,481 excess deaths not officially assigned to COVID. This means 26 percent of all excess deaths were not directly attributed to COVID—or, in other words, that excess deaths were 36 percent higher than the number directly attributed to COVID.

“Counties with high levels of COVID-19 mortality also had exceptionally high levels of mortality in 2020 from other causes of death,” says study senior author Samuel H. Preston, professor of sociology at the University of Pennsylvania. “This result suggests that the epidemic is responsible for many more deaths than are attributed to COVID-19 alone.”

You might recognize BU’s rationale as something that supports both undercounting and overcounting enthusiasts in their established views; if you are prone to the former, you believe we’ve missed a lot of stuff. If you are prone to the latter, you see a huge opportunity for over-excited doctors to have marked down death certificates with their death-cause du jour.

So what do you do if your best standard for counting covid deaths is still subject to pretty large errors and the continued disagreements those errors cause? The strongest argument is to look at excess mortality rates, essentially the amount of deaths we’ve had this year above and beyond what we’d expect judging by previous years. This isn’t perfect, but it does a much better job of capturing total covid deaths; the only weakness being that it doesn’t differentiate between deaths caused by covid and deaths caused by our reaction to covid, but it’s probably good enough for government work (actual government work, in this case). The CDC has this number at 328,900 or 446,406 since February, depending on how strict they are with the standard they use.

I think this is probably where most normal people stop caring. If a death is a death is a death to you, these numbers are all that matters, and they are either right or pretty close to right. 300,000-ish people is either a lot to you or it isn’t. But some people care about the relative value, as measured in years of life left; a 20 year old who otherwise would have lived to be 70 loses 50 years of life, while a 69 year old loses a single year.

The usual yardstick used to quantify this relative loss is the QALY, or quality-adjusted life year. This standard looks not only to the amount of years left to a person but also to how good those years are expected to be. A QALY is modified by adjusting from a value of one for a year lived in perfect health to fractions of one for less healthy years, eventually bottoming out at 0 when a year would be valued by the person as no more desirable than death. QALY’s do some of what we want here - they take into account the age of the person, and to some extent take into account their health. We are getting a little closer!

Although the QALY tells us a lot more information about our loss than death-is-a-death did, there’s still a significant problem with using the QALY for figuring out our loss. QALY’s are a little awkward here because they are intended for estimating the life-years-gained benefit of keeping a person from getting sick at all, or from giving them a treatment with a pretty well understood positive effect.

But what a QALY doesn’t do incredibly well is quantify how much life is lost if covid is preferentially killing especially ill people. Imagine we have a group of heart patients, half of whom have about five years year to live and half of whom have a year left. If some catastrophe claimed 10 of them, and they were evenly selected from both groups, we’d lose thirty years. If it were all from the especially sick contingent, though, we’d only lose 10; still tragic, but less so. My dad had diabetes and a heart condition for the last five years of his life; to the QALY, he would have looked identical for all five years. Not only are QALY models not being fed the kind of information that would let us know if covid is mostly reaping the sick-within-the-sick, data this granular doesn’t seem to be gathered within covid data-sets in the first place (Note: It’s possible I’ve missed some good models, QALY or otherwise, that do capture this kind of data better than I think they do. If so, let me know).

This brings me finally to the point of all this: we have a good idea that covid is bad, and terribly so. But there’s a limit to our ability to gather data on something like this - we don’t seem to have have a great idea of exactly how bad it is, even before we consider the supposed long-term health effects it causes even when survived. What we need is gold-standard data, and that will be a long time coming - not because we are bad at gathering it, but because we can only have it in hindsight.

When an event targets those with vulnerable health, we typically see death rates rise, but in the ensuing weeks and months afterwards see a corresponding dip in death rates. An example of this is a heat wave, which typically kills the very weakest people and leaves fewer very weak people to die when they otherwise would have. This dip in the death rates is usually referred to as a harvesting effect, a part of the overall concept of mortality displacement.

If covid was to magically disappear tomorrow, we would almost certainly see a large fall in death rates - today’s high death rates demand lower death rates in the future. But the more we would find that covid is killing the especially weak, the larger we’d expect that dip to be. This doesn’t mean that covid isn’t a gigantic catastrophe in any case - it clearly is. But if a loss of 350,000 life years is better than a loss of several million, it’s what you hope for.

Since covid is still going strong, it’s entirely possible last year’s harvesting effect will be covered up by this years excess deaths. If - God help us, if - covid gets substantially beaten this year, we would then likely be able to start tracking the fall in death rate, if any were to materialize. It’s the few years directly after covid stops that should give us our best idea of how many life years were lost; we can compare, say, the 2020-2021 death rate with 2022-2023 and hopefully find the average rate is lower than we’d expect - we would still have lost the people, but hopefully less total life-years. But until then I’m not sure we will really know anything with precision (again, if I’m wrong, please let me know).

As stated at the head of this post, I’m sorry to drag you down covid-road with me - I’m sure you get enough of it out in the world. Be safe, get vaccinated as soon as possible, and keep your chin up - there’s a light at the end of the tunnel we can see from where we’re at. We will get there.