SMTM's Chemical Hunger III: Scooped!

In which someone else does a better job so I don't have to do mine

If you read me often, you know I’m sort of a qual who sometimes pretends to be quant, or (as the kids say) a wordcel who can sort of fake shape-rotator if it’s necessary. Like a guitarist filling in on bass, I can often do a good job at this; also similar to a guitarist filling in on bass, it means that an actual quant can usually do a much, much better job than me at that particular element of the job.

So while I think I did a pretty good job in parts I and II of pointing out various places where SMTM wasn’t as credible as they could have been, it was predominantly in places where a high-verbal-score person could step in and find places where they had made verbal errors. So I was looking at different standards of rigor applied to different things, places where they claimed a source said something it didn’t, and so on.

I was dreading having to actually break down and do the quant part, probably doing it poorly/wrong, and then getting rightfully slapped down on some of the points I was making. But at some point, this angel of a person’s post sprung into being and does all the quant stuff a lot better than I was realistically going to do it.

At this point, you should absolutely go read that post. I mean it. I mean it even if it means you don’t read mine.

That’s good in several ways; first, you are legitimately better served by Natália’s work than mine. Second, I’m for better or worse a bit contrarian; it’s right in the title. It’s probably helpful for the reader to know I’m not just being a jerk here, or at least that if I am I’m not the only jerk. Another person looking into this and finding a lot of the same kind of “all information is interpreted in a way that supports my points” problems I saw means you don’t have to trust me as much as an individual source. This is good.

It’s also a mixed bag for me, because even if I was going to do it poorly I had a lot of work dumped into writing part III, and now I need to do something different. So what you will get is, for better or worse, a less-quant-more-qual version of what I was thinking about writing, first looking at Natália’s piece and then talking a bit about what all this means.

---

Natália’s piece examines several key points of A Chemical Hunger, and mostly finds the same thing I found: A carelessness-at-best treatment of the data in ways that, if believed, end up reinforcing their point. I’m breaking out a few of them below:

Altitude

SMTM shows that people living at altitude are thinner, and tries to claim that thinner air and similar aren’t to blame. Instead, they say, you’d expect less environmental lithium build-up at higher altitudes; since there’s less lithium and other stuff doesn’t make sense, that’s why people are less fat.

She counters with this:

Using publicly-available data from the USGS and the Open Elevation API, I found that across 1,027 domestic-supply wells (all wells whose coordinates were available), the correlation between altitude and log(lithium concentration) is 0.46. I also checked the correlation between altitude and topsoil log(lithium concentration) in the United States, with data I found here, and, again, it was positive (0.3). So lithium exposure is probably higher, rather than lower, in high-altitude areas in the United States (which, as a reminder, have lower obesity rates).

Since most of the food anyone eats is trucked in AND they take into account topsoil lithium, this is (if correct) pretty damning. Since this was a major component of the argument, you don’t want to see this data if you are making SMTM’s argument.

Texas lithium/obesity link

Natalia:

In Part VII: Lithium (a), the SMTM authors say:

In Texas, a survey of mean lithium levels in public wells across 226 counties (Texas has 254 in total) found lithium levels ranging from 2.8 to 219.0 ng/mL. Now Texas is not one of the most obese states — but it tends to be more obese along its border with Lousiana [sic], which is also where the highest levels of lithium were reported.

However, if you look at the source, their map of lithium levels across Texas counties actually says the opposite – that counties along the border with Louisiana have lower lithium levels than other counties (though it’s a bit confusing because on the map darker areas represent lower levels):

Someone else has already pointed this out in the comments of that post, several months ago, as have I, on a 05/08/2022 Twitter thread tagging the SMTM authors, but the post has not been fixed, and the authors have not acknowledged or addressed the error in any way.

When you calculate the correlation between log(water lithium levels) and log(obesity %) in Texas, you find that it is -0.13.

I don’t want to quote her whole article, but I could. In almost any situation she looks at, she finds similar stuff. Is the threshold of where we’d see lithium start to affect people higher than SMTM says? Probably! Did they cherry-pick studies showing high levels of lithium in food to further bridge that gap? Probably!

Please, please go read that article. You might see something different than I do, and you shouldn’t take my recounting of what a source says at face value. But overall, it’s pretty damning; my qual analysis and Natalia’s higher-effort quant analysis seem to agree that this is a house of cards that falls apart no matter where you poke it from.

---

If I wrote a deep-dive article 10% as long as A Chemical Hunger, it would take at minimum many hours, at maybe several days. It’s not easy. It’s a lot of work, and by the time you are done you end up afraid to hit publish. Here’s an accurate representation of what my internal dialogue might look like at that point:

What if I’m clearly wrong in a way I don’t see? What if people immediately notice that, point it out, and I look like a liar? What if I’m wrong in a way that people don’t notice, and it hurts people? I criticize a lot of people - I can’t avoid criticism. And I need to write about this. But what if publishing this ruins literally everything?

This isn’t a reaction that comes out of the rational part of my mind. I’ve occasionally been caught in some kind of error and the feeling is terrible; it’s sort of a ball of embarrassment, shame, and guilt that boils just a few inches below my clavicle notch. The loss of credibility feels huge. It’s a bad time, is what I’m saying.

Given that, I have mixed feelings about really digging into SMTM here. SMTM, if you do read this, I don’t think you are a bad group of humans. But at the same time, if there aren’t any real incentives for not being careless, everyone is going to do it. Make no mistake: this theory has done wonders for SMTM’s brand. It’s too sexy (as judged by a certain kind of nerd) to ever be fully negated by criticisms, no matter how many valid counter-articles pop up.

So while I actually dislike making them feel bad here (and I’m sure they do, even in the case they think both me and Natalia are wrong) I don’t think there’s any way around it. And even if I wanted to point out that it’s just inaccurate and leave out the likely possibilities of why, “This is factually wrong” only addresses the error, not the “biased, uncareful, or dishonest” potential causes.

If there aren’t incentives against misrepresentation, people are more likely to misrepresent whether it’s on purpose or by accident. If there’s no incentive to read closely, less people will do the work. And very certainly if there’s not a clear incentive structure that leads you to expect good things/rewards if you are critical of your own work and bad things/punishments if you aren’t, people aren’t going to do it. It’s too unnatural a reflex; if people don’t get “paid” to do it, they won’t.

I’m not at all sure I hit on the correct balance that often. It’s easier when I’m going after a big entity (the FDA, every living human, etc.); it just carries less weight when it’s not poking at an individual (or a few people) and their work. But I tend to think the failure mode - people feeling comfortable being less than careful - is worse. I’m not immune to that failure mode, either - I need the incentives as much as the next guy.

On the more factual side, I don’t think we ended up knowing a lot more about obesity than when we started. SMTM’s deductive arguments (those that eliminate non-lithium causes) are pretty bad. Their direct arguments for lithium don’t seem much better. Lithium isn’t absolutely disproven, but close examination has left me with the impression that it’s still just as unlikely as most things; there’s just too many problems with SMTM’s arguments about too many pieces of information to trust any individual piece without close examination.

That doesn’t mean I think SMTM is bad or shouldn’t be read or any number of things beyond what I specifically said in the last paragraph. But on this subject, at least, we need to demand and wait for better.

---