If The New England Journal Of Medicine Doesn’t Correct This Error, You Cannot Trust Anything It Publishes

Some things are annoyingly simple

I’m going to be a bit blunter, brusquer, and briefer than usual here.

I recently wrote about how hard it is to know who to trust in our increasingly fractured epistemic landscape, and I argued that there are some simple rules that can help.

While I didn’t use the word humility in that post, that’s what my recommendations really came down to: asking whether the person whose trustworthiness is in question exhibits humility. Along those same lines, it’s obviously important, trustworthiness-wise, to note whether or not a person or institution which publishes false information swiftly and transparently corrects it.

In January The New England Journal of Medicine published an article titled “The Future of Gender-Affirming Care — A Law and Policy Perspective on the Cass Review,” authored by the law professors Daniel G. Aaron of the University of Utah and Craig Konnoth of the University of Virginia. (Aaron is also a physician.) “The Cass Review on gender-affirming care, which has been used to justify bans in U.S. states, transgresses medical law, policy, and practice, which puts it at odds with mainstream U.S. expert guidelines,” claims the abstract. (This article ran in the NEJM’s Perspective section, which houses essays and more opinion-y fare rather than original research articles. Everything in it is still peer-reviewed. More on this section in a bit.)

What follows is a regurgitation of a large number of falsehoods, distortions, and misunderstandings that have been used to denigrate the Cass Review ever since it was published. A good number of those falsehoods, distortions, and misunderstandings seem to have come from the Yale “Integrity Report” white paper lead-authored by Meredithe McNamara, which is Aaron and Konnoth’s first citation. That paper, as I showed in a trio of posts that almost entirely sapped my will to live, was chock-full of. . . well, falsehoods, distortions, and misunderstandings.

Back when I wrote about that white paper, I was disturbed not only by the fact that Yale Law School published and promoted such propaganda-posing-as-science on its website, but also by what happened next. Despite the fact that critics (myself included) pointed out numerous errors and distortions in the paper, some of them of a straightforward factual nature, none of the authors — Meredithe McNamara, Kellan Baker, Kara Connelly, Aron Janssen, Johanna Olson-Kennedy, Ken C. Pang, Ayden Scheim, Jack Turban, and Anne Alstott — took their names off the paper or pushed for a correction. They failed to clear the lowest bar imaginable for a public intellectual, in other words. I try to give people the benefit of the doubt, but if a researcher is willing to affix their name to a claim that they know is false — and I specifically emailed Alstott, Olson-Kennedy, and Jannsen about this — then what won’t they do to further whatever causes they think are righteous? If someone is openly willing to lie, it should call into question everything that comes out of their mouth.

The same thing applies to publications, from the lowliest tabloids to the most celebrated medical journals. And I think The New England Journal of Medicine is about to put itself into that camp.

I am going to slice off only one very tiny, very wrong piece of Aaron and Konnoth’s article, but there are so many galling problems with the whole thing that it’s taking every ounce of my self-control not to turn this into an extended diatribe. Luckily, the excellent writer Void if Removed published a thorough takedown on his Substack, and I highly recommend that you read it (and follow his work in general).

For now, let’s focus on this and only this:

The [Cass] Review calls for evidentiary standards for GAC [gender-affirming care] that are not applied elsewhere in pediatric medicine. Embracing RCTs [randomized-controlled trials] as the standard, it finds only 2 of 51 puberty-blocker and 1 of 53 hormone studies to be high-quality.

Just about everything in these two sentences is wrong.

First, a seemingly minor but actually pretty important issue: The phrasing it finds suggests the Cass Review itself, as in Dr. Hilary Cass and those who helped her put together the report, were the ones who evaluated the evidence base for blockers and hormones. That isn’t so: Rather, they commissioned independent reviews from University of York University that were subsequently peer-reviewed and published in the British Medical Journal’s Archives of Disease in Childhood. So in effect, Cass and her colleagues were reporting what an independent group of scholars found, which, in full context, makes it harder to accuse Cass and co. of bias. (It’s a factually challenged allegation, anyway, given that the Cass Review and the University of York University scholars effectively found what every other systematic European review of the evidence has found.)

Anyway, as it turns out, Aaron and Konnoth couldn’t even get the basic numbers right. The puberty blockers review examined 50 studies, not 51, and it found one study to be high-quality, not two. This is a strange error to make given that this information is right in the abstract (bolding mine):

Results 11 cohort, 8 cross-sectional and 31 pre-post studies were included (n=50). One cross-sectional study was high quality, 25 studies were moderate quality (including 5 cohort studies) and 24 were low quality.

Much worse, Aaron and Konnoth claim that these papers “embrac[e] RCTs as the standard,” which is insanely, totally wrong! In fact, both reviews used a grading rubric specifically designed for evaluating the strength of non-randomized studies. It’s called the Newcastle-Ottawa Scale (NOS), and the underlying issues get slightly wonky, but again, all you have to do is read the abstracts of both studies to see this.

Here’s the relevant part of the abstract from the blockers study:

Methods A systematic review and narrative synthesis. Database searches (Medline, Embase, CINAHL, PsycINFO, Web of Science) were performed in April 2022, with results assessed independently by two reviewers. An adapted version of the Newcastle-Ottawa Scale for cohort studies was used to appraise study quality. Only moderate-quality and high-quality studies were synthesised. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting guidelines were used.

And here it is from the hormones study:

Methods Systematic review and narrative synthesis. Database searches (MEDLINE, Embase, CINAHL, PsycINFO, Web of Science) were performed in April 2022, with results assessed independently by two reviewers. An adapted version of the Newcastle-Ottawa Scale for Cohort Studies was used to assess study quality. Moderate- and high-quality studies were synthesised.

A cohort study is a study where you simply track a group’s outcomes over time. It is, definitionally, not an RCT, because an RCT entails randomly assigning participants in the study to one of at least two treatment or control groups.

So the authors of the University of York systematic reviews broadcast, about as explicitly as could be broadcast — right in the abstracts of their publications — that they were not evaluating the strength of the studies on blockers and hormones on the basis of whether they were RCTs. Rather, they were holding them to a much lower standard, using an instrument that isn’t designed to evaluate RCTs, as is plainly clear from reading it. (That’s why none of the few studies rated “high-quality” by the University of York reviewers were RCTs.)

This is a dismal, dismal paragraph. As Void if Removed notes, “In this short quote, the authors get the total number of studies wrong, the number of high quality studies wrong, the standard of evidence wrong, and the requirement for randomized-controlled trials wrong.” He is correct.

That this passage — not to mention the countless other problems ViR highlights — could be published in the NEJM in the first place suggests a serious problem with the editorial processes there. By the time this article was posted online, the fights over the Cass Review had been going on for many months. In fact, “Cass and her minions demanded RCTs, which are impossible in this area of medicine!” was a false rumor about the Cass Review that some activists began spreading to discredit it almost as soon as it came out, and it was — for those of us following this stuff closely — debunked not long after. And then The New England Journal of (freakin’) Medicine breathed new life into it nine months after the Review was published! What is the point of having top-tier journals if they will publish false claims? If pseudonymous Substackers can more competently and accurately describe the contents of academic studies?

Simply put, and for the sake of this exercise setting the whole rest of the article: If The New England Journal of Medicine will not correct this error — I don’t mean “update” the article, I don’t mean add an “erratum,” I mean correct it — then I honestly don’t understand why anyone who isn’t a rube would ever trust anything published in the NEJM again. It’s the most obvious call imaginable. This might sound harsh, but why is it harsh? Complete this sentence: “The New England Journal of Medicine won’t correct an objectively false claim it published, but I still trust it in general because _________.” I’m sorry, but there is no version of this sentence that would not mark you as a rube.

I reached out to the NEJM’s media address, Aaron and Konnoth, Perspective section editor Debra Malina, and to the journal’s editor in chief, Eric Rubin, to alert them that I’d be writing this and to ask if they wanted to comment. I explained I planned to publish it Monday afternoon but could wait if they wanted to discuss or push back. I got back from a note from Aaron that read “We are working on a response to this question and others, and we should be able to get you an answer by the end of the week.”