Maybe It’s A Bad Idea To Give A Bunch Of Kids Double Mastectomies Without Checking Whether It Helps Them

Just sorta spitballin’ here

(This was originally a paid post. I unlocked it 3/17/2023.)

“Top surgery drastically improves quality of life for young transgender people, study finds,” read the CBS headline from September 27. If you’re even faintly familiar with this newsletter, you know what I’m going to say next: 

Nope.

We don’t actually know, from the study in question, whether “top surgery drastically improves quality of life for young transgender people” because the authors didn’t ask. 

The study, published in JAMA Pediatrics by a group of Northwestern University medical researchers, went as follows:

This is a nonrandomized prospective cohort study of patients who underwent top surgery between December 2019 and April 2021 and a matched control group who did not receive surgery. Patients completed outcomes measures preoperatively and 3 months postoperatively. This study took place across 3 institutions in a single, large metropolitan city. Patients aged 13 to 24 years who presented for gender-affirming top surgery were recruited into the treatment arm. Patients in the treatment arm were matched with individuals in the control arm based on age and duration of testosterone therapy.

For obvious reasons you can’t do a true double-blind randomized controlled trial of top surgery, so some of the study weaknesses here aren’t the fault of the researchers. But it’s important to note that apples aren’t being compared to apples, because there are, of course, going to be some differences between kids who do and don’t seek top surgery. And three months is a very short follow-up period for this sort of major surgery (NHS: “Most people find their wounds take around 2 to 3 weeks to heal, but it may be several months before your chest and arm area fully recover.”). Even if the researchers had collected excellent, in-depth data, there would be some serious questions about what we could extrapolate from their findings. (Update, shortly after publication: I added the bolded term just to be a bit clearer. A gold-standard RCT is one in which the participants don’t know which arm of the trial they’re in — as in, some are taking the drug being studied, while others are taking a placebo. With surgery this is really difficult. People will know which arm they’re in, and, among the other methodological issues here, you’d expect those randomly given access to a surgery they want to have a very different psychological response from those randomly denied access to it.)

But they didn’t collect excellent, in-depth data. In fact, their study includes zero validated mental health measures: There is nothing on anxiety, depression, or suicidality. This struck me as strange, especially given that the researchers filled in the study’s “Importance” field as follows: “Transgender and nonbinary (TGNB) adolescents and young adults (AYA) designated female at birth (DFAB) experience chest dysphoria, which is associated with depression and anxiety. Top surgery may be performed to treat chest dysphoria.”

Being a generally suspicious and cynical person — a jerk, in other words — the complete absence of these measures in the study itself made me wonder if maybe the researchers had collected this data at baseline but simply chose not to report it because the trajectories were not what they wanted. I emailed Northwestern and posed the question this way:

I'm a New York based journalist (more information here) and I might be writing about this study Northwestern Medicine just published. It's excellent that researchers are gathering data on these patient populations, but I noticed that there didn't appear to be any data about anxiety, depression, or suicide. Would it be possible to check with the researchers whether they did evaluate the patients for these conditions at any point, and if not, why not? If they did, I'm curious whether it would be possible for them to share that analysis. And is the anonymized raw data shareable as well? 

No response.

So what did they measure? “The primary outcome was the Chest Dysphoria Measure (CDM). Secondary outcomes included the Transgender Congruence Scale (TCS) and Body Image Scale (BIS).”

The CDM was introduced in a 2018 JAMA Pediatrics paper lead-authored by Johanna Olson-Kennedy, an adolescent medicine physician at the Children’s Hospital of Los Angeles who is an enthusiastic and well-known proponent of youth gender medicine. That paper also looked at the outcomes of a cohort of young people who got double mastectomies — between the ages of 14 and 25, in this case (I guess the surgeons didn’t get the message, oft communicated to me on Twitter, that “No one under 16 is getting top surgery.”). Olson-Kennedy and her authors note that there are “no data documenting the effect of chest surgery on minors.” Not little data, but no data. Ah, okay — so it’s probably time to collect some, right? 

Nope. This study also has no validated mental health measures. Instead, its focus is “a novel measure of chest dysphoria,” the Chest Dysphoria Measure, or CDM.

Now, to be fair, this was a retrospective survey study asking people to report on their mental health after getting top surgery, so maybe there just wasn’t baseline mental health data available. But it appears all the kids came through the CHLA gender clinic, so why isn’t this clinic collecting mental health data before referring kids to surgery? I’m not sure, but one potential answer is that Olson-Kennedy simply doesn’t believe in this — as she told me once, she wouldn’t force kids into a mental health exam before giving them insulin, so why would she make that a prerequisite for youth gender medicine?

Anyway, Olson-Kennedy and her team didn’t collect mental health data during the survey they conducted for this study, either: “The 10-minute survey collected demographic information, characteristics of surgery, and chest dysphoria.” Whatever one’s philosophy on “gatekeeping” gender dysphoric youth, couldn’t you get a ton more data with 10 more minutes of questions about anxiety, depression, or suicide, even if you didn’t have a “before” dataset to compare it to? Why not do that? The authors even point out this limitation: “[T]he Chest Dysphoria Scale is not yet validated, and may not represent distress or correlate with validated measures of quality of life, depression, anxiety, or functioning.” (More on that in a moment.)

The CDM still isn’t validated, though you wouldn’t know that from this latest study, which describes it as, well, “the only validated instrument published to date that measures chest dysphoria in [transgender and nonbinary adolescent and young adult] patients.” Do the authors not know that this scale isn’t validated? That’s a bad sign. Weirdly, the authors of an accompanying editorial comment (paywalled) note this limitation — “The Chest Dysphoria Measure, which is the primary outcome in this study, has not yet been validated for use,” they write. But there’s nothing to worry about, because the headline of that comment — about a study that didn’t include any validated measures — is “Top Surgery in Adolescents and Young Adults—Effective and Medically Necessary.” (Listen to experts. Trust the science.)

Here’s the actual CDM, from the Olson-Kennedy study. There are 17 items and each is ranked from 0 (the respondent never feels that way) to 3 (they feel that way all the time), meaning the total score range is 0 to 51: 

One thing you might notice is that a lot of these items simply have to do with having breasts. As in, it’s hard to imagine someone’s CDM scale not going down following top surgery. To take perhaps the most obvious example, “I have to buy/wear certain clothes because of my chest” could certainly apply to bras, and after a double mastectomy there’s no longer any reason to wear a bra, so boom — there’s a reduction of two or three points.

Another thing you might notice is that many of these items could be endorsed by a cisgender adolescent female getting used to having breasts for the first time, which can surely be an uncomfortable and sometimes even traumatizing experience. This is concerning given the potential issues with differential diagnosis here. Would this scale be able to differentiate a kid high in anxiety, depression, or other mental health symptoms who is (understandably) freaking out about growing breasts from a kid who has the sort of deep-rooted, specific gender dysphoria supposedly ameliorated by youth gender medicine? There’s no way to know because it doesn’t appear anyone has asked.

Anyway, let’s get to what the JAMA Pediatrics study that CBS News reported on actually found. Its most impressive findings all have to do with the CDM:

Table 2 outlines IPTW [inverse probability treatment weighting] analysis results for the primary and secondary outcomes of interest. Summary statistics for 3-month outcomes scores in each group are also reported. eFigure 2 in the Supplement illustrates boxplot distribution of propensity scores for each group. eFigure 3 in the Supplement illustrates mean and standard deviation of propensity scores across quintiles for each group. The IPTW model estimated a 25.58 (95% CI, –29.18 to –21.98) point decrease in CDM for the surgery group relative to the control group. The Figure illustrates 3-month CDM as a function of baseline CDM. Three-month CDM scores for all surgery patients were low, regardless of baseline CDM score. The mean (SD) difference between baseline and 3-month scores in the treatment group was –28.12 (8.32). In contrast, among control patients, CDM scores did not appear to change significantly at 3 months. The mean (SD) difference between baseline and 3-month scores in the control group was –0.52 (6.29). Table 3 outlines subgroup analysis of outcomes among patients under 18 years. The IPTW model estimated a 25.48 (95% CI, –32.85 to –18.11) point decrease in CDM for the surgery group relative to the control group.

Don’t worry about all the jargon. And I don’t know anything about the IPTW, but I don’t think you need to: The point is, researchers found that scores on a scale that is tightly centered on having breasts went down three months later, when the respondents no longer had breasts. That’s the most impressive finding here, and validated mental health measures are entirely absent.

Maybe that’s because the authors already published a study linking the CDM scores to anxiety and depression: “Our retrospective review demonstrated associations between chest dysphoria and anxiety and depression; top surgery can improve chest dysphoria, thereby leading to improvements in quality of life.” Sure enough, the citation points to a paper by the same team published in the Journal of Adolescent Health that is one of the only pieces of research — maybe the only one — attempting to link the CDM to validated mental health measures. 

But if you read that paper, you’ll see that there isn’t much of a correlation there: “Chest dysphoria was positively correlated with anxiety (r = .146; p = .002) and depression (r = .207; p < .001). In multivariate linear regression models, chest dysphoria showed a significant, positive association with anxiety and depression, after accounting for gender dysphoria, degree of appearance congruence, and social transition status.” A univariate correlation of r = .146 means that CDM scores accounted for about 2.1% of the variance in anxiety scores, while a univariate correlation of r = .207 means CDM scores account for about 4.3% of the variance in depression scores. I’m not going to dig deeply into this study’s numbers right now, but surely when you account for other, potentially confounding variables, the correlation grows even weaker.

I wanted to make sure I wasn’t missing something here, so I ran these numbers by my buddy Stuart Ritchie, a psychologist and the author of the (truly excellent) book Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth. Ritchie is much better than I am at quantitative stuff (but on the other hand, I’m taller and could probably beat him at basketball). “Yeah,” he said via text. “It’s a weak correlation — so it’s all about how you interpret it. If you’re gonna make it seem like ‘chest dysphoria’ is a massive deal for people’s mental health, it doesn’t seem that way from the correlation. An interesting contributor, maybe! But not a massive deal.”

And we aren’t being told that chest dysphoria is “an interesting contributor” to adolescents’ mental health problems — we’re being told we need to give these kids top surgery because otherwise their chest dysphoria will cause them immense psychological harm. The surgery is “medically necessary.” But according to this data, at least, CDM scores just aren’t all that big a predictor of anxiety or depression. Certainly not big enough that a potential reduction in CDM scores alone should make a doctor confident that a double mastectomy for an adolescent is the right choice. And yet, that CBS headline: “Top surgery drastically improves quality of life for young transgender people, study finds.”

¹

And the headline on the editorial comment accompanying the journal article: “Top Surgery in Adolescents and Young Adults—Effective and Medically Necessary.”

A lot of doctors are performing these operations in the States, and a lot of other doctors and psychologists are not only promoting them, but suggesting that if you don’t agree that they are medically necessary, you could do real and permanent harm to trans kids. Suicide is discussed ubiquitously. You’d think that along the way, some of these doctors would collect good data on exactly what effects these surgeries do and don’t have on kids’ mental health. 

That doesn’t appear to be the case. It’s completely possible I’m missing something, but I asked a listserv of sex researchers I’m on if any of them were aware of American studies from clinical samples that used validated mental health measures and attempted to control for obvious confounds like counseling, puberty blockers, or cross-sex hormones. Other than James Cantor (who follows this stuff very closely) responding to tell me he wasn’t aware of any, I got crickets. (I posed the same question to Twitter and no one could point me to any.)

Olson-Kennedy’s study was published in 2018, and the majority of the subjects in it had had their mastectomies more than a year prior. These treatments are new, as far as adolescents and gender dysphoria are concerned, but it’s not like they started yesterday — we should certainly have more and better data in 2022. It just seems like no one’s really bothering to collect any, perhaps because some of the clinicians involved are true believers for whom data would be beside the point, given that they just know these treatments work. Like, in their hearts.

What’s the worst that could happen?

---

Questions? Comments? Proposals for me to get a bunch of unnecessary surgeries as some sort of weird performance art about lax medical standards? I’m at singalminded@gmail.com or on Twitter at @jessesingal. Image: “Surgeon adjusting glove in operating room - stock photo” via Getty.

1. I was torn about whether to include this, so I’m compromising by relegating it to a footnote. The author of the CBS story is himself trans, and has Instagram posts from not long ago showing off his mastectomy scars. I absolutely think this requires a disclosure: You can’t get a medical procedure, discuss how much it helped you on social media, and then pose as a dispassionate news reporter simply relating the results of a new study on that same procedure weeks later. It’s terrible journalistic ethics. I reached out to CBS’s PR folks about this but never heard back. To be clear, I do not think the Instagram posts are disqualifying. There simply should have been a disclosure early on in the study writeup. I would say the exact same thing about a reporter who had touted an antidepressant she had just started taking on Twitter shortly before writing a news article highlighting a new study supposedly demonstrating its effectiveness.