Study-Smoothing Will Never Not Drive Me Crazy
I know I’m being a nerd but this is important!
New York Times health and wellness reporter Simar Bajaj has a new article about “The Return of Blaming and Shaming in Public Health,” as the headline puts it. It’s pegged to our esteemed Secretary of Health and Human Services, RFK Jr., and discusses the relationship between stigma and public-health concerns like obesity and smoking.
I don’t think there’s any good reason to be skeptical of Bajaj’s general thesis that stigma, in the sense of shaming or ridiculing or othering, is ineffective as a public health strategy. And there were things I liked about this article, which is a bit more nuanced than the headline suggests.
But I also think Bajaj makes a common mistake you see all the time in science writing, and it’s one that drives me a bit nuts. In arguing against approaches steeped in stigma, Bajaj writes the following:
Researchers found that antismoking messages focused on industry deception — like how Big Tobacco targets teens or manipulates the public — were some of the most effective at reducing cigarette consumption.
As soon as I read this sentence, I knew what would happen when I clicked on the link: I would be brought to a study that did not, in fact, meaningfully support the claim that we have such rich data about which antismoking messages work better than others.
How could I be so confident? Two reasons: First, we, as a species, don’t really know how to change people’s beliefs and behavior. Second, our hypotheses on this front are notoriously difficult to test scientifically.
Smoking itself is a fine example. The best way to know which anti-smoking messages work would be to take a huge group of otherwise similar people, divvy them up into groups that are otherwise as similar as possible, and randomly assign the different groups to be exposed to different messages about smoking. If, after the experiment ends, differences have emerged in the rates of smoking/quitting/other important behavioral variables between the groups, you might be able to infer that those differences were caused by the messaging.
But this is quite difficult to do in practice. It is difficult to find the resources to run a study this large and rigorous. And even if differences are observed, that’s not the end of the story: There could be all sorts of other explanations. The best-laid plans of researchers often go awry.
Of course the vast majority of studies aren’t nearly this ambitious. While we do have some randomized controlled trials of anti-smoking interventions, they are small and sometimes difficult to interpret. In this one, for example, 357 smokers in San Diego who had cigarettes delivered to their home by the researchers (not a bad study to participate in!) were randomly assigned to regular packs, blank packs, or to packs containing nasty imagery designed to deter smoking. The researchers found that “graphic warning labels decreased positive perceptions of cigarettes associated with branded cigarette packs but without clearly increasing health concerns. They also increased quitting cognitions but did not affect either cigarette cessation or consumption levels.”
One the one hand, a useful study, both because of the key result but also because it reveals something important and frequently ignored: People often profess beliefs that don’t translate into their actual behaviors. On the other hand — and this is not a knock on the researchers — even this fairly basic result is difficult to interpret! For example, “demand effects” (or characteristics) are a well-known phenomenon in experimental psychology — people often seem to catch on to the fact that they are being studied and to provide responses oriented (perhaps unconsciously) toward satisfying researchers. Let’s say I measure your beliefs about smoking, start sending you packs containing graphic warnings, and then measure your beliefs about smoking again. You report that you feel less positively about cigarettes. Was that because the imagery actually affected your perception of cigarettes, or because the experiment basically contains a giant blinding neon sign stating WE ARE TRYING TO MAKE YOU FEEL WORSE ABOUT CIGARETTES?
Also: There are huge cultural, political, and health-behavior differences between different parts of this great nation of ours. Would the results of a San Diego study, whatever they are, generalize to Peoria? Hell, would they generalize to San Francisco? It’s very difficult to know.
All of which brings me back to my skepticism of that New York Times sentence. If it’s difficult to establish reliably effective ways to change beliefs or behavior, and if in even the best cases (like an RCT) it’s difficult to interpret attempts to measure such interventions, it seems unlikely that anyone has actually established something as specific as “antismoking messages focused on industry deception. . . [are] some of the most effective at reducing cigarette consumption.”
