39 Comments

Grant notes his colleague's criticism of him as a "professional debunker" as though it's a bad thing for our nation's psychology professors to set out to debunk research. But that's really just the scientific method. Scientific claims are not presumed to be true -- instead, we only treat them as true if rigorous, neutral methods demonstrate that the claim provides a better explanation of a phenomenon than the dreaded null hypothesis. Professional debunkers are very important, particularly in a discipline where much research has been shown to be wrong. KEEP DEBUNKING, JESSE!

Expand full comment
founding
Jan 31·edited Jan 31Liked by Jesse Singal

One of the problems I have with Primeworld is that it seems to imply an incoherent model of the world where there are lots of factors that each have large effects. It seems fundamentally impossible for e.g. 50 different factors to *each* explain 70% of some outcome.

Consider walking speed. Priming research tells us that observing or thinking about old people has an effect. I assume things like being late, being pursued, being in pain, being "in the moment" and taking in the world around you, etc., are also important determinants of walking speed. Given this, Primeworld seems to need to assert that one of two things must be true: either (a) we happened to discover that observing old people belongs to small number of potent stimuli which strongly affect walking speed, or (b) walking speed is an incredibly fragile property, and hundreds of small things have to be *just right* all the time in order for a person to sustain a fast or even normal walking pace. Both possibilities seem very unlikely to me.

Expand full comment

That is an excellent way to put the issue!

Expand full comment

Andrew Gelman wrote a paper on this exact observation, which he called "the piranha problem": https://stat.columbia.edu/~gelman/research/unpublished/piranhas.pdf

> A fundamental tenet of social psychology and behavioral economics, at least how it is presented in the news media, and taught and practiced in many business schools, is that small “nudges,” often the sorts of things that we might not think would affect us at all, can have big effects on behavior.

> The model of the world underlying these claims is not just the “butterfly effect” that small changes can have big effects; rather, it’s that small changes can have big and predictable effects, a sort of “button-pushing” model of social science, the idea that if you do A, you can expect to see B.

> In response to this attitude, we present the piranha argument, which states that there can be some large and predictable effects on behavior, but not a lot, because, if there were, then these different effects would interfere with each other, a “hall of mirrors” of interactions (Cronbach, 1975) that would make it hard to detect any consistent effects of anything in observational data.

> This article collects several mathematical results regarding the distributions of correlations or coefficients, with the aim of fostering further work on statistical models for environments with a multiplicity of effects. What is novel in this paper is not the theorems themselves but rather viewing them in the context of trying to make sense of clusters of research studies that claim to have found large effects.

> There are many ways to capture the dependence among random variables, and thus we should expect there to be a correspondingly large collection of piranha theorems. We formalize and prove piranha theorems for correlation, regression, and mutual information in Section 4. These theorems illustrate the general phenomena at work in any setting with multiple causal or explanatory variables, with implications for the replication crisis in social science.

As Ian states, it is mathematically impossible for many independent factors to simultaneously have large, predictable effects.

Expand full comment
Jan 31·edited Jan 31

First of all I'm glad to call myself a 3%er Thanks for your writing Jesse.

Second, I can't recall how often you're including snippets of your book rather than just mentioning it, but this Primeworld example and they way you're presenting the univariate structure of fad thinking has convinced me to give you yet again more money.

Keep producing your niche work, it is real value to at least some.

Edit: A thing I want to add to this, something that frustrates me. I used to be a big consumer of "Skeptic" podcasts SGU, Skeptoid, Reality Check, others I can't recall off the top of my head. They were constantly applying these styles of critiques to what you would consider typical skeptic subject matter, Homeopathy, Chiropractic (may have theraputic value but the founding assumptions of the practice are wacky), Acupuncture, Creationism vs Evolution, paranormal, and occasionally the social sciences. Beyond just plausibility they would call out the actual flaws in the methodology or conclusions in research, but always to these very easy targets, rarely towards stuff that is having a significant impact currently. To the extent that they would, well just look at the SBM coverage of youth gender medicine, the same criticisms disappear.

That bit of critique just to say, we need more of this kind of reporting, ideally from people who aren't single issue or narrowly focused, but from someone with the time and basic scientific literacy to cover a broad array of subjects. Don't sell yourself short on ability and the level of expertise needed to bring this kind of critique to bear.

Expand full comment

Meta-analyses are great, but they seem to work best in contexts where we have pretty standardized measures and operationalizations of constructs.

For a lot of psychology research, the process of making a meta analysis that synthesizes a bunch of different studies ends up becoming a game of telephone as studies are described in more abstract/high-level ways.

“100 19-year olds at university X rated their warmth or cold perceptions of university Y students, and those who had colder perceptions of university Y students were more likely to identify cases of plagiarism and advocate for harsher punishments”

Becomes

“Cold perceptions of an outgroup result in increased attention to wrongdoing and subsequently higher punitive intentions”

Becomes

“We are less fair and more punitive to our groups we dislike”

Becomes

“We want to punish outgroups”

Each individual step is mostly justifiable, but you end up with a claim so far removed from the original context it’s hard to interpret in any useful way

Expand full comment

> you end up with a claim so far removed from the original context it’s hard to interpret in any useful way

You might even say such a claim is... Approaching_Insignificance

Expand full comment

About seven years ago, I questioned the claim made by Grant, Francesca Gino, and David A. Hofmann that “Whereas just 50% of the general population is extroverted, 96% of managers and executives display extroverted personalities.” I took a look at the cited study, found much reason to question the claim, and wrote briefly about it here: https://dianasenechal.wordpress.com/2016/12/29/are-96-percent-of-managers-and-executives-extraverted/. I see no need for such an overconfident claim in the first place; a bit of uncertainty would have been more informative in the end.

Part of the problem lies in the social sciences' "studies have shown" ritual. For social scientists who write for the general public, It isn't considered enough to state an informed opinion (based on life experience, contemplation, etc.); they are expected to cite studies (verified or not, applicable or not) to support their claim. Publishers expect this. Often, if you look closely at the cited or linked studies, they don't quite demonstrate the truth of the claim; moreover, the space taken up by conclusive citations of inconclusive studies could be better devoted to an investigation of the question itself.

Back in 2011 or 2012, when I was pitching an article on solitude to a range of popular magazines (in support of my book), I was told that "our readers expect studies and data." About solitude? I'm sure someone could come up with studies and figures related to solitude, but how much would they really show? I replied that the questions pertaining to solitude could be better illuminated by philosophy and literature than by social science research. But most of the journals weren't having it.

The "research has shown" habit and expectation reflects a problem within the popularized social sciences (and their overreach into the humanities): a tendency to present hypotheses as certainties/discoveries and to skip essential steps of definition and questioning. The best social science writing draws on the humanities, at least by recognizing how little we know and how tenuous our certainties can be. I appreciate your skeptical analyses, Jesse, both here and in your book.

Expand full comment

Joining the 3 percenter club. I enjoyed this one.

While reading about the experiments, I kept asking myself: "Is 'multiculturalism' really incompatible with 'colorblindness' here?" Some students were asked to treat everyone the same (i.e. be colorblind); other students were asked to appreciate differences (i.e. be multicultural). These aren't mutually exclusive requests. I can appreciate the differences between my 6 foot friend who is good at basketball and my awkward friend who is good at chess while treating them pretty much the same.

The experiments are so trivial that it doesn't seem to matter much, but I think that the research also is flawed by a lack of clarity about the differing social philosophies that they are trying to study. I noticed that too in the Coleman Hughes / TED controversy. I listened to the debate that TED forced Coleman to do with Jamelle Bouie. It was really hard to understand exactly why Coleman's thesis is so objectionable to Bouie. Obviously, they disagree on affirmative action, but Bouie didn't want to focus on AA, and the rest of his concerns were so vague and slippery.

Expand full comment

Exactly the thought I had. It’s entirely possible, even laudable, to simultaneously believe that “We’re all human, with similar fundamental needs, desires and aspirations, who sometimes can be noble and too often fall short, but should have an opportunity to thrive and to find acceptance and safety.” and “Each person and group have unique experiences, talents and concerns that they can bring to the community to make it more capable and resilient, and to enhance our common society.”

Expand full comment

I had this same thought! So frustrating. I guess people obsessed with binaries see them everywhere.

Expand full comment

Honestly, I would say that colorblindness and multiculturalism aren't compatible. I doubt you do treat your awkward nerd friend and 6 foot friend jock friend the same. You may feel you are perfectly nice and friendly to both. But that is different than treating them "the same". There are times you would prefer the 6 footer (like if you needed something high reached) to the chess nerd and vice versa (like if you needed advice on a chess move). You would treat them different because you would intuitively believe they are, because in this case ...they are! Now how much does that same inutuition affect how we view people who look different than us? I think it will essentialy always be greater than zero. The problem, imho, is defining what "the same" means and how to measure it.

I know this sounds like picking nits, but when Abram Kendi is demanding that all outcomes be racially neutral it matters! I dont think that people will ever treat each other "the same" enough to get exactly proportional racial outcomes. This doesn't have to be nefarious or even necessarily lead to a negative outcome, btw. But I think it's true. And we have to have an answer other than "Dude Im totally sure colorblindness gets to where you want to go." Because it won't. Do you think it will? I think arguing about it is because it's much easier to argue about studies and methodologies than about outcomes.

The GOP in their argumentation on this issue in the 90's (and still do, although the rhetoric is slowly rotting with the intellectual death of the party) specifically avoided discussing outcomes. Because they knew that was a fools game. So they said "equality of opportunity" instead of "equality of outcome". Kendi and his ilk are demanding outcomes. Arguing about methodologies is a bit of a dodge. An understandable dodge. And a worthwhile one from a scientifically critical viewpoint (and we need science critics imho!). But a dodge nonetheless.

Expand full comment

Sorry for the long and philosophical response, but it is an interesting question to me ...

I agree that "treating everyone the same" is ambiguous. That's part of the challenge with those studies and also with the broader discussion about colorblindness versus multiculturalism. Obviously, no two human beings are the same, and we interact with people differently based on their differing characteristics -- age, sex, attractiveness, intelligence, cultural background, wealth, race, sexual orientation, etc. The colloquial sense of "treating people the same" that I used in my comment refers to the idea that we ought to see people as morally equivalent despite their differences. I think that its the idea in the Declaration of Independence "All men [we would say people now] are created equal." It's not that everyone is equally tall or smart but that they deserve the same rights and that no one should be considered inherently superior to another based on their natural differences.

Based on your comment, you think that this is an impossible goal, especially when it comes to race, and won't lead to proportional racial outcomes. But it is a goal that many Americans aspire to.

And I don't think that it is necessarily incompatible with multiculturalism. It depends on how multiculturalism is understood. We can appreciate cultural diversity in the same way that we appreciate the diversity of natural human talents. B/c there are so many Mexican and Korean immigrants in my neighborhood, I am happily exposed to a variety of food, art, literature, and music that I wouldn't have experienced if all my neighbors were descended from people who sailed over in the Mayflower. While valuing the differing cultural traditions of Mexican and Korean immigrants, I can still try to treat them the same as citizens and moral agents.

Multiculturalism starts to conflict with colorblindness if you believe that different cultures are so different that there is no way that they could agree on a common set of moral rules as the basis of a shared society (thus, Koreans might need their own courts and laws b/c their sense of justice is so incompatible with their neighbors). Or if you think that social goods unrelated to cultural differences need to be allocated to people based on their cultural background. For example, if you have two air traffic control jobs open, and you decide that one has to go to a Mexican and another has to go to a Korean regardless of which applicants are best qualified to maintain air safety.

I doubt that most Americans automatically interpret multiculturalism in this second, more separatist, sense. I definitely didn't when I was growing up in CA and being taught to celebrate the cultural diversity of the state. And -- to end in the same place after a long story -- I don't think that the studies that Jesse talked about define multiculturalism clearly enough to be able to say that it is incompatible with colorblindness.

Expand full comment

I think there's a lot of levels to all this, and a lot of "True Scotsman" problems with terms, and I don't want to drag us down a rabbit hole. Also people will take disagreements personally in a "You calling me racist way?!" (Your individual behaivor is not what I'm discussing here.)

Anyway I agree with the main thrust of the articles that these studies are probably not useful for much. The societal issue which all these kinds of arguments glide over is what I was thinking of when I posted, which is the mismatch between 1) the demands of an aggrieved race vs 2) What colorblindness or multiculturalism can, in reality, provide toward that goal.

I dont know that "2", whatever route we choose, can ever produce an outcome to meet the current demands of "1". I dont think colorblindness can, because I'm 45 and that's been the accepted plan, more or less, for my entire life until recently. Thats what we were taught to do in school. Thats what the community message at the end of GI Joe said, even right wing radio hosts told people to do it. Everybody has been on board the "treat people equally" train for a long time... and despite all that, people still seem ticked about race, man. Maybe going "even further colorblind" than our as yet imperfectly practiced last 30-40 years of colorblindness will work? I'm game but you'd have to convince me that that's a behaivor the human species is capable of and how exactly your going to implement it. (I'd say your self described behaivor isn't colorblind at all. You notice ethnicity after all! You maybe be extremely equally polite to all involved. But you ain't colorblind in a way that will produce the mathematically completely neutral outcomes that are currently demanded.)

Anyway, sorry I've rabbit holed us enough, but all I mean is that arguing about how good "colorblindess" vs "multiculturalism", however you define those terms is only a tiny part of the battle. The real battle is to get everyone to agree the outcome is justified. And I don't see colorblindness doing that, unfortunately. At least not without a huge other yet-to-be-named ingredient.

Expand full comment

I still think it’s a false dichotomy. It’s not like you only get to choose your stance once. I don’t see any hypocrisy or contradiction in demanding hard colour-blindness on say, sentence reform while floridly advocating for multiculturalism at the school talent show. Or even for observing that college admissions issues are controversial and complex precisely because they demand a balancing of both concerns: are the benefits of a diverse campus and society worth some lack parity in the process and if so, how much? You can come down on different sides of that issue, but pretending it’s easy or obvious is disingenuous.

Expand full comment

Well, we might actually agree. The specific problem with this specific part of the conversation, as I see it, is this:

Humans are cruddy at colorblindness!

Now I think there's a mushy middle here, which you may be invoking. Instead of "multiculturalism" call it "open minded melting pot-ism". Classic American assimilation, meaning all Americans slowly over individual lifetimes and generations converging on a single identity. It'll be ongoing as long as new immigrants keep coming, so it's never "complete". But everyones on a similiar "track" as it were. But with people taking a little extra effort to make sure everyone is respected and included. I think that's doable. I think it makes functional colorblindness a little harder, but that's hopefully offset by the "respect other cultures" part.

But that's not what the people advocating for multiculturalism seem to want. I think 30 years ago when the word emerged it might have meant that. But now it seems like people are advocating for hard, big lines between groups. i.e. "You must honor and respect Group X and their ways. But you are not of them and they are not of you and you adopting some of their ways is, in itself, not acceptable or at least its extremely sensitive and probably not a great idea."

It seems like that's what they want, don't you think?

And that....well.... I just don't think we can turn colorblindness on and off like that. The more group differences are emphasized the more people will treat others differently. This doesn't necessarily treating them "bad". It doesn't even mean you purposely think to yourself "Ah this person is not like me and I shall treat them as an other." But I think that'll seep in. Don't you? If you don't I hope you're right. Our species will be much better off if I'm wrong and we can get past this. But I'm pessimistic.

Expand full comment

I really appreciate that you call people like Adam Grant and Jonathan Katz out on their BS and show how dishonest they are. They are symptomatic of a broken, self-congratulatory media culture that considers itself “important” and “correct” when it’s just lazy and biased.

Expand full comment

I don't know the details of this exchange between Coleman Hughes and Adam Grant. However, I do know that Adam Grant should not be trusted. See "Contra Grant On Exaggerated Differences" (https://slatestarcodex.com/2017/08/07/contra-grant-on-exaggerated-differences/) by Scott Alexander. The context is the J. Damore memo.

Expand full comment
founding

Fuck yeah Jesse, you fuckin’ fuck!!

I can only imagine how petty and tedious these types of responses might feel at times, yet the work is SO IMPORTANT and I’m thankful that we have you to re-establish some level of accountability in these fields.

Thank you and please keep up the fantastic work. The real, actual impact will be orders of magnitude greater than any kudos or recognition you receive (not to mention the negative blowback that comes with the territory).

Seriously, thank you.

Expand full comment

I pay for this solely because almost no one actually reads the thing they are critiquing.

Expand full comment

Ted Talk hosts are so poorly vetted that I've stopped wasting anymore time watching them.

Expand full comment

> > I think an unintended consequence of the replication crisis is that critics often approach a body of research like prosecutors seeking a conviction. That makes it easy to discount arguments based on the weakest evidence.

Mr. Grant: the null hypothesis is Good, Actually.

aye caramba

Expand full comment

I'm sure it was accidental, but I liked that your pluralized Adam Grant as Adams Grant, like attorneys general or runs batted in.

Expand full comment

Or pussies cat . . .

Expand full comment

If you described theses studies to me without telling me who did they, I would have assumed they were some silly studies an undergrad did for class credit. The all involve incredibly contrived setups, have very ambiguous interpretations and generally tiny effect sizes. It bums we out that this is the kind of crap we get from supposedly highly trained scientists. It's not like we understand everything about psychology etc. so there's nothing left to do but fiddle at the edges. There a big questions left! There's tons we don't understand.

Academia is just so disappointing.

Expand full comment

Academic publishing is such a con. Publishers charge eye watering amounts of money of money, but authors and peer reviewers (and editors?) do the vast majority of the work for free. And in a publish or perish world of academia there is a glut of not very good research which is being spewed out by not-very picky journals.

My husband is a researcher and recently peer reviewed an article. It was really flawed - the methodology didn't make sense, the results didn't add up, and the conclusions they drew were completely unfounded. So he gave the authors a lot of feedback (in a lot nicer and more constructive language) but did not recommend it for publication. Guess what - the journal published it anyway.

Expand full comment

we need a 3%er line of merch lol

Expand full comment

I have generally appreciated coleman hughes perspective, but lately I have been finding him to be less reliable. Probably first indication was his endorsement of rfk Jr. I liked his Ted talk and I think the controversy around him was stupid. Have yet to read this article, but I imagine overall agree with jesse. However, kind of tangential but just read this article very thoroughly debunking coleman's take on the Derek Chauvin and George Floyd. Pretty damning for coleman and rather disappointing. Anyone else read it? https://open.substack.com/pub/radleybalko/p/the-retconning-of-george-floyd?utm_source=share&utm_medium=android&r=1dvvyx

Expand full comment

Wow, thanks for linking that piece. I hadn’t seen it. I was fairly shocked when I first heard Glenn Loury and John McWhorter talk about the documentary exonerating Chauvin, but then figured, “I guess it’s yet another example of how mob rule is ruining us.” I saw Coleman Hughes’s piece in TFP and it seemed like another nail in the coffin (never watched the documentary). I’m really happy I saw your comment and read the article… Disappointing to feel like I can’t trust anyone to do their homework (except maybe Jesse?)… I feel like I need to do my own deep dive on every single current event. Sadly can’t take anyone at face value, even the people who claim to have done a deep dive.

Expand full comment

I think it just when something aligns with one's bias they will examine it less critically often times. I think good writers are better at acknowledging and seeing there bias. Coleman has a critical eye and I think is a genuinely interesting thinker, but he has lost my trust to cover certain topics. But I think this could be a good growth moment for him though. It is another good remainder for me to always look for multiple view pionts on a given subject.

Expand full comment

This is an excellent article. I am not a scientist and took just one statistics course, so I always appreciate the way Jesse can write about scientific studies in ways that are comprehensible to a layman like me.

Expand full comment