Grant notes his colleague's criticism of him as a "professional debunker" as though it's a bad thing for our nation's psychology professors to set out to debunk research. But that's really just the scientific method. Scientific claims are not presumed to be true -- instead, we only treat them as true if rigorous, neutral methods demonstrate that the claim provides a better explanation of a phenomenon than the dreaded null hypothesis. Professional debunkers are very important, particularly in a discipline where much research has been shown to be wrong. KEEP DEBUNKING, JESSE!
One of the problems I have with Primeworld is that it seems to imply an incoherent model of the world where there are lots of factors that each have large effects. It seems fundamentally impossible for e.g. 50 different factors to *each* explain 70% of some outcome.
Consider walking speed. Priming research tells us that observing or thinking about old people has an effect. I assume things like being late, being pursued, being in pain, being "in the moment" and taking in the world around you, etc., are also important determinants of walking speed. Given this, Primeworld seems to need to assert that one of two things must be true: either (a) we happened to discover that observing old people belongs to small number of potent stimuli which strongly affect walking speed, or (b) walking speed is an incredibly fragile property, and hundreds of small things have to be *just right* all the time in order for a person to sustain a fast or even normal walking pace. Both possibilities seem very unlikely to me.
First of all I'm glad to call myself a 3%er Thanks for your writing Jesse.
Second, I can't recall how often you're including snippets of your book rather than just mentioning it, but this Primeworld example and they way you're presenting the univariate structure of fad thinking has convinced me to give you yet again more money.
Keep producing your niche work, it is real value to at least some.
Edit: A thing I want to add to this, something that frustrates me. I used to be a big consumer of "Skeptic" podcasts SGU, Skeptoid, Reality Check, others I can't recall off the top of my head. They were constantly applying these styles of critiques to what you would consider typical skeptic subject matter, Homeopathy, Chiropractic (may have theraputic value but the founding assumptions of the practice are wacky), Acupuncture, Creationism vs Evolution, paranormal, and occasionally the social sciences. Beyond just plausibility they would call out the actual flaws in the methodology or conclusions in research, but always to these very easy targets, rarely towards stuff that is having a significant impact currently. To the extent that they would, well just look at the SBM coverage of youth gender medicine, the same criticisms disappear.
That bit of critique just to say, we need more of this kind of reporting, ideally from people who aren't single issue or narrowly focused, but from someone with the time and basic scientific literacy to cover a broad array of subjects. Don't sell yourself short on ability and the level of expertise needed to bring this kind of critique to bear.
Meta-analyses are great, but they seem to work best in contexts where we have pretty standardized measures and operationalizations of constructs.
For a lot of psychology research, the process of making a meta analysis that synthesizes a bunch of different studies ends up becoming a game of telephone as studies are described in more abstract/high-level ways.
“100 19-year olds at university X rated their warmth or cold perceptions of university Y students, and those who had colder perceptions of university Y students were more likely to identify cases of plagiarism and advocate for harsher punishments”
Becomes
“Cold perceptions of an outgroup result in increased attention to wrongdoing and subsequently higher punitive intentions”
Becomes
“We are less fair and more punitive to our groups we dislike”
Becomes
“We want to punish outgroups”
Each individual step is mostly justifiable, but you end up with a claim so far removed from the original context it’s hard to interpret in any useful way
About seven years ago, I questioned the claim made by Grant, Francesca Gino, and David A. Hofmann that “Whereas just 50% of the general population is extroverted, 96% of managers and executives display extroverted personalities.” I took a look at the cited study, found much reason to question the claim, and wrote briefly about it here: https://dianasenechal.wordpress.com/2016/12/29/are-96-percent-of-managers-and-executives-extraverted/. I see no need for such an overconfident claim in the first place; a bit of uncertainty would have been more informative in the end.
Part of the problem lies in the social sciences' "studies have shown" ritual. For social scientists who write for the general public, It isn't considered enough to state an informed opinion (based on life experience, contemplation, etc.); they are expected to cite studies (verified or not, applicable or not) to support their claim. Publishers expect this. Often, if you look closely at the cited or linked studies, they don't quite demonstrate the truth of the claim; moreover, the space taken up by conclusive citations of inconclusive studies could be better devoted to an investigation of the question itself.
Back in 2011 or 2012, when I was pitching an article on solitude to a range of popular magazines (in support of my book), I was told that "our readers expect studies and data." About solitude? I'm sure someone could come up with studies and figures related to solitude, but how much would they really show? I replied that the questions pertaining to solitude could be better illuminated by philosophy and literature than by social science research. But most of the journals weren't having it.
The "research has shown" habit and expectation reflects a problem within the popularized social sciences (and their overreach into the humanities): a tendency to present hypotheses as certainties/discoveries and to skip essential steps of definition and questioning. The best social science writing draws on the humanities, at least by recognizing how little we know and how tenuous our certainties can be. I appreciate your skeptical analyses, Jesse, both here and in your book.
While reading about the experiments, I kept asking myself: "Is 'multiculturalism' really incompatible with 'colorblindness' here?" Some students were asked to treat everyone the same (i.e. be colorblind); other students were asked to appreciate differences (i.e. be multicultural). These aren't mutually exclusive requests. I can appreciate the differences between my 6 foot friend who is good at basketball and my awkward friend who is good at chess while treating them pretty much the same.
The experiments are so trivial that it doesn't seem to matter much, but I think that the research also is flawed by a lack of clarity about the differing social philosophies that they are trying to study. I noticed that too in the Coleman Hughes / TED controversy. I listened to the debate that TED forced Coleman to do with Jamelle Bouie. It was really hard to understand exactly why Coleman's thesis is so objectionable to Bouie. Obviously, they disagree on affirmative action, but Bouie didn't want to focus on AA, and the rest of his concerns were so vague and slippery.
I really appreciate that you call people like Adam Grant and Jonathan Katz out on their BS and show how dishonest they are. They are symptomatic of a broken, self-congratulatory media culture that considers itself “important” and “correct” when it’s just lazy and biased.
I don't know the details of this exchange between Coleman Hughes and Adam Grant. However, I do know that Adam Grant should not be trusted. See "Contra Grant On Exaggerated Differences" (https://slatestarcodex.com/2017/08/07/contra-grant-on-exaggerated-differences/) by Scott Alexander. The context is the J. Damore memo.
I can only imagine how petty and tedious these types of responses might feel at times, yet the work is SO IMPORTANT and I’m thankful that we have you to re-establish some level of accountability in these fields.
Thank you and please keep up the fantastic work. The real, actual impact will be orders of magnitude greater than any kudos or recognition you receive (not to mention the negative blowback that comes with the territory).
> > I think an unintended consequence of the replication crisis is that critics often approach a body of research like prosecutors seeking a conviction. That makes it easy to discount arguments based on the weakest evidence.
If you described theses studies to me without telling me who did they, I would have assumed they were some silly studies an undergrad did for class credit. The all involve incredibly contrived setups, have very ambiguous interpretations and generally tiny effect sizes. It bums we out that this is the kind of crap we get from supposedly highly trained scientists. It's not like we understand everything about psychology etc. so there's nothing left to do but fiddle at the edges. There a big questions left! There's tons we don't understand.
I have generally appreciated coleman hughes perspective, but lately I have been finding him to be less reliable. Probably first indication was his endorsement of rfk Jr. I liked his Ted talk and I think the controversy around him was stupid. Have yet to read this article, but I imagine overall agree with jesse. However, kind of tangential but just read this article very thoroughly debunking coleman's take on the Derek Chauvin and George Floyd. Pretty damning for coleman and rather disappointing. Anyone else read it? https://open.substack.com/pub/radleybalko/p/the-retconning-of-george-floyd?utm_source=share&utm_medium=android&r=1dvvyx
This is an excellent article. I am not a scientist and took just one statistics course, so I always appreciate the way Jesse can write about scientific studies in ways that are comprehensible to a layman like me.
Grant notes his colleague's criticism of him as a "professional debunker" as though it's a bad thing for our nation's psychology professors to set out to debunk research. But that's really just the scientific method. Scientific claims are not presumed to be true -- instead, we only treat them as true if rigorous, neutral methods demonstrate that the claim provides a better explanation of a phenomenon than the dreaded null hypothesis. Professional debunkers are very important, particularly in a discipline where much research has been shown to be wrong. KEEP DEBUNKING, JESSE!
One of the problems I have with Primeworld is that it seems to imply an incoherent model of the world where there are lots of factors that each have large effects. It seems fundamentally impossible for e.g. 50 different factors to *each* explain 70% of some outcome.
Consider walking speed. Priming research tells us that observing or thinking about old people has an effect. I assume things like being late, being pursued, being in pain, being "in the moment" and taking in the world around you, etc., are also important determinants of walking speed. Given this, Primeworld seems to need to assert that one of two things must be true: either (a) we happened to discover that observing old people belongs to small number of potent stimuli which strongly affect walking speed, or (b) walking speed is an incredibly fragile property, and hundreds of small things have to be *just right* all the time in order for a person to sustain a fast or even normal walking pace. Both possibilities seem very unlikely to me.
First of all I'm glad to call myself a 3%er Thanks for your writing Jesse.
Second, I can't recall how often you're including snippets of your book rather than just mentioning it, but this Primeworld example and they way you're presenting the univariate structure of fad thinking has convinced me to give you yet again more money.
Keep producing your niche work, it is real value to at least some.
Edit: A thing I want to add to this, something that frustrates me. I used to be a big consumer of "Skeptic" podcasts SGU, Skeptoid, Reality Check, others I can't recall off the top of my head. They were constantly applying these styles of critiques to what you would consider typical skeptic subject matter, Homeopathy, Chiropractic (may have theraputic value but the founding assumptions of the practice are wacky), Acupuncture, Creationism vs Evolution, paranormal, and occasionally the social sciences. Beyond just plausibility they would call out the actual flaws in the methodology or conclusions in research, but always to these very easy targets, rarely towards stuff that is having a significant impact currently. To the extent that they would, well just look at the SBM coverage of youth gender medicine, the same criticisms disappear.
That bit of critique just to say, we need more of this kind of reporting, ideally from people who aren't single issue or narrowly focused, but from someone with the time and basic scientific literacy to cover a broad array of subjects. Don't sell yourself short on ability and the level of expertise needed to bring this kind of critique to bear.
Meta-analyses are great, but they seem to work best in contexts where we have pretty standardized measures and operationalizations of constructs.
For a lot of psychology research, the process of making a meta analysis that synthesizes a bunch of different studies ends up becoming a game of telephone as studies are described in more abstract/high-level ways.
“100 19-year olds at university X rated their warmth or cold perceptions of university Y students, and those who had colder perceptions of university Y students were more likely to identify cases of plagiarism and advocate for harsher punishments”
Becomes
“Cold perceptions of an outgroup result in increased attention to wrongdoing and subsequently higher punitive intentions”
Becomes
“We are less fair and more punitive to our groups we dislike”
Becomes
“We want to punish outgroups”
Each individual step is mostly justifiable, but you end up with a claim so far removed from the original context it’s hard to interpret in any useful way
About seven years ago, I questioned the claim made by Grant, Francesca Gino, and David A. Hofmann that “Whereas just 50% of the general population is extroverted, 96% of managers and executives display extroverted personalities.” I took a look at the cited study, found much reason to question the claim, and wrote briefly about it here: https://dianasenechal.wordpress.com/2016/12/29/are-96-percent-of-managers-and-executives-extraverted/. I see no need for such an overconfident claim in the first place; a bit of uncertainty would have been more informative in the end.
Part of the problem lies in the social sciences' "studies have shown" ritual. For social scientists who write for the general public, It isn't considered enough to state an informed opinion (based on life experience, contemplation, etc.); they are expected to cite studies (verified or not, applicable or not) to support their claim. Publishers expect this. Often, if you look closely at the cited or linked studies, they don't quite demonstrate the truth of the claim; moreover, the space taken up by conclusive citations of inconclusive studies could be better devoted to an investigation of the question itself.
Back in 2011 or 2012, when I was pitching an article on solitude to a range of popular magazines (in support of my book), I was told that "our readers expect studies and data." About solitude? I'm sure someone could come up with studies and figures related to solitude, but how much would they really show? I replied that the questions pertaining to solitude could be better illuminated by philosophy and literature than by social science research. But most of the journals weren't having it.
The "research has shown" habit and expectation reflects a problem within the popularized social sciences (and their overreach into the humanities): a tendency to present hypotheses as certainties/discoveries and to skip essential steps of definition and questioning. The best social science writing draws on the humanities, at least by recognizing how little we know and how tenuous our certainties can be. I appreciate your skeptical analyses, Jesse, both here and in your book.
Joining the 3 percenter club. I enjoyed this one.
While reading about the experiments, I kept asking myself: "Is 'multiculturalism' really incompatible with 'colorblindness' here?" Some students were asked to treat everyone the same (i.e. be colorblind); other students were asked to appreciate differences (i.e. be multicultural). These aren't mutually exclusive requests. I can appreciate the differences between my 6 foot friend who is good at basketball and my awkward friend who is good at chess while treating them pretty much the same.
The experiments are so trivial that it doesn't seem to matter much, but I think that the research also is flawed by a lack of clarity about the differing social philosophies that they are trying to study. I noticed that too in the Coleman Hughes / TED controversy. I listened to the debate that TED forced Coleman to do with Jamelle Bouie. It was really hard to understand exactly why Coleman's thesis is so objectionable to Bouie. Obviously, they disagree on affirmative action, but Bouie didn't want to focus on AA, and the rest of his concerns were so vague and slippery.
I really appreciate that you call people like Adam Grant and Jonathan Katz out on their BS and show how dishonest they are. They are symptomatic of a broken, self-congratulatory media culture that considers itself “important” and “correct” when it’s just lazy and biased.
I don't know the details of this exchange between Coleman Hughes and Adam Grant. However, I do know that Adam Grant should not be trusted. See "Contra Grant On Exaggerated Differences" (https://slatestarcodex.com/2017/08/07/contra-grant-on-exaggerated-differences/) by Scott Alexander. The context is the J. Damore memo.
Fuck yeah Jesse, you fuckin’ fuck!!
I can only imagine how petty and tedious these types of responses might feel at times, yet the work is SO IMPORTANT and I’m thankful that we have you to re-establish some level of accountability in these fields.
Thank you and please keep up the fantastic work. The real, actual impact will be orders of magnitude greater than any kudos or recognition you receive (not to mention the negative blowback that comes with the territory).
Seriously, thank you.
Ted Talk hosts are so poorly vetted that I've stopped wasting anymore time watching them.
> > I think an unintended consequence of the replication crisis is that critics often approach a body of research like prosecutors seeking a conviction. That makes it easy to discount arguments based on the weakest evidence.
Mr. Grant: the null hypothesis is Good, Actually.
aye caramba
I'm sure it was accidental, but I liked that your pluralized Adam Grant as Adams Grant, like attorneys general or runs batted in.
If you described theses studies to me without telling me who did they, I would have assumed they were some silly studies an undergrad did for class credit. The all involve incredibly contrived setups, have very ambiguous interpretations and generally tiny effect sizes. It bums we out that this is the kind of crap we get from supposedly highly trained scientists. It's not like we understand everything about psychology etc. so there's nothing left to do but fiddle at the edges. There a big questions left! There's tons we don't understand.
Academia is just so disappointing.
we need a 3%er line of merch lol
I have generally appreciated coleman hughes perspective, but lately I have been finding him to be less reliable. Probably first indication was his endorsement of rfk Jr. I liked his Ted talk and I think the controversy around him was stupid. Have yet to read this article, but I imagine overall agree with jesse. However, kind of tangential but just read this article very thoroughly debunking coleman's take on the Derek Chauvin and George Floyd. Pretty damning for coleman and rather disappointing. Anyone else read it? https://open.substack.com/pub/radleybalko/p/the-retconning-of-george-floyd?utm_source=share&utm_medium=android&r=1dvvyx
This is an excellent article. I am not a scientist and took just one statistics course, so I always appreciate the way Jesse can write about scientific studies in ways that are comprehensible to a layman like me.