What's particularly frustrating is that in a lot of ways hackish research is MORE of a threat than deliberately false research! False research is usually discovered and un-published; it's harmful but more temporal. hackish research lives forever because it's not TECHNICALLY wrong and can really damage discussion of public policy issues for a generation or more. There's serious serious serious ethical questions surrounding research done using garbage methods and/or garbage data that almost always ends up being shoved off into the corner because a cadre of people want to make laziness and conformation bias some kind of shady conspiracy to undermine their own pet beliefs. We as a political science and public policy community can't have a desperately-needed discussion about how massively damaging p-hacking, garbage data, lazy analysis, and overconfident conclusions are to the field when every opportunity to have those discussions becomes a story about fabrication instead.
As a physicist, I'm always curious about how radically different research practices are in the social sciences, and for good reason. Obviously the kind of highly quantitative, repeated, high-sample-size projects we do in physics are simply not practical in the social sciences. But many times it seems to be simply a matter of resources. There are many questions in the social sciences where there are hundreds of individual, uncoordinated studies that each have low SNR and many confounding factors. But if you were able to devote all the resources across all those studies into a single, coordinated, large longitudinal study, you could get something much more transformative. But academic funding just isn't set up for that. You would need every professor and grad student at hundreds of institutions to forego leading their own projects to participate in a huge research network.
But the science benefit would be huge.
Am I wrong in thinking of that as a way forward for complex social science? Is it a problem of the structure of how social science operates in that way, or are there more fundamental issues where scaling up the work would not actually address the problems of signal size, confounding variables, and replicability?
I can try to answer some of this. My father does meta-analytic social science research and I did some of my own as an undergrad at a different university, so I think I can provider a basic explanation of the issue. The main problem is that all of those studies are trying to answer their own questions and have reason to squabble about what exactly is in their datasets. One reason census data and the ACS are so valuable is that they try to anticipate that and generate a huge amount of data, but those obviously don't exist on the state and local level and the privacy concerns that the data generate become even more severe on those smaller scales (imagine trying to do research on the scale of the Atlanta Metro Area with census level datasets on a large proportion of households--you'd have PII everywhere and a confined enough area that it can really be used for identification). So your enormous database would have some serious privacy concerns to start with, but that doesn't include the large costs of maintenance and how to handle the researchers who want to explore outside the confines of the database.
This caught my eye, because it was an Enos-supported study (https://scholar.harvard.edu/files/renos/files/carneyenos.pdf) that really triggered me into becoming skeptical of academic studies showing high racial resentment among conservatives, a big step in my personal depolarization process. So it is funny (?) to me that Bruenet seems to have pegged Enos as a categorical enemy. Regardless, I really appreciate your level-headed and rational approach to looking at these questions!
I thought that was SUCH an important finding. Such a basic question to ask, that apparently no one had...
Anyway, yeah, separate question from whether or not everything is above-board here, but Ryan Enos is the last guy you'd accuse of being some sort of biased, raging SJW.
Interesting situation developing here, thanks for the quick work Jesse! Enjoy the trip.
I read the Substack article from Karlstack when you tweeted the link this morning. A couple things made me immediately suspicious of Brunet's post.
First, linking racial threat theory to critical race theory stood out immediately to me. I read a few racial threat articles in undergrad, it can absolutely offer testable and falsifiable hypotheses — CRT on the other hand is mostly explicitly not testable (not necessarily a knock, a lot of legal theory shares this characteristic). As far as I'm aware the academics using both theories may or may not share sympathies and their methods differ substantively.
Second, Christopher Brunet announced his Substack was going on permanent hiatus so he can take a job at Daily Caller in the last post before the Enos post. So he gets this new gig and then they decline to publish a major scoop about an academic studying race and politics at Harvard? I mean I get that journalists work in a bunch of different mediums now-adays, but it's suspicious he revived the Substack just for this. Did Daily Caller decline to publish this? Was it below *their* editorial standards?
All this feels like professional jealousy weaponizing right wing cancellation to me. Enos might have done sloppy work, I haven't read the papers. Excluding a bunch of precincts is not ipso facto suspicious to me, maybe they didn't fit the characteristics of his hypothesis and weren't adequate for controls. PoliSci is building an airplane at 30,000 feet, sometimes parts of your available data just don't work and introduce a ton of statistical noise, he would not be the first to cut big parts of a set for totally benign reasons. At the same time, maybe Enos was genuinely sloppy, which Jesse's sources seem to suggest. The part about one years olds being included in a dataset seems incredibly sloppy, if true. From Enos' recent activity it seems like he might be big enough to admit his mistake if confronted with evidence. This has happened in social science lately where academics are owning their own bad work. I hope he takes that route if its necessary.
Jesse, Is Mr. Brunet the same person who writes this substack below? The picture looks the same and has this person said something about going to the Daily Caller?
I want nuance to catch on with the mainstream so that I have to be rabidly polarized just to feel edgy again. In the meantime, thank you for this tonic. It is so important for us, in this our Ponzi era, to get a clear view of the ways journalism works (and doesn't) and how studies and statistics can be twisted in the service of storytelling.
I say, "DOWN WITH FACE VALUE!" (Until 'carefully judging both sides of the topic' reaches the tipping point, at which point I'm going hard left to keep my iconoclasm, sorry not sorry.)
What's particularly frustrating is that in a lot of ways hackish research is MORE of a threat than deliberately false research! False research is usually discovered and un-published; it's harmful but more temporal. hackish research lives forever because it's not TECHNICALLY wrong and can really damage discussion of public policy issues for a generation or more. There's serious serious serious ethical questions surrounding research done using garbage methods and/or garbage data that almost always ends up being shoved off into the corner because a cadre of people want to make laziness and conformation bias some kind of shady conspiracy to undermine their own pet beliefs. We as a political science and public policy community can't have a desperately-needed discussion about how massively damaging p-hacking, garbage data, lazy analysis, and overconfident conclusions are to the field when every opportunity to have those discussions becomes a story about fabrication instead.
As a physicist, I'm always curious about how radically different research practices are in the social sciences, and for good reason. Obviously the kind of highly quantitative, repeated, high-sample-size projects we do in physics are simply not practical in the social sciences. But many times it seems to be simply a matter of resources. There are many questions in the social sciences where there are hundreds of individual, uncoordinated studies that each have low SNR and many confounding factors. But if you were able to devote all the resources across all those studies into a single, coordinated, large longitudinal study, you could get something much more transformative. But academic funding just isn't set up for that. You would need every professor and grad student at hundreds of institutions to forego leading their own projects to participate in a huge research network.
But the science benefit would be huge.
Am I wrong in thinking of that as a way forward for complex social science? Is it a problem of the structure of how social science operates in that way, or are there more fundamental issues where scaling up the work would not actually address the problems of signal size, confounding variables, and replicability?
I can try to answer some of this. My father does meta-analytic social science research and I did some of my own as an undergrad at a different university, so I think I can provider a basic explanation of the issue. The main problem is that all of those studies are trying to answer their own questions and have reason to squabble about what exactly is in their datasets. One reason census data and the ACS are so valuable is that they try to anticipate that and generate a huge amount of data, but those obviously don't exist on the state and local level and the privacy concerns that the data generate become even more severe on those smaller scales (imagine trying to do research on the scale of the Atlanta Metro Area with census level datasets on a large proportion of households--you'd have PII everywhere and a confined enough area that it can really be used for identification). So your enormous database would have some serious privacy concerns to start with, but that doesn't include the large costs of maintenance and how to handle the researchers who want to explore outside the confines of the database.
This caught my eye, because it was an Enos-supported study (https://scholar.harvard.edu/files/renos/files/carneyenos.pdf) that really triggered me into becoming skeptical of academic studies showing high racial resentment among conservatives, a big step in my personal depolarization process. So it is funny (?) to me that Bruenet seems to have pegged Enos as a categorical enemy. Regardless, I really appreciate your level-headed and rational approach to looking at these questions!
I thought that was SUCH an important finding. Such a basic question to ask, that apparently no one had...
Anyway, yeah, separate question from whether or not everything is above-board here, but Ryan Enos is the last guy you'd accuse of being some sort of biased, raging SJW.
I've been to Cuba. AMA.
Really nice and nuanced article (also long, but what can you do?). Enjoy Cuba.
Interesting situation developing here, thanks for the quick work Jesse! Enjoy the trip.
I read the Substack article from Karlstack when you tweeted the link this morning. A couple things made me immediately suspicious of Brunet's post.
First, linking racial threat theory to critical race theory stood out immediately to me. I read a few racial threat articles in undergrad, it can absolutely offer testable and falsifiable hypotheses — CRT on the other hand is mostly explicitly not testable (not necessarily a knock, a lot of legal theory shares this characteristic). As far as I'm aware the academics using both theories may or may not share sympathies and their methods differ substantively.
Second, Christopher Brunet announced his Substack was going on permanent hiatus so he can take a job at Daily Caller in the last post before the Enos post. So he gets this new gig and then they decline to publish a major scoop about an academic studying race and politics at Harvard? I mean I get that journalists work in a bunch of different mediums now-adays, but it's suspicious he revived the Substack just for this. Did Daily Caller decline to publish this? Was it below *their* editorial standards?
All this feels like professional jealousy weaponizing right wing cancellation to me. Enos might have done sloppy work, I haven't read the papers. Excluding a bunch of precincts is not ipso facto suspicious to me, maybe they didn't fit the characteristics of his hypothesis and weren't adequate for controls. PoliSci is building an airplane at 30,000 feet, sometimes parts of your available data just don't work and introduce a ton of statistical noise, he would not be the first to cut big parts of a set for totally benign reasons. At the same time, maybe Enos was genuinely sloppy, which Jesse's sources seem to suggest. The part about one years olds being included in a dataset seems incredibly sloppy, if true. From Enos' recent activity it seems like he might be big enough to admit his mistake if confronted with evidence. This has happened in social science lately where academics are owning their own bad work. I hope he takes that route if its necessary.
Jesse, Is Mr. Brunet the same person who writes this substack below? The picture looks the same and has this person said something about going to the Daily Caller?
https://karlstack.substack.com/p/woke-mathematicians-are-putting-their
I want nuance to catch on with the mainstream so that I have to be rabidly polarized just to feel edgy again. In the meantime, thank you for this tonic. It is so important for us, in this our Ponzi era, to get a clear view of the ways journalism works (and doesn't) and how studies and statistics can be twisted in the service of storytelling.
I say, "DOWN WITH FACE VALUE!" (Until 'carefully judging both sides of the topic' reaches the tipping point, at which point I'm going hard left to keep my iconoclasm, sorry not sorry.)