How Not To Cover A Social Science Controversy (Updated)

A useful inadvertent cautionary tale

If you find this sort of thing useful or interesting, you should consider becoming a paid subscriber to my newsletter:

I’m only able to do this for a living because of my premium subscribers.

---

This is going to be a weird and slightly unsatisfying-to-read post, but I’ll try to explain everything concisely. 

Someone has been circulating an anonymously authored document accusing Ryan Enos, a well-regarded political scientist at Harvard, of serious academic misconduct. The document contains many claims, but they generally center on the idea that work he has published on the subject of so-called racial threat, or (to oversimplify) the ways white people react to the presence of black people, is based on data with very suspicious features. In some cases, argue the authors of the document, discrepancies between Enos’ work and publicly available datasets of registered voters are so suspicious that data fraud seems quite likely (presumably there is more than one author given that they call themselves “Social Scientists for Research Integrity,” but who knows?).

Last week I pitched a story about this to The Chronicle of Higher Education, they accepted the pitch, and I sent the anonymous document to a few transparency-minded social scientists I know to quietly get their view on how seriously I should take the accusations in the document. I also spoke to one of them on the phone a bit.

All three of them told me versions of the same thing: As far as they could tell, the document’s accusations were overblown, and in some cases its authors treated as highly suspicious things that may well have innocent explanations. “It’s possible that a more thorough investigation of these very facts would uncover a true red flag, and even a smoking gun,” said Uri Simonsohn, a behavioral scientist and leading debunker of questionable behavioral science, in an email. “As it stands, I do not consider this a red flag. I would not, for example, write about any piece of evidence in this report in [his website] Data Colada as evidence of something being uniquely wrong with a paper.” (Data Colada is a great read and I reference some of its content in my book about half-baked behavioral science.)

Andrew Gelman, arguably the leading don of online statistical debunking, pointed out in an email that some of the criticisms in the document apply to a lot of research, and said the claims of fraud seemed premature to him. “These appear to be legit criticisms, but, realistically it’s common for submitted data and code to be missing various steps,” he wrote of one section. “I think they’re too quick to use the term ‘fraud’ here,” he wrote of another. “It’s well known that there’s sloppiness in election administration, people check the wrong box so it looks like someone voted but they didn’t, etc. When [social security numbers] don’t line up, that’s not necessarily fraud, it could just be data entry error. Similarly, claims of over 100% turnout in precincts can arise from various incompatible datasets being combined.”

Gelman also addressed one of the most serious-seeming claims from the anonymous document:

Of these 1,132,646 observations (roughly 20% less than official numbers) [in one dataset], only 669,115 have variable value 1 on the vote2004 variable, and the remainder have value 0, i.e., 463,531 registered voters did not vote according to the Enos (2016) replication dataset, which is 104,260 more non-voters than the official figure. While the attrition of 20% of observations can be attributed to the author's stated inability to geocode a number of addresses and/or compute a probability of voter's race given their last name, even assuming all 283,455 registered voters discarded on those grounds had voted, leaving the author with a particularly unlucky draw of registered voters (which is statistically implausible), it would have been mathematically impossible for the maximum number of non-voters in a subsample, however peculiar, to exceed the official number of non-voters for the entire sample.

Gelman wrote:

This seems like a big deal. The way these anonymous authors put it, Enos is characterizing 463,531 people as being registered voters by not voting, and this contradicts the official data from 2004. This could be a data coding mistake on Enos's part: perhaps a large proportion of these 463,531 had some missing records, and Enos could've done a crude missing-data-exclusion procedure causing it to look like they didn't vote, but actually they just have missing data. For example, what if he only included people who voted on every item on the ballot. I have no idea. In any case, assuming the numbers in the anonymous article are correct, this seems like a big mistake--also, it's a good advertisement for why a replication policy is a big idea! The anon article has a bunch more details that suggest data problems, also they make a case that these problems affected the conclusions of the published paper. I have not tried to follow these details.

Gelman agreed that some of the points raised in the anonymous document suggested that Enos’ claims in one paper “weren’t supported by his data.” He also said he agreed with this harsh comment about a 2014 Enos paper someone else posted to Gelman’s own blog (archived here), which was actually cited in the anonymous document, though he wouldn’t go as far as to label the paper in question “pseudoscience” as the commenter did. Rather, said Gelman, “I’d say that work followed standard operating procedure of that era which indeed was to draw overly strong conclusions from quantitative data using forking paths.” (More information on forking paths is here, but the short version is that it’s a research practice that leads to, in effect, false statistical positives.)

All three researchers I contacted had the same general takeaway from the report: The case that Enos used questionable, but unfortunately quite common, statistical techniques to generate some of his findings was far stronger than the case that he had committed fraud. 

I woke up yesterday having decided to email my Chronicle editor and suggest we drop the story. I’m headed to Cuba for nine days tomorrow anyway, and it seemed like there wasn’t strong enough evidence of outright data fraud to pursue this further. I’d actually considered doing a meta post about my decision to drop the story: If some of Enos’ findings actually aren’t that robust, that would certainly be a useful thing for society to know and for a journalist to dig into. It’s unfortunate that journalists feel less incentivized to pursue stories about potentially wobbly but not fraudulent data than we do about outright fraud, given how much more common the former likely is than the latter, but that’s just the way it is — there are only so many hours in the day and most reporters are sitting on somewhere between three and dozens of story ideas they just haven’t had the time to fully pursue (I certainly am). And in my defense, I did write a whole book about wobbly-but-not-fraudulent findings, so I know how time-consuming it is to pursue such stories.

On top of all that, I’d heard another journalist was poking around on this, so I figured, Cool, maybe while I’m in Cuba whoever this is will either uncover evidence of something truly salacious — stranger things have happened, and no one I spoke with said it’s impossible something more sinister is going on here — or they’ll at least write an interesting piece about legitimate flaws in Enos research that don’t rise to the level of outright misconduct. 

Unfortunately, this happened instead.

***

It’s a Substack post by Christopher Brunet, an investigative reporter at the conservative Daily Caller News Foundation, headlined “EXCLUSIVE: Leaked Report Shows Harvard Professor Fabricated Data.” (Update: Shortly after this article went up, I got an email from Thomas Phippen, acting editor in chief of the Foundation. “I wanted to let you know that Chris Brunet no longer works for the Daily Caller News Foundation,” he said. “The story he published to Substack Sunday evening was not edited by the DCNF.” For what it’s worth, I based my description of Brunet on this post he published in January stating he had just gotten a job there, as well as the contents of his own emails — see below.) (Update to the update, 8:57 p.m.: Geoffrey Ingersoll pointed out that I didn’t change the subheadline, which read “A useful inadvertent cautionary tale from a Daily Caller reporter.” I’ve deleted “from a Daily Caller reporter” from the subhed and also struck out some language later in the post, as you’ll see.)

As you can tell from the headline, Brunet is convinced there’s actual data fraud here. You’d think his story would include new facts or analysis bolstering the case that the document really is a smoking gun. He doesn’t.

The article does show that Brunet isn’t familiar with the journalistic basics of how to approach a story like this. Luckily for us, he included all his email correspondence with the American Journal of Political Science (one of the journals where Enos published) and Harvard. They show he approached this story in a deeply unprofessional and confused way. 

Here’s part of an email Brunet sent to (apparently) the entire masthead of the journal:

I have obtained a leaked document (attached) that shows research misconduct involving a paper in your journal written by Dr. Ryan Enos. This document shows that his 2016 paper "What the Demolition of Public Housing Teaches Us about the Impact of Racial Threat on Political Behavior” does not comply with the AJPS Replication Policy. It is not simple p-hacking. It shows he blatantly manipulated/fabricated data. 

The document doesn’t show research misconduct. It alleges it. To send an email to a bunch of a social scientist’s peers claiming he “blatantly manipulated/fabricated data” simply because an anonymous document said so is not good journalism. 

Even weirder was this email Brunet sent to a handful of folks at Harvard, including Enos himself and Claudine Gay, Dean of Social Science for the Faculty of Arts and Sciences:

Dear Harvard Team, 

I am an investigative reporter at the Daily Caller News Foundation. 

I have obtained a document (attached) that shows there was an internal Harvard investigation into alleged research misconduct by Dr. Ryan Enos. 

I have also received the following information: Enos' tenure case was under review in 2018. The FAS dean at the time, Michael Smith was also chairing the committee on appointments and promotions in charge of reviewing this tenure case. Smith approved tenure. When Smith was informed of the misconduct, he swept it under the rug. Smith had announced a few months before he would step down, and that a FAS dean search would be conducted. To make sure he would get away with this embarrassing gaffe, instead of the scheduled FAS dean search, Smith impromptu appointed as his successor one of the main culprits in putting forward this dubious tenure case. The move was sold as a “diversity gain” to an unsuspecting then new president Bacow, who approved it. 

1. Can you please confirm that this is an authentic internal document? Was there ever any sort of internal investigation into Dr. Enos' alleged research misconduct? If yes: what was the outcome of that investigation? 

2. Are there any factual inaccuracies in the italicized paragraph? 3. Anything else you would like to add? 

Thanks, Chris [emphasis in the original]

First off, this suggests Brunet was confused about the document and didn’t read it closely. Nothing in the document showed anything about “an internal Harvard investigation” — it’s just a random document written by some individuals with stats knowledge. (Brunet did confirm that Harvard looked into the document and didn’t find it credible, for what that’s worth.) “Social Scientists for Research Integrity” generates zero Google hits, or it did before this article went up, at least.

More importantly, I recognized the italicized writing. It’s a chunk of text, mostly verbatim, from PoliSciRumors, an anonymous message board where anyone can post whatever they want without any accountability. The thread in question (archive) appears to be what led to the sudden recent surge of interest in the anonymous document — someone posted it there before the link stopped working.

PSR came up in the first feature-length article about a social science controversy I ever wrote that got some national attention. Here’s what I said about it: 

Three different people independently described PSR to me as a “cesspool.” No one knows exactly who the site’s primary denizens are, because hardly anyone will admit to perusing it, but it seems to skew young — mostly political-science grad students and untenured professors. While the ostensible purpose of PSR is to provide information about job openings, posts on it have a tendency to devolve into attacks, rumor-mongering, and bitterness fueled by an apocalyptic academic job market. “It is essentially the 4chan of political science,” a political-science researcher told me via email.

I’m not saying every single piece of scuttlebutt posted to the board is false, and in the instance I was writing about at the time, concerns posted there turned out to be well-founded. But you can’t copy a chunk of text from PSR, send it to a bunch of folks at Harvard, say “I have also received the following information,” and demand they comment! This is cartoonish. (I guess it’s technically possible some individual sent the text directly to Brunet and he didn’t realize it originated from or was also posted to PSR, but I’d level the same critique: You can’t just take a rumor someone sent you, neglect to make any effort to verify it, and demand people respond to it on the record.)

It really seems like Brunet’s investigation here is politically motivated —̶ ̶n̶o̶t̶ ̶j̶u̶s̶t̶ ̶b̶e̶c̶a̶u̶s̶e̶ ̶h̶e̶ ̶w̶o̶r̶k̶s̶ ̶a̶t̶ ̶T̶h̶e̶ ̶D̶a̶i̶l̶y̶ ̶C̶a̶l̶l̶e̶r̶ ̶N̶e̶w̶s̶ ̶F̶o̶u̶n̶d̶a̶t̶i̶o̶n̶,̶ ̶b̶u̶t̶ because of how he frames things: “Enos’ preferred theory in this instance is ‘racial threat theory,’ which is closely related to ‘critical race theory,’ and attempts to quantify the degree to which white people feel threatened by minorities.”

Just… no. There’s no meaningful connection between racial threat theory and critical race theory. Brunet should have known this because, well…

Christopher Brunet @realChrisBrunet

I should note that Enos' pet theory of Racial Threat Theory (RTT) is closely related to Critical Race Theory (CRT) Well, technically, I spoke to a CRT scholar and they were like "no no no no no they are not the same at all" Okay, but it is close. Same shit. 8/n

So a scholar told him accurately that the two concepts aren’t related, and Brunet, perhaps sensing this wouldn’t be conducive to his article’s ability to latch onto an ongoing culture war fracas and therefore go viral, just sorta… said the opposite? 

Okay.

There are a lot of other problems with Brunet’s article. Perhaps most egregious is how he handles a claim in the anonymous document about IP addresses: that, because they were associated with Boston/Cambridge commuters, it’s suspicious that they pointed to computers located elsewhere. “Unfortunately, upon being notified of potential data issues, Enos deleted the data in an attempt to scrub it from the internet,” writes Brunet.

This is a serious distortion of what Enos actually did.

Brunet embeds this tweet:

Ryan Enos @RyanDEnos

Today I removed a variable from the replication data for my article "Causal effect of intergroup contact on exclusionary attitudes" (PNAS 2014). There is an explanation of why I did so available here: scholar.harvard.edu/files/renos/fi… Please contact me if you have any questions.

If you simply click the link and read Enos’ statement, it becomes very hard to believe that what’s going on here is an attempted cover-up. You’ll see that Enos 1) explains why he removed the IP addresses from the data (confidentiality concerns), 2) explains why the IP addresses seem inaccurate, and says 3) “If anybody would like to request the original files, including IP addresses, they can contact me to arrange viewing privileges.” Enos’ statement also explicitly mentions that all this happened because he was accused of fraud. If this is his attempt at a cover-up, it seems strange that he would tweet an announcement about it.

I don’t want to go through the rest of Brunet’s article line by line. I think he botched this completely. His article takes the anonymous document’s claims at face value without presenting any evidence he attempted to verify them independently.

I emailed him to ask about this, and here was his response:

Hello! It depends what you meant by "independently confirm." I tried various official channels (e.g. going to AJPS/Harvard directly) and that was obviously fruitless as per the archived AJPS emails.

I spoke at great length with dozens of HRM economists/statisticians/data science/political scientists about the details of the leaked report and the publishing process. I walked them through the data. I ran the code myself. I studied the data myself. I trust my own judgement as an economist. If that means I am taking on legal risk for slander/libel or whatever, so be it. I knew I was undertaking that risk when I posted. Truth is a defence to libel. 

I could have written a whole post about the data. But readers would not understand or care about code. Maybe I will dedicate an entire substack article to the code. 

If Brunet put this much time and research into his efforts — if he even did what I did and simply sent the report to qualified independent experts — none of this comes through in his article itself. I also don’t understand how it could be that all three statistical experts I spoke with said there was a risk these allegations were overblown, but that Brunet never encountered this view (or if he did, why he left it out of his article).

Brunet’s piece really does spread wild speculation and rumor:

It's now possible/likely that a full professor at Harvard's 3 most influential and highly cited works (AJPS, PNAS, CUP) — plus his PhD dissertation! — are fabricated.

Is this enough to retract the articles? Is this enough to revoke tenure? Probably. I think Enos’ fate is a foregone conclusion. The wheels are already in motion. I am more interested in what will happen to the Harvard Deans (Smith + Gay) who helped cover this up. [emphasis his]

To state, as a fact, that two deans at Harvard “helped cover… up” an academic scandal, without presenting any substantive evidence, is extremely unprofessional.

Just to reiterate: Maybe there is stuff here worth taking a closer look at! I hope Enos will be transparent with regard to reasonable and logistically feasible requests for his raw data, code, methods, and so forth. I asked him about the “mathematically impossible” sample size shortly before I posted this and will update this article if he responds. I should also note that I don’t love writing this type of halfway article where I criticize another journalist’s efforts but don’t fully investigate the claims myself. It just would have felt weird to depart the country and leave Enos hanging out in the wind, with people publicly spreading rumors about him, when I knew that at least a few qualified experts did not find the claims in the anonymous document to be as damning as Brunet is treating them. I’m definitely open to revisiting this when I’m back in the States, so email me if you have any thoughts.

For now, though, the biggest point is that you can’t just run around accusing people of data fraud unless you have been very, very diligent in proving that that’s what actually happened. This whole thing has become far messier because of how Chris Brunet chose to approach it.

---

Questions? Comments? Ideas for data we should fabricate? I’m at singalminded@gmail.com or on Twitter at @jessesingal.

Image: “Question Mark Wood Cube on Blue Background” via Getty.