A Chatbot Coined A Phrase I Really Like. Am I Allowed To Use It?
We're about to face some really weird ethical (and citational) questions
PREORDER KATIE HERZOG’S BOOK, Drink Your Way Sober: The Science-Based Method to Break Free from Alcohol.
Over the weekend I got started on a post about the ethical questions faced by professional writers who use AI to help them edit or research their work. The early parts of the early draft started to feel a bit ponderous, just not quite what I was going for, so I set it aside.
Then, earlier today, reality generated a perfect example of the sort of ethical quandary I’m talking about.
I was messing around with Claude Pro. I asked it to adopt the role of someone with a PhD in research methods, and then fed it my critique of the Olson-Kennedy et al. preprint I just published. I wanted to get Claude’s thoughts on my critique of it. (I’d already asked another chatbot about the preprint itself, which was how I was tipped off to the potential issue with how the researchers approached missing data — see my second footnote from that piece.)
During the ensuing exchange, Claude said this:
Overall Assessment
From a research methods perspective, these are not minor technical issues but fundamental threats to both internal and external validity. The combination of outcome switching, high attrition without adequate analysis, uncontrolled confounders, and shifting theoretical frameworks makes it extremely difficult to draw meaningful conclusions from this study.
The preprint appears to suffer from what might be called "confirmatory research theater" - maintaining the appearance of rigorous hypothesis testing while systematically undermining the conditions that make such testing meaningful.
That phrase “confirmatory research theater” jumped off the screen at me. Did Claude. . . coin it? It seemed to have, given the “what might be called” construction, which in writerly contexts is usually a fancy way of saying “I just came up with this.” I googled the phrase and sure enough, it isn’t indexed anywhere with either the American (i.e., real) or British (i.e., fake) spelling:
I really like this phrase.
If a human writer — imagine she’s a sociologist alliteratively named Sally S. Smith — produced the above text, I would file away “confirmatory research theater” as a phrase to use later on. At some point I’d write something like “This case perfectly exemplifies what the sociologist Sally S. Smith calls ‘confirmatory research theater,’ which she describes as research that maintains the appearance of careful, rigorous work while violating the conditions that underpin such work — all in the service of confirming the hypothesis the researchers believed in, or sought to prove, all along.”
