Artificial Intelligence Is More Complicated Than “Like Humans, But Better”

A “New York Times” column reveals the limitations of half-baked AI prognostication

I’ve had a fun and sometimes fascinating time playing around with ChatGPT, but I haven’t really figured out anything interesting enough to say to devote a whole newsletter to it. I am, however, increasingly interested in the question of how commentary and predictions about AI are likely to get a bit crazy — or at least seriously half-baked — and I want to focus on an example of that today.

Okay, one observation about ChatGPT first! I think the earliest waves of hype over this program have been the result of basically a form of p-hacking, where you throw out instances of the program being quite dumb. Look, at its best, it’s remarkable. I’m not denying that. But interacting with ChatGPT is a bit like rooting for a rookie quarterback who is going to be great but who doesn’t have his legs under him yet. You’ll have a couple drives in a row in which he looks like he’s already a lock for the Pro Bowl, and then he’ll throw an absolutely mind-boggling interception, and it’s like, Oh yeah — he’s 22.

I won’t torture this metaphor any further, but ChatGPT has plenty of moments where it looks like an inexperienced 22-year-old. It has trouble following simple directions, it sometimes fails at basic logic, and on at least one occasion, it didn’t seem to understand the arrow of time (I asked it to list some reasons you’d be unlikely to see Fidel Castro in a New York City pizzeria in 2003, and it said that since Castro died in 2016, he wouldn’t be alive then — I kept blinking and asking myself if I were reading it right, but I was). So I think at least some of the folks who were posting early screenshots of it performing magnificently were also not posting screenshots of it being a total doofus. Maybe the academic analogy I’m looking for is the file-drawer problem rather than p-hacking.