A lawyer handed six court cases to a federal judge. None of them existed.
It happened in New York in 2023 (opens in a new tab). He’d asked ChatGPT to find cases supporting his client’s claim against an airline, and it produced six: case names, case numbers, quotes from the rulings. They looked exactly like real citations. He filed them.
The airline’s lawyers went looking and found nothing. The judge went looking and found nothing. The AI had invented all six. The court fined him $5,000 (opens in a new tab) and ordered him to write to every judge the fake rulings had named.
He didn’t set out to lie. He trusted a confident answer.
Why that is the dangerous part
AI mistakes don’t look like mistakes. The wrong answer arrives in the same confident tone as the right one, formatted just as neatly, with no tell. That’s the whole problem: there’s nothing in the output to catch your eye.
AI will get things wrong. Plan for it. The job isn’t to prevent every error. It’s to catch them before they reach anyone who matters.
The mistakes follow a pattern
They show up most when AI cites sources, gives exact numbers, or states facts that need outside checking.
They show up least when it summarises, drafts, or reorganises information you gave it, because there, everything it needs is already in front of it.
That pattern tells you where to put your attention. You don’t need to read every sentence twice. You need to check every fact, every number, and every named source (opens in a new tab). The rest can move fast.
One rule is enough to start
Before anything touches a customer, a contract, or a public channel, a person checks the facts.
One sentence. The mistake it catches could cost you a client. Or five thousand dollars and a letter to a judge.