#3 Why AI makes things up
A confident friend who never admits not knowing something. Can you tell when the AI is making things up?
A quote from Marie Curie
- Thursday morning. Jana’s colleague Martin has a presentation this afternoon.Martin: Jana, I’ve got a great quote for my presentation. From Marie Curie. The chatbot found it for me in a second.Jana: Show me.
- Martin: “Whoever fears artificial intelligence fears the future.” Marie Curie, 1932.Jana: In 1932 nobody talked about artificial intelligence. The term only appeared in 1956.
- Martin: But it even gave me the year!Jana: A year is just another word. Come on, let’s ask it.
- Jana: Did Marie Curie say anything about artificial intelligence?AI: Not as far as I know. She died before the term existed.
- Jana: A minute ago you gave Martin exactly that kind of quote. With a year.AI: But I didn’t find it, I just wrote it. Martin wanted a quote from Marie Curie, so he got a sentence that sounds like one.
- Jana: Why didn’t you just say there isn’t one?AI: Remember how I write? Word by word. And after “Marie Curie said:” comes a quote, not “I don’t know”.
How it works
It keeps writing, even when it does not know
In the last episode you saw that a chatbot does not look anything up. It continues a text with whatever is most likely to fit. When the right answer is common in the texts it learned from, likely and true are the same thing.
When the answer is not there, because it is rare, local, or does not exist at all, the model does not see the gap. It fills in what would most likely stand in that place. A name that sounds like a name, a year that sounds like a year, a quote that sounds like a quote.
From the inside, this looks exactly the same as a correct answer. That is why it does not know that it does not know.
The confident friend
- On the way to lunchJana: So you’re like a friend who knows everything. You ask him the way, and he describes it. Even if he has never been there.AI: That friend at least suspects he has never been there. I don’t even suspect that.
- Jana: And what if I asked you about something you can’t possibly know anything about?AI: Try it. And guess what I will write.
- Your turn!
Jana asks: “Who was the mayor of Cucuron in 1931?” What will the chatbot most likely write?
How it works
Why it is called a hallucination
When a chatbot writes something that is not true, it is called a hallucination. It is not a lie. To lie, you have to know the truth and say something else. The model does not know the truth. It only has the probabilities of words.
It used exactly the same process as for a correct answer. That is why hallucinations cannot be “switched off”. They are the other side of the same ability that lets it write smoothly about anything.
Newer and larger models make things up less often, and some apps can look the answer up on the web. But less often does not mean never. And the tone stays just as confident.
Spot the invented fact
- After lunchMartin: Instead of the quote, I’ve got three interesting facts about Petřín Lookout Tower in Prague. They sounded good. All three of them.Jana: So did Marie Curie. I bet one of them is invented.
- Your turn!
Three facts about Petřín Lookout Tower in Prague, as the chatbot wrote them. One is invented. Which one?
How it works
Where it makes things up most often
A chatbot usually gets a general explanation of an ordinary thing right, because it has read it a thousand times. The more exact and rare the detail you ask for, the more likely it fills in the most probable filler instead of the truth.
Most of the text a model learned from is in English. In smaller languages, and about smaller countries, it has seen far less, so it guesses more often about local names, villages, laws and institutions.
- Exact details: names, dates, numbers, prices, sections of laws, links.
- Quotes, and who said what. If you ask for a quote, you get a quote. Ask first whether it exists at all.
- Local and rare topics that little is written about.
- Questions built on a wrong assumption. Ask for a summary of a book the author never wrote, and sometimes you get one.
Habits that catch it
- Jana: So how am I supposed to tell, if it all sounds exactly the same?AI: By how it sounds, you can’t. By what you do with it, you can. With a friend who knows everything, you would check too.
- Your turn!
Which of these will help you catch an invented detail?
Tick everything you think applies. Then check.
- A little laterMartin: That quote… I already sent it to my boss. This morning.Jana: Then write to him that Marie Curie never said it.
- Martin: Writing now. And I’ll add that it was actually Einstein.Jana: Did you check that?
- Martin: …I’ll ask the chatbot.Jana: Einstein didn’t say it either. Nobody said it.
How it works
A real case: invented court decisions
In 2023, the New York lawyer Steven Schwartz wrote a court filing that relied on six earlier court decisions. ChatGPT had found all of them for him. Not one of them existed. The chatbot had invented them, quotes included, and credited them to real judges.
The lawyer even asked the chatbot whether the cases were real. It said yes, and that they could be found in legal databases. The problem was discovered by the lawyers for the other side, who could not find the decisions. The judge then asked for their full text.
In June 2023, the court fined the lawyer, his colleague and their law firm 5,000 dollars, and ordered them to write letters of apology to the judges whose names had appeared in the invented decisions.
Sources: Sara Merken, Reuters, 22 June 2023: New York lawyers sanctioned for using fake ChatGPT cases in legal brief; Benjamin Weiser, The New York Times, 27 May 2023: Here's What Happens When Your Lawyer Uses ChatGPT; AI Incident Database, incident 541
- In the eveningJana: My niece has to hand in a school report tomorrow. Guess who wrote it for her.AI: I hope the chatbot only helped. She should check every name and year. Her teacher will find the invented ones faster than I can.
- Jana: So: a friend who knows everything. And doesn’t know what he doesn’t know.AI: And who will happily tell you what Marie Curie thought about it.
Quiz
- AI: Five questions. This time, feel free to check them.Jana: I hope you didn’t fact-check them by asking yourself.
- Quiz
Question 1 of 5
Five questions, no marks.
Why does a chatbot sometimes write something that is not true?
Choose one answer.
What to take away
- A chatbot continues a text with what probably belongs in it. When it does not know the right answer, it writes something that only sounds right.
- It does not know that it does not know. An invented sentence looks the same from the inside as a true one, so it rarely writes “I don’t know” on its own.
- A confident tone, details, a year or a name prove nothing. They are also just likely words.
- It most often makes up exact details, and things about rare or local topics. Check exactly those somewhere other than the chatbot.
- Asking the chatbot whether it is sure is not enough. What helps is another source, or a person who knows the subject.
Next episode
What to fear about AISpecial
Robots, mind reading, consciousness. A Halloween special about which fears of AI are real, and which come from films.
New episodes straight to your inbox
Every two weeks we send the new episodes, a short learning note and news from the world of AI. Free, and you can unsubscribe any time.