How a chatbot writes

How it works

How a chatbot writes

A chatbot takes a text and calculates how likely each possible next word is. It picks one of the likely ones, adds it to the text, and repeats the whole process. Again and again, until the answer is finished.

To be precise, it does not work with words but with tokens, which are pieces of text. A short word is often one token. A long or unusual word is split into several. Many other languages, such as Czech or German, usually need more tokens than English for the same text.

How it works

Training in two steps

First, the model learns from a huge amount of text: books, articles and web pages. It has only one task, to estimate the next piece of text. The result is a machine that can smoothly continue any text.

Then comes a second, smaller round of training. It uses example conversations and ratings of answers, some of them made by people. Only this step turns the model into an assistant that answers questions and follows instructions.

It does not contain a database of sentences. What it learned is stored as billions of numbers. But texts it saw many times, it sometimes remembers almost word for word.

How it works

Temperature

Temperature is a setting that decides how much risk a chatbot takes. When it is low, the chatbot almost always picks the most likely word. When it is high, even words it would almost never write get a chance.

In most chat apps, the app sets the temperature, not you. That is one reason why the same question can get a slightly different answer each time. The exact wording of your question, and what you wrote about before, also play a part.

How it works

Likely is not the same as true

The model learned to write text that fits the situation. When the right answer is common in texts, it gets it right. When the answer is rare, or does not exist at all, it writes something that only sounds right.

A confident tone means nothing here. The model writes just as smoothly whether it is right or not. That is what the next episode is about.

How it works

How much has the model read?

We do not know how much text the newest models from OpenAI, Google or Anthropic have read. The companies do not publish it. For GPT-4, in 2023, OpenAI explicitly declined to.

We do know the numbers from companies that published them. Meta said its model Llama 3, from 2024, was trained on more than 15 trillion tokens, which is roughly 11 trillion words. GPT-3, from 2020, learned from 300 billion tokens, about fifty times less.

If that were novels of a hundred thousand words each, it would be more than a hundred million books. Most of it is actually web pages, but for scale: someone who reads one book a week finishes about three thousand in sixty years.

And if you read all that text at a normal pace, day and night without a break, it would take you almost 90,000 years.

Sources: Meta: Introducing Meta Llama 3 (2024); Brown et al.: Language Models are Few-Shot Learners (2020); OpenAI: GPT-4 Technical Report (2023); OpenAI: What are tokens (a token is about ¾ of an English word); Brysbaert: How many words do we read per minute? (2019), 238 words per minute