Skip to main content

Capstone: the journey of one prompt

BeginnerLesson 16 of 166 min readReviewed by Mayank Raj Parmar on

In short

When you send a prompt, the app wraps it in instructions, and the model splits it into tokens, turns them into numbers, runs them through its layers and picks the next token, again and again until it stops. Every step is something you've already learned in this track.

Over 15 lessons, you've taken an LLM apart: tokens, embeddings, attention, training, fine-tuning, reasoning and tools. Now let's put it back together and follow one prompt all the way, from the moment you press Enter to the last word of the reply.

For the inside of the model, we'll use GPT-2 again (lessons 10 and 12), because it's small enough to run and every number is visible; the big chatbots don't show theirs. Where today's chatbots add extra steps, we'll say so. Our prompt is "Mumbai is the capital of the state of".

The whole journey at a glance

  1. 1. You press Enterlesson 15

    Your message leaves your device for the company's computers.

  2. 2. The app builds the contextlessons 5, 8, 12 and 14

    It writes the system message, the chat so far and your message into one long text, marked with special tokens. It may add search results or descriptions of tools.

  3. 3. Tokenslesson 5

    The text is split into tokens, and each token becomes its ID number.

    You get: 9 tokens, for our prompt.

  4. 4. Embeddingslessons 7 and 10

    Each ID picks out a list of numbers, and the token's position is added.

    You get: 9 lists of 768 numbers each, in GPT-2.

  5. 5. Transformer blockslessons 9 and 10

    In each block, attention lets the tokens share information, and a small neural network works on each one. The block repeats, again and again.

    You get: 12 blocks, with 144 attention heads, in GPT-2.

  6. 6. Chances for the next tokenlesson 4

    The last token's numbers become a chance for every token the model knows.

    You get: 50,257 chances, in GPT-2.

  7. 7. Pick one, and go round againlessons 4 and 13

    One token is picked, guided by the temperature, added to the text, and everything runs again. A reasoning model writes its thinking in this loop before its answer.

  8. 8. Stop and showlesson 12

    A chatbot stops when it writes its end-of-turn token. The app shows the words as they arrive.

  9. 9. You check itlessons 6 and 15

    Is it true? Does it leave something out? Is it safe to use?

Figure 1: the journey of a prompt, with the lessons that explain each step. Steps 3 to 7 happen inside the model.

Before the model sees anything

When you press Enter, your message leaves your device for the company's computers, which is why lesson 15 said to think before you paste.

Then the app builds the text the model will actually read. A model only ever reads one long run of tokens, so the app writes everything into one text, marked with special tokens (lesson 12): a system message with the app's rules, the conversation so far, and your new message. It may add search results (RAG, lesson 8) or descriptions of tools the model can ask for (lesson 14). All of it has to fit in the context window (lesson 5).

GPT-2 is a base model, so it can't chat (lesson 12). We'll give it a plain sentence to continue, which is all it was trained to do.

Tokens

Mumbai is the capital of the state of

  1. M, ID 44
  2. umbai, ID 21645
  3. space then is, ID 318
  4. space then the, ID 262
  5. space then capital, ID 3139
  6. space then of, ID 286
  7. space then the, ID 262
  8. space then state, ID 1181
  9. space then of, ID 286
Figure 2: our prompt as GPT-2's 9 tokens, each with its ID. A dot marks a space at the start of a token.

Common words get a token of their own, like " the" (ID 262) and " of" (ID 286). Each appears twice, with the same ID both times. "Mumbai" is rarer in the text GPT-2's tokenizer learned from, so it's built from two pieces, "M" and "umbai" (lesson 5).

From IDs to numbers

Each ID picks out one row from a big table of embeddings: a list of 768 numbers that GPT-2 learned during training (lesson 7). The token's position is added in (lesson 10), so the two " the" tokens start out almost the same, but not quite. Our 9 tokens are now 9 lists of 768 numbers.

Through the Transformer blocks

The 9 lists now pass through GPT-2's 12 Transformer blocks, one after another (lesson 10). In each block, attention lets every token gather information from the tokens before it, so " state" can take in that this sentence is about "capital" and "Mumbai". Then a small neural network (lesson 9) works on each token's numbers on its own.

By the last block, the numbers for the final token, " of", hold what the model needs to guess what comes next. All this work is done by GPT-2's 124 million parameters, which were set during training (lesson 11).

A chance for every token

The final numbers are turned into a score for each of the 50,257 tokens GPT-2 knows, and then into chances that add up to 100% (lesson 4). Here are GPT-2's real chances:

Mumbai is the capital of the state of ___

  1. ·Maharashtra64.4%
  2. ·Gujarat8.7%
  3. ·Mad8.0%
  4. ·Karn2.7%
  5. ·Tamil2.3%
  6. the other 50,252 tokens14.0%

A dot marks a space at the start of a token.

Figure 3: GPT-2's real chances for the token after our prompt. The top five get 86% between them, and the other 50,252 tokens share the remaining 14%.

" Maharashtra" is the clear favourite, at 64.4%. " Mad" and " Karn" are the first pieces of longer names, Madhya Pradesh and Karnataka. The model never sees whole words, only tokens.

Pick one, and go round again

Next, one token is picked, and the temperature decides how adventurous the pick is (lesson 4). Here's what GPT-2's real chances become at the three temperatures from lesson 4:

Next tokenLow (0.5)Normal (1)High (1.5)
Maharashtra96.3%64.4%22.8%
Gujarat1.7%8.7%6.0%
Mad1.5%8.0%5.7%
Karn0.2%2.7%2.7%
Tamil0.1%2.3%2.5%
Everything else0.2%14.0%60.2%

At a high temperature, 60.2% of the chance goes to tokens outside the top five, spread across more than 50,000 of them.

Let's take the top token every time, like a temperature of 0. GPT-2 adds " Maharashtra" to the text, and the whole thing runs again, through all 12 blocks, now with 10 tokens. Then again, and again:

Mumbai is the capital of the state of Maharashtra, which is home to the largest number of people of…

  1. ·Maharashtra64.4%
  2. ,36.1%
  3. ·which18.7%
  4. ·is37.9%
  5. ·home26.1%
  6. ·to99.6%
  7. ·the13.8%
  8. ·largest23.1%
  9. ·number37.3%
  10. ·of99.6%
  11. ·people3.7%
  12. ·of25.3%
Figure 4: the reply GPT-2 wrote, one token at a time, always taking its top choice. Each bar is the chance it gave the token it picked.

Some picks were nearly certain: after " home", the token " to" had 99.6%. Others were close calls: " people" was the top choice with just 3.7%. That's why the same prompt can get different replies (lesson 4). At a temperature of 1, the chance of getting exactly this reply again is about 1 in 2 million.

A reasoning model runs this same loop to write its thinking first, often hidden, before it writes the answer (lesson 13).

Knowing when to stop

GPT-2 would have carried on writing. At every step, it gave its end-of-text token less than a hundredth of a percent. It's a base model, trained only to continue text. A chatbot is fine-tuned on conversations that end with an end-of-turn token (lesson 12), so once its answer is complete, that token becomes its likely next pick, and the app stops there. Meanwhile, the app shows you each piece as it's picked, which is why replies appear a few words at a time.

The last step is you

The reply starts with "Maharashtra", which is right. But GPT-2 gave Gujarat 8.7% and " Mad" 8.0%, and at a high temperature it would pick something other than Maharashtra 77.2% of the time, writing it just as smoothly. The rest of its reply, about Maharashtra being "home to the largest number of people of", is the model continuing a pattern, not checking a fact. That's lesson 6 in one example: it sounds just as sure either way, so check what matters (lesson 15).

Where the knowledge came from

Every number above came from GPT-2's parameters, and those came from training (lesson 11). GPT-2 learned from about 8 million web pages, roughly 40 GB of text, by predicting the next token over and over. That's how it learned that "Mumbai is the capital of the state of" is usually followed by "Maharashtra". Today's chatbots go through the same kind of pre-training at a vastly larger scale, then fine-tuning and feedback from people (lesson 12) to become assistants.

Where to go from here

You now know how the whole thing works, from your keyboard to the reply. Some ways to keep going:

Key takeaways

  • Before the model sees your prompt, the app wraps it with its rules, the chat so far and sometimes search results or tools, all in one long text.
  • Inside the model, the text becomes tokens, then embeddings, then passes through stacked Transformer blocks of attention and small neural networks.
  • At the end of each pass, the model gives a chance to every token it knows, picks one and goes round again. A chatbot stops at its end-of-turn token.
  • Temperature and close calls are why the same prompt can give different replies.
  • The model writes what's likely, not what it has checked. You're the last step: check what matters.

Check your understanding

Question 1 of 8
Inside the model, in which order do these steps happen?

Sources and further reading

Report an error in this lesson