What Happens Inside an LLM When You Hit "Send"?

Why This Matters

You type a message into ChatGPT or Claude, hit send, and a few seconds later, words start appearing on screen. For most people, that gap in between is a black box. For a beginner learning GenAI, understanding what actually happens in that gap is one of the most useful mental models you can build — it makes everything else (tokens, context windows, hallucinations, temperature) click into place much faster.

So let's walk through it, step by step, from the moment you hit send to the moment the response finishes.

Step 1: Your Message Gets Tokenized

Before the model can "think" about anything, your text has to be converted into a format it actually understands — numbers, not words. This is tokenization, which I covered in detail in an earlier post. Your sentence gets broken into small chunks (tokens), and each token gets mapped to a unique ID number.

You type: "Explain gravity simply"
Tokenizer output: [Explain] [ grav] [ity] [ simply]
Token IDs: [37471, 1122, 615, 8890]

The model never actually "reads" your sentence the way you do. It only ever sees a sequence of numbers.

Step 2: Those Tokens Become Embeddings

Each token ID gets converted into an embedding — a long list of numbers (a vector) that represents the meaning of that token in a high-dimensional space. Words with similar meanings end up with embeddings that are mathematically "close" to each other.

This is the model's way of representing meaning numerically. It's also how the model can understand that "king" and "queen" are related, or that "happy" and "joyful" are similar, without anyone explicitly telling it so — it learned these relationships during training.

Step 3: Your Whole Conversation Gets Loaded as Context

Your new message doesn't go in alone. It gets combined with everything else the model needs to "see" in order to respond well:

• The system prompt (invisible instructions set by the app, like "you are a helpful assistant")
• The conversation history (previous messages in this chat, if any)
• Your new message

All of this together is called the context window — the total amount of text the model can "see" at once. This is why long conversations eventually hit limits, and why the model can lose track of something you said much earlier in a very long chat.

Step 4: The Model Processes Everything Through Attention

This is the core of what makes an LLM actually work — the transformer architecture, specifically a mechanism called self-attention.

In simple terms: for every token in your input, the model calculates how much "attention" it should pay to every other token, to figure out relationships and context. This happens across many layers (sometimes dozens or over a hundred), each layer refining the model's understanding a little further.

This is how the model figures out that in the sentence "the trophy didn't fit in the suitcase because it was too big," the word "it" refers to the trophy, not the suitcase — by weighing the relationships between all the words, not just reading left to right.

Step 5: The Model Predicts the Next Token

After all that processing, the model doesn't generate a full sentence in one shot. It predicts one token at a time. For every position, the model outputs a probability distribution over its entire vocabulary — essentially a ranked list of "here's what's likely to come next."

Given: "The sky is"
Model's top predictions: "blue" (62%), "clear" (14%), "grey" (9%), "falling" (2%)...

A setting called temperature controls how the model picks from this list. Low temperature almost always picks the top prediction (safe, predictable output). Higher temperature allows more randomness, occasionally picking less likely options (more creative, less predictable output).

Step 6: The Chosen Token Gets Added, and the Loop Repeats

Once a token is picked, it gets added to the sequence, and the entire process repeats — the model looks at everything so far (your message + every token it has generated) and predicts the next one. Token by token, this is why responses "stream" in — you're literally watching the prediction loop happen in near real-time, one token at a time.

This continues until the model generates a special "stop" token, or hits a maximum length limit set by the application.

Step 7: Tokens Get Converted Back to Text

Finally, the sequence of predicted token IDs gets converted back into readable text — the reverse of step 1 — and that's what shows up on your screen as the model's response.

Putting It All Together

Here's the whole pipeline, start to finish:

Your message → tokenized into IDs → converted into embeddings → combined with context (system prompt + history) → processed through attention layers → next token predicted → token added, loop repeats → stop token reached → tokens converted back to text → response shown to you

None of this involves the model "understanding" your question the way a human does. It's an extremely sophisticated pattern-matching and prediction system — which is also exactly why it sometimes confidently produces wrong answers (hallucinations). It's not "looking up facts." It's predicting statistically likely sequences of tokens based on everything it learned during training.

Why This Mental Model Helps

Once this clicks, a lot of confusing LLM behavior starts making sense:

• Why longer conversations get expensive and slow — every single token in the context window gets reprocessed through attention each time.
• Why the model sometimes "forgets" earlier context — if the conversation exceeds the context window, older tokens literally aren't visible to the model anymore.
• Why the same prompt can give different answers — temperature and probabilistic sampling mean the "next token" isn't always the same choice.
• Why hallucinations happen — the model is predicting plausible-sounding text, not retrieving verified facts from a database.

Closing Thought

"Hit send and get a response" feels instant and simple from the outside, but underneath it's tokenization, embeddings, layers of attention, and a token-by-token prediction loop happening at incredible speed. You don't need to understand the deep math to build with LLMs — but having this mental model makes every other GenAI concept (context windows, temperature, RAG, fine-tuning) a lot easier to reason about instead of just memorizing definitions.