A static visual walkthrough for understanding how an LLM turns text into the next token. Read it left to right: tokens become vectors, vectors exchange context through attention, repeated blocks refine the state, and the final vector becomes vocabulary scores.
At every layer, each token position has a numeric vector. Attention lets that vector pull information from earlier tokens. The MLP then reshapes the vector's internal features. After many repeats, the last visible token is used to predict what comes next.
The same position keeps one hidden state, but every block rewrites it with more context.
Text is split into token IDs. The model sees numbers, not words.
Each ID becomes a learned vector, plus position information.
Linear projections create search, address, and content vectors.
The focus token mixes useful information from prior tokens.
The update is added back, normalized, then transformed per token.
Many layers refine the sequence representation.
The final vector becomes scores over the vocabulary.
The cards use the same example token, "because", so the path stays concrete. On wide screens the cards form a left-to-right learning flow; on smaller screens they stack vertically.
The token ID points into an embedding table. The model retrieves one learned vector and combines it with position information, so "because at position 7" is represented differently from the same token elsewhere.
The vector is not a dictionary definition. It is a learned numeric starting point that later layers keep rewriting.
Each token vector is multiplied by three trained matrices. These create three views of the same token: what it is looking for, how other tokens can match it, and what information it can contribute.
Q, K, and V are activations. The matrices Wq, Wk, and Wv are the learned model weights.
The focus query is compared with earlier keys. Softmax turns the scores into weights, then those weights mix the value vectors into one context update for the focus token.
Attention is not a permanent memory. It is computed fresh for the current prompt and current layer.
Decoder-only LLMs predict the next token, so a position cannot inspect future positions. The focus token can use the left side of the sentence, but not tokens that have not been generated yet.
The triangle shape is the reason a decoder can be trained to predict text from left to right.
The attention update is added back to the incoming vector. Then the MLP expands, gates, and compresses each token independently. Attention mixes across tokens; the MLP reshapes features inside each token.
Residual connections make deep stacks trainable because each layer can add a correction instead of replacing everything.
A modern LLM stacks many decoder blocks. Every block repeats the same pattern, but each block has its own learned weights and can refine the representation in a different way.
Depth is repeated refinement. The sequence does not move to one single vector; every token position keeps its own state.
The last-layer vector at the current position is multiplied by an output matrix. The result is one score per possible next token. Sampling or decoding policy chooses the next token from those scores.
An LLM does not produce a whole answer in one pass. It predicts one token, appends it, and repeats the process.
z = x + Attention(RMSNorm(x))
y = z + MLP(RMSNorm(z))
The same shape repeats many times. Normalization stabilizes the values, residual paths preserve information, and each sublayer adds a learned update.
Attention gives each position controlled access to prior context. The MLP changes the internal feature representation. Repeating the block lets simple token vectors become rich context-sensitive vectors that can predict the next token.