{"t0":"Apply the feed-forward transformation","t1":"Multiplication makes three views of the input: queries, keys and values. A query is compared with available keys. A larger dot product gives that position more attention after normalization.","t2":"A fixed transformation mixes coordinates within each position and replaces negative intermediate values with zero. This feed-forward step computes new values; it does not rearrange the old ones.","t3":"An output matrix turns the final position’s representation into eight scores. Softmax converts those scores into a distribution over our entire toy vocabulary. Real models use much larger vocabularies.","t4":". That ID selects four stored numbers from the table. This starting list is its embedding. Here each word gets one ID to keep the example small.","t5":"The program makes three sets of numbers called queries, keys and values. Comparing a query with available keys gives each word a share of attention. Larger comparison scores get larger shares.","t6":"The program mixes information from the available words using those shares. The working numbers for “","t7":"” now depend on its context. It is calculating, not reading a fact from a cell.","t8":"A fixed calculation mixes the numbers at each word position and changes negative intermediate numbers to zero. It creates new working numbers instead of just moving old numbers around.","t9":"The final word position produces eight scores. A calculation called softmax turns them into probabilities for all eight words in this toy. Real models have much larger word-piece lists.","t10":"Prediction uses the final “is” row, even if you inspect another row. Each position uses the same input snapshot; the updated “bank” row does not feed into “is” in this block.","t11":"Yes. The same ID selects the same embedding row. Context can change its later representation.","t12":"The lookup stays the same for the same ID and fixed table. Context changes later calculations, not that initial row. Try the other answer.","t13":"Future words included for comparison; this is not causal attention."}

Inside a prediction

How context changes a prediction

Try “river bank” and “savings bank”. The surrounding word changes which next words look likely. The model’s stored settings stay the same; the calculations change with your sentence. That is why changing a prompt is different from training a model.

Real calculations with hand-designed teaching weights. This is a miniature explanation, not a trained language model.

Thebank is…

Change river to savings to compare the results. Both sentences use the same fixed parameters.

Pace Still view

4 snapshots × 4 positions × 4 features

Turn the cube Separate slices

Rows: The, river, bank, is. Columns: f1 to f4. Depth: four calculation snapshots.

Each tile is a number, not a fact or physical memory location. Inspect the selected snapshot with its tiles or arrow keys. Choose another stage to inspect its numbers, or use Flat view to unlock every snapshot.

Step 1 of 5

Look up the token embedding

Inspect a number

Word positionCoordinate

After the final word “is”

What could come next?

All eight words in this toy vocabulary. These are next-token probabilities, not confidence that a statement is true.

Change the context word to compare predictions, or select Next step to inspect the calculation.

This interactive calculation needs JavaScript. The worked example below remains readable: “river” selects [2, 0, 1, 0]; “savings” selects [0, 2, 1, 0]. “Bank” selects [0, 0, 1, 1] in both. Attention combines earlier value vectors, so “bank” later changes with its context.

Inspect the tables and calculations

The cube, tables and arithmetic show the same selected word and coordinate. Expand a section below to inspect its values or read how they are calculated.

Read the cube as a table

Show full values in the table and arithmetic

The table and cube read the same four arrays. Cube tiles round to three decimal places for readability; full values and exports preserve the browser numbers. More dimensions are possible; a three-dimensional picture is only one way to show them.

Where do the first numbers come from?

A token ID selects a row in an embedding table. That is an actual lookup. The ID itself is an address, not a score for meaning. Real models learn their embedding values during training; Jonathan and AI assistance authored this tiny table to make the arithmetic legible.

Fixed toy embedding table. Word-level tokenizer, eight IDs.
Token ID	Word	f1	f2	f3	f4
0	The	0	0	0	1
1	river	2	0	1	0
2	savings	0	2	1	0
3	bank	0	0	1	1
4	is	0	0	1	0
5	muddy	1	0	0	0
6	closed	0	1	0	0
7	quiet	0	0	0	0

No coordinate is a universal “river” or “money” axis. Larger language models usually split text into subword tokens and learn distributed representations.

Are these numbers still made of bits?

Yes. Bits encode the numbers; tensors organize them into arrays. Select a tile above to see that number rounded to a 32-bit floating-point representation.

One sign bit, eight exponent bits and 23 fraction bits. This browser computes using 64-bit JavaScript numbers. The 32-bit display is a separate conversion, so rounding may change the value.

How does attention mix information?

Attention mixes information from the available words. It calculates how much each word contributes, with shares adding up to one, then combines their numbers using those shares. Select a word and a number in the cube to inspect the calculation. The technical names for these parts are query, key and value.

Hide future words

Causal language models cannot use later words to predict an earlier one. Turn this off as a comparison experiment; it is no longer causal attention. The final position has no future word in either mode.

Q = XWq; K = XWk; V = XWv; A = softmax(QKᵀ / √4 + mask); C = AV

The next step applies a feed-forward transformation independently to each position: F = ReLU(CW1)W2. Finally softmax(FWout + bias) gives the eight output probabilities.

Each position is calculated from the same input snapshot. Within this one block, the updated “bank” row does not feed into “is”. A later block could use the updated rows. The final “is” row supplies the next-word prediction shown here.

Weights, working numbers and retrieved text

Parameters

These are the model’s stored settings. Real models usually learn them during training. They stay fixed while the model answers. This toy’s settings were designed by hand.

Activations

These are the working numbers you see in the cube. A different sentence changes them. They are temporary calculations, not a permanent store of facts.

External retrieval

A separate search can find documents and add text to a model's context. This exhibit does not retrieve documents. That process is different from selecting an embedding row or combining vectors.

Real language models repeat many blocks and commonly include position information, multiple attention heads, residual connections and normalization. Those are omitted here. The four slices are teaching snapshots, not four complete Transformer layers. An output also need not be the highest-probability token; sampling can choose another.

Try the idea on a new sentence.

Imagine “The wooden bat is” and “The flying bat is”. Before context is mixed, what happens to the embedding for the same token “bat”?

Choose a prediction. Assume the same token ID and fixed embedding table.

Save or rerun this experiment

Keep every input, parameter and result, or run the calculation yourself with standard Python. No package installation needed.

Next: See what changes when a model receives a missing source sentence. Retrieving a document supplies context; it is different from looking up a token's embedding.

Sources and how this was made

An original explanation by Jonathan Simone with AI assistance. No pretrained model or live inference service is running here. All lesson calculations run in your browser.

PyTorch: embedding lookup
Vaswani and colleagues: attention, feed-forward layers and output softmax
NVIDIA: binary floating-point representation
Lewis and colleagues: retrieval-augmented generation
Brendan Bycroft's LLM visualization and Transformer Explainer inspired the linked views. No code or artwork copied.