Inside a large language model

How a language model generates text

Embed looks up each token's starting list of numbers in a table the model learned. Prefill then runs every prompt token up through the layers, computing a key/value pair for each token at every layer based on the model's weights. Decode writes the reply one token at a time, picking each one from the odds computed at the top of the previous token's column.

Where it runs, typically: data centers→ many model copies→ one copy = many chips→ shared by many users

One request, layer by layer

embedcomputingkey/value pairscached from request 1

Input

next-token odds

Where this fits, and further reading

This is step 05 of How Claude Code works, which follows one prompt from your terminal to the model and back. What happens inside the provider's data centers is described from public documentation and standard techniques for serving large models; real implementations may differ. Token counts and odds are representative examples.

Background: how transformers run

Vaswani et al., Attention Is All You Need (2017): attention, keys, values and MLP layers
Pope et al., Efficiently Scaling Transformer Inference (2022): prefill vs. decode and the KV cache
Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (2023): storing and reusing KV caches

Claude API documentation

Prompt caching: how cached key/value pairs are reused across requests