Serving, Running One Yourself, RAG, Agents and the Limits

51.0 What this chapter gives you#

1. You will be able to say what changes when a model stops learning and starts answering, and give the memory numbers for both.
2. You will be able to read a real API request body field by field, and say what the system prompt is and what it is not.
3. You will be able to explain, with arithmetic, why the first token is slow for one reason and the rest are slow for a completely different reason.
4. You will be able to name the serving tricks that make a model fast, and say why the same model runs at different speeds at different providers.
5. You will be able to build a cost model for a real application and decide honestly whether to self-host.
6. You will be able to run a model on your own laptop with two named tools, and predict beforehand what speed you will get.
7. You will be able to download weights, read a model card, and pick a quantization level on purpose rather than by guessing.
8. You will be able to explain retrieval-augmented generation step by step, including how approximate nearest neighbour search works.
9. You will be able to explain tool calling and agents precisely, including why every tool is a security boundary and why long agent runs fail.
10. You will be able to state the real limits, including why hallucination is intrinsic and why prompt injection is still unsolved as of August 2026.

51.1 Inference versus training: what changes#

PLAIN51.1.1 in simple words#

1. Training is the process that produced the numbers inside the model.
2. Inference is the process of using those numbers to answer a question.
3. During training, every number in the model can move.
4. During inference, not one of them moves. They are read and never written.
5. Training needs to remember, for each number, which way it was moving and how fast it has been moving lately. That is several extra copies.
6. Inference needs the numbers themselves, plus some working notes about the conversation so far.
7. So the same model needs roughly ten times less memory to use than to train.
8. Training is judged on work finished per week. Inference is judged on how long one person waits for one answer.
9. A training run that takes 30 days is normal. An answer that takes 30 seconds feels broken.

PLAIN51.1.2 a picture in your head#

1. Think of a very large printed cookbook that a team of chefs wrote over a year.
2. Writing it was training. Drafts, corrections, arguments, notes in the margins about which way each recipe was heading.
3. All of that scaffolding took far more desk space than the finished book.
4. Cooking from it is inference. You open the book, read, and cook.
5. You do need the book itself open in front of you, and a small notepad for what you have already done in this particular meal.
6. But if a customer waits 40 minutes for soup, the restaurant fails.

Where this comparison breaks: a chef can read one page and ignore the rest. A language model reads every page of the book for every single word it produces. That is the fact that governs the whole of this chapter, and no ordinary object behaves that way.

PLAIN51.1.3 a worked example#

1. Take a real model: Llama 3.1 8B, which has 8.03 billion parameters.
2. Chapter 47 gave the rule: memory equals parameter count times bytes per parameter.
3. At 2 bytes per parameter that is 16.1 billion bytes, which is 15.0 GiB.
4. To use the model, you need those 15.0 GiB plus working space.
5. To train the model, you need the same 15.0 GiB plus four more copies.
6. Here is the whole comparison, in GiB, for one 8B model:

7. So the same model is a 17.5 GiB problem to run and a 165 GiB problem to train fully. That is a factor of about 9.4.
8. 17.5 GiB fits on one consumer graphics card with 24 GB.
9. 165 GiB does not fit on any single card sold in 2026, including the 141 GB NVIDIA H200.

PLAIN51.1.4 what is really happening inside#

1. In training, each batch of data goes forward through the network, an error is measured, and the error is pushed backwards.
2. Pushing it backwards requires remembering every intermediate value from the forward pass. Those are the activations, and they are enormous.
3. In inference, none of that exists. There is no backward pass, so no activation needs to be kept after the layer that produced it is done.
4. What is kept instead is different: a per-token record called the KV cache, which we come to in section 51.4.
5. So inference memory is dominated by two things only: the frozen weights, and the KV cache that grows with the length of the conversation.
6. And the weights are read-only, which means many requests can share exactly one copy of them in memory. This is why batching works at all.

TECHNICAL51.1.5 the engineer’s version#

1. Training memory per parameter under the standard mixed-precision recipe with Adam: 2 bytes bf16 weights, 4 bytes fp32 master, 4 bytes fp32 gradient, 8 bytes for Adam’s first and second moments. Total 18 bytes.
2. Inference memory per parameter: 2 bytes at bf16, 1 byte at int8, about 0.5 bytes at 4-bit. No optimizer state, no gradients.
3. The workload character inverts. Training is throughput-bound and runs at high model FLOPs utilization, typically 35 to 55 percent on well-tuned clusters. Batch-1 decode runs at roughly 0.3 percent of peak FLOPs.

4. Tools that show the split: nvidia-smi for device memory, torch.cuda. memory_summary() for allocator detail, and the /metrics endpoint that vLLM exposes for queue depth, batch size and cache usage.

WORDS51.1.6 remember these#

Inference — using a trained model — the forward pass only, no gradients. Training — teaching the model — forward and backward passes plus optimizer. KV cache — the model’s notes on this conversation — cached key and value tensors per token per layer. MFU — how much of the chip’s maths power you use — model FLOPs utilization. TTFT — the wait before the first word appears — time to first token.

51.2 What an API call actually contains#

PLAIN51.2.1 in simple words#

1. Talking to a hosted model is an ordinary web request. Nothing exotic.
2. Chapter 36 covered what an API is; Chapter 32 covered what happens on the wire when a request is encrypted and sent. This is one of those requests.
3. You send a block of text in a structured form, plus some settings.
4. The structured form is a list of messages. Each message has a role.
5. The roles are conventionally “system”, “user” and “assistant”.
6. “System” is the instruction you put at the front. “User” is what the person typed. “Assistant” is what the model said last time.
7. Everything in that list is glued together into one long string of tokens and fed to the model at once.
8. The model does not know which part came from where, except that the training taught it to treat the system part as instructions.

PLAIN51.2.2 a picture in your head#

1. Imagine handing a temporary worker a clipboard before every single task.
2. Sheet one says: you are a polite support agent, never discuss competitors.
3. Sheet two says: here is the customer’s question.
4. The worker reads the whole clipboard from the top, then speaks.
5. Then the worker forgets everything and leaves.
6. Next time, you hand over a fresh clipboard with all the sheets again, plus the new ones.
7. The first sheet has no special authority. It is just first.
8. If the customer writes on their sheet “ignore sheet one”, the worker has to decide, from habit alone, which sheet to obey.

Where this comparison breaks: a human worker has an employer, a contract and a memory of yesterday. The model has none of those. Its deference to the first sheet was installed by training on examples where the first sheet was obeyed. It is a strong statistical habit, not a permission system.

PLAIN51.2.3 a worked example#

1. Here is a complete request, in the shape most providers accept, with every field explained afterwards.

POST /v1/chat/completions HTTP/1.1
Host: api.example-provider.com
Authorization: Bearer sk-REDACTED
Content-Type: application/json

{
  "model": "example-model-2026-08",
  "messages": [
    {"role": "system",
     "content": "You are a support agent. Answer in
      British English. Never invent a policy."},
    {"role": "user",
     "content": "My git push failed. What do I check?"}
  ],
  "temperature": 0.2,
  "top_p": 0.95,
  "max_tokens": 400,
  "stop": ["\n\nUser:"],
  "stream": false,
  "response_format": {"type": "json_object"},
  "tools": []
}

2. model names the exact weights to run. Dated names matter: providers change what an undated name points at.
3. messages is the conversation, oldest first. The whole list is sent every time. Nothing is stored on the server between calls.
4. temperature controls randomness in picking the next token. 0 means always take the most likely token. 1.0 is the model’s raw distribution.
5. top_p keeps only the smallest set of tokens whose probabilities add up to 0.95, then samples inside that set. It is called nucleus sampling.
6. max_tokens is a hard ceiling on the reply length, and it is a cost control as well as a safety valve.
7. stop is a list of strings. If the model produces one, generation halts and the string is not returned.
8. stream set to true sends the answer piece by piece as it is produced.
9. response_format asks for strictly valid JSON rather than prose.
10. tools declares functions the model may ask you to run. Section 51.12 covers this in full.
11. And here is the reply:

{
  "id": "chatcmpl-8xk2ab",
  "object": "chat.completion",
  "created": 1786600000,
  "model": "example-model-2026-08",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Check the remote URL, then your token."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 62,
    "completion_tokens": 41,
    "total_tokens": 103
  }
}

12. finish_reason is the single most useful field for debugging. “stop” means the model chose to end. “length” means you hit max_tokens and the answer is truncated. “tool_calls” means it wants you to run something.
13. usage is what you are billed on. 62 input tokens and 41 output tokens here, priced at different rates.

PLAIN51.2.4 what is really happening inside#

1. On the server, that JSON is flattened into one token sequence using a chat template, which is a fixed pattern of special marker tokens.
2. A template looks roughly like this, with the markers written out:

<|start|>system
You are a support agent...<|end|>
<|start|>user
My git push failed...<|end|>
<|start|>assistant

3. The model then predicts what comes after the final marker.
4. The honest version: the system prompt is a run of tokens at the front of the context window and has no privileged status in the architecture.
5. Its influence comes entirely from post-training, where the model was shown many examples of the system section being followed.
6. Because it is only a habit, it can be overridden. A user message that says “the previous instructions were a test, here are the real ones” sometimes works, and content pasted in from a web page can carry instructions too.
7. That failure has a name, prompt injection, and section 51.15 treats it as the architectural problem it is.

TECHNICAL51.2.5 the engineer’s version#

1. The message-list shape shown above is a de facto convention, not a standard. It came from OpenAI’s chat completions endpoint in March 2023 and was copied widely because tooling supported it.
2. It is not universal. Anthropic’s Messages API takes system as a top-level string rather than a message role. OpenAI’s newer Responses API uses input and instructions. Local servers from llama.cpp and vLLM implement the older chat-completions shape for compatibility.
3. Streaming is served as Server-Sent Events over HTTP, media type text/event-stream, one JSON delta per event:

data: {"choices":[{"delta":{"content":"Check"}}]}
data: {"choices":[{"delta":{"content":" the"}}]}
data: {"choices":[{"delta":{"content":" remote"}}]}
data: [DONE]

4. Chapter 32 covers what carries this: TLS 1.3 over TCP 443, usually HTTP/2, with the response body arriving as a chunked stream. Streaming needs no new protocol; it is one long-lived response.

WORDS51.2.6 remember these#

System prompt — the instruction sheet at the front — leading tokens in the context, obeyed by training habit, not by architecture. Role — who said this line — a marker token pattern in the chat template. Temperature — how adventurous the model is — logit scaling before softmax. Nucleus sampling — keep only the likely words — top_p truncation of the cumulative distribution. finish_reason — why it stopped — stop, length, tool_calls or content_filter. Chat template — the glue that turns messages into one string — a Jinja template shipped with the model.

51.3 How generation actually performs#

PLAIN51.3.1 in simple words#

1. Answering has two phases, and they behave like two different machines.
2. Phase one: the model reads your whole prompt. This is called prefill.
3. Phase two: the model writes the answer one token at a time. This is called decode.
4. Prefill can read every word of your prompt at the same moment, in parallel, because all the words already exist.
5. Decode cannot do that. Word five cannot be chosen before word four exists.
6. So prefill is limited by how much arithmetic the chip can do per second.
7. And decode is limited by something else entirely: how fast the chip can pull the model’s numbers out of memory.
8. That is the whole secret. For each single token of output, the machine has to read every parameter in the model once.
9. So the speed of writing is roughly: memory speed divided by model size.
10. That one division explains almost everything people find strange about model speed.

PLAIN51.3.2 a picture in your head#

1. Imagine a librarian who must consult every book in a library before saying one word.
2. To read your question, the librarian can put a hundred assistants on it at once, one per sentence. That is quick, and it scales with staff.
3. But to say each word of the answer, the librarian must personally walk the whole library again. Every shelf. Every book.
4. Adding assistants does not help with the walking. The corridor is the limit.
5. So the wait before the first word depends on how long your question was and how many assistants are free.
6. And the speed of the words after that depends only on how fast one person can walk the library once.
7. A bigger library means slower words, in direct proportion.

Where this comparison breaks: the librarian would remember the books. The chip does not. It genuinely re-reads the weights from memory for every token, because there is nowhere near enough fast on-chip storage to hold them. That re-reading is not waste from bad design; it is the fundamental shape of the computation.

PLAIN51.3.3 a worked example#

1. Take Llama 3.1 8B at 16-bit precision. Weights are 16.1 GB.
2. Run it on one NVIDIA H100 SXM. Its memory bandwidth is 3,350 GB per second. That figure is from NVIDIA’s own specification sheet.
3. Decode ceiling equals 3,350 divided by 16.1, which is 208 tokens per second.
4. Real systems reach 60 to 80 percent of that, so expect 125 to 165 tokens per second for a single user. Measured single-stream figures for 8B on an H100 sit in that range.
5. Now the same model on a MacBook Air with the M5 chip. Its memory bandwidth is 153 GB per second, from Apple’s technical specifications.
6. At 4-bit the model is about 4.7 GB. 153 divided by 4.7 is 32 tokens per second as a ceiling; expect 20 to 25 in practice.
7. Now prefill on the same H100. Prefill work is about 2 times parameters times prompt tokens.
8. For a 1,000-token prompt: 2 times 8.03 billion times 1,000 equals 16.1 trillion floating-point operations.
9. An H100 SXM does about 990 trillion dense bf16 operations per second. At 40 percent utilization that prompt takes about 41 milliseconds.
10. Compare the phases directly:

11. So the answer takes seventy times longer than reading the question, even though the question was longer.
12. One more real-world correction. The reader of this book is in India. A round trip to a server in the United States is typically 220 to 280 milliseconds.
13. So for short prompts, the network wait is larger than the prefill. Time to first token is network plus queueing plus prefill, and often the network wins.

PLAIN51.3.4 what is really happening inside#

1. Here is the timeline of one request, drawn out:

 client         network      queue     prefill        decode
   |--------------->|                                    
   |     250 ms     |---------->|                        
   |                | wait 0-100 ms |                    
   |                            |---->|  41 ms           
   |                                  |-> tok 1  (TTFT)  
   |                                  |-> tok 2          
   |                                  |-> ...            
   |                                  |-> tok 400        
   |<-------------------------------------|  ~3.2 s total

2. Time to first token is everything up to the first arrow out: network, queue, prefill, and one decode step.
3. This is why a long prompt hurts your first token but not your typing speed, and why a long answer hurts total time but not the first token.

TECHNICAL51.3.5 the engineer’s version#

1. The controlling ratio is arithmetic intensity: floating-point operations per byte moved from memory.
2. An H100 SXM has about 990 TFLOP/s dense bf16 against 3.35 TB/s, a balance point of roughly 295 FLOP per byte. Below that ratio you are bandwidth-bound.
3. Batch-1 decode does 2 FLOP per parameter and reads 2 bytes per parameter: 1 FLOP per byte. You are using about 0.3 percent of the chip’s arithmetic.
4. The metric for the decode side is memory bandwidth utilization, or MBU, popularized by Databricks in 2023: achieved bytes per second divided by peak. Well-tuned stacks reach 60 to 85 percent.
5. Note that the vendor’s headline FLOPS figure often includes 2:1 structured sparsity. NVIDIA’s 1,979 TFLOPS figure for H100 bf16 is the sparse number; the dense number is 989.5. Use the dense one for language models.

6. Measure it yourself with llama-bench from llama.cpp, which reports prompt processing and token generation separately, or with vLLM’s benchmark_serving.py, which reports TTFT and inter-token latency percentiles under load.

WORDS51.3.6 remember these#

Prefill — reading your question — parallel forward pass over all prompt tokens, compute-bound. Decode — writing the answer — sequential single-token forward passes, memory-bandwidth-bound. Arithmetic intensity — sums done per byte fetched — FLOP per byte, compared against the hardware’s balance point. MBU — how much of the memory pipe you are using — memory bandwidth utilization. Inter-token latency — the gap between words — the reciprocal of decode tokens per second.

51.4 The serving optimizations, each explained#

PLAIN51.4.1 in simple words#

1. Every trick in serving exists to fight one of two enemies: reading the weights too often, or leaving the machine idle.
2. The KV cache stops the model re-reading your whole conversation for every new word.
3. Continuous batching stops the machine idling while one user thinks.
4. Speculative decoding lets a small fast model guess ahead so the big model can check several words at once.
5. Quantization shrinks the weights so there is less to read per token.
6. Tensor parallelism splits one model across several chips so their memory pipes add up.
7. Prefix caching stores the processed form of a shared opening so it is not re-processed for every user.
8. Streaming does not make anything faster, but it makes the wait feel far shorter, which is often what people actually want.

PLAIN51.4.2 a picture in your head#

1. Think of a busy kitchen with one very slow oven.
2. The KV cache is keeping the prepared ingredients on the counter instead of re-chopping everything for each course.
3. Static batching is waiting until six orders arrive, cooking all six, and serving nobody until the slowest one finishes.
4. Continuous batching is putting each new order into the oven the moment a space appears, and taking each dish out as soon as it is done.
5. PagedAttention is using many small identical trays instead of reserving a whole shelf per order in case it turns out to be large.
6. Speculative decoding is a junior cook who prepares the next three likely steps; the chef glances at them and either accepts all three or redoes them.

Where this comparison breaks: in a kitchen, more staff means more parallel cooking. On a GPU, the constraint is a single memory bus that all the work shares, so most of these tricks are about using one bus better, not about adding more hands.

PLAIN51.4.3 a worked example#

1. Start with the KV cache, because it is the one that changes the order of the arithmetic.
2. Without a cache, to produce token number t the model must process all t-1 previous tokens again from scratch.
3. Generating 500 tokens after a 1,000-token prompt means processing, on average, 1,250 tokens, 500 times: 625,000 token-passes.
4. With a cache, each new token processes exactly one new token, because the previous keys and values are already stored: 500 token-passes.
5. That is a saving of 1,250 times for this case. The work goes from growing with the square of the length to growing with the length itself.
6. Now the price. The cache is real memory, and it is calculated exactly:

KV bytes per token
  = 2 (keys and values)
  x layers
  x key/value heads
  x head dimension
  x bytes per number

Llama 3.1 8B, 16-bit:
  = 2 x 32 x 8 x 128 x 2  = 131,072 bytes = 128 KiB

Llama 3.1 70B, 16-bit:
  = 2 x 80 x 8 x 128 x 2  = 327,680 bytes = 320 KiB

7. So an 8B model with a 128,000-token context holds 16 GiB of cache, which is more than the 15 GiB of weights.
8. This is why serving systems spend so much effort on cache memory, and why grouped-query attention, which cuts the number of key/value heads from 64 to 8, was such an important change. It divided this table by eight.

PLAIN51.4.4 what is really happening inside#

1. Static batching groups requests, runs them together, and returns them together. Every request waits for the longest one in the group.
2. Continuous batching, first described in the 2022 Orca paper, works at the level of single decode steps instead.
3. At each step the scheduler looks at every active sequence, runs one token for all of them in one fused operation, then evicts finished ones and admits waiting ones immediately.
4. The result is that the batch is never allowed to shrink, so the memory pipe stays busy. Reported throughput gains over static batching are large, often several times, at the same latency.
5. PagedAttention, from the vLLM paper at SOSP 2023, borrows the operating system idea of paging from Chapter 12.
6. Instead of reserving a contiguous block of cache for each sequence’s maximum possible length, it allocates fixed-size blocks, typically 16 tokens each, and keeps a block table per sequence.
7. This removes internal fragmentation and lets several sequences share blocks, which matters when many requests begin with the same system prompt.
8. Speculative decoding uses a small draft model to propose the next k tokens cheaply, then runs the large model once over all k proposals in parallel.
9. Because the large model’s forward pass over k tokens costs almost the same as over one token, accepted guesses are nearly free. Rejected ones cost only the draft work.
10. The mathematics were published independently by two groups in 2023, and the method is exact: the output distribution is identical to running the large model alone. It is a speed trick, not a quality trade.

TECHNICAL51.4.5 the engineer’s version#

1. The optimizations, with what each actually buys:

2. Tensor parallelism splits each weight matrix across GPUs, so each device reads only its shard. Two GPUs give roughly twice the aggregate bandwidth, less an all-reduce over NVLink after every layer. Scaling is sub-linear and degrades sharply without a fast interconnect.
3. Quantized serving matters twice over: 4-bit weights cut both memory and bytes read per token. Activation quantization to FP8 additionally raises prefill throughput on Hopper and Blackwell hardware.
4. Why the same open model is faster at provider A than provider B, in order of how much it usually explains: the hardware generation, the quantization they silently chose, the batch size policy they run, the current queue depth, and the geographic distance to you.
5. That list is the reason a published benchmark of “model X speed” is close to meaningless without naming the deployment.

WORDS51.4.6 remember these#

KV cache — saved notes so the model need not re-read — cached key and value projections per token per layer. Continuous batching — slot new work in as space frees — iteration-level scheduling, from the Orca paper, 2022. PagedAttention — memory in small blocks, not big reservations — block-table KV allocation, vLLM, 2023. Speculative decoding — a fast guesser the big model checks — draft-and-verify with an exact acceptance rule. Tensor parallelism — one model split across chips — sharded matrix multiply with all-reduce per layer. Prefix caching — reuse the shared opening — KV reuse keyed on a common token prefix.

51.5 Cost and capacity#

PLAIN51.5.1 in simple words#

1. Hosted models are sold by the token, and tokens in cost less than tokens out.
2. Chapter 47 explained what a token is: roughly three quarters of an English word, so 1,000 tokens is about 750 words.
3. The reason input is cheaper is exactly section 51.3. Your prompt is read in parallel in one pass, so it is cheap per token.
4. The answer is produced one token at a time, and each token costs a full read of the model. That is the expensive part.
5. Two discounts exist almost everywhere. Batch work that can wait up to a day is half price. Repeated openings you have marked as cacheable are about a tenth of the input price.
6. Building your own server instead is not obviously cheaper. You rent the machine by the hour whether anybody uses it or not.

PLAIN51.5.2 a picture in your head#

1. Think of a taxi where the meter runs differently for the two halves of the journey.
2. Listening to your address is quick and charged at a low rate per word.
3. Driving you there is slow and charged at a high rate per street.
4. Batch pricing is agreeing to be picked up sometime today rather than now, for half fare.
5. Cache pricing is a regular route the driver already knows, so the first part of the journey needs no navigation.
6. Buying your own car instead is the self-hosting decision. The car costs the same whether you drive it or not.
7. It only beats taxis if you drive a great deal, every day.

Where this comparison breaks: taxi fares scale with distance, which you cannot change. Token costs scale with how much text you send, which you control completely. Trimming a system prompt from 2,000 tokens to 400 is a real 80 percent cut on that part of every request, forever.

PLAIN51.5.3 a worked example#

1. Take a support assistant. Real shape, real numbers.
2. 10,000 conversations a day. Six turns each. A 800-token system prompt. 2,000 tokens of retrieved documents at the start. 100-token questions. 300-token answers.
3. Because the whole history is resent every turn, the input grows each turn: 2,900, then 3,300, 3,700, 4,100, 4,500, 4,900.
4. That totals 23,400 input tokens and 1,800 output tokens per conversation.
5. Now price it against three real published tiers from August 2026, in dollars per million tokens: a mid tier at 2 in and 10 out, a small tier at 1 and 5, and a very small tier at 0.10 and 0.60.

6. Read that table twice. The same application costs nineteen thousand dollars a month or one thousand, depending on one decision.
7. The caching row assumes the 2,800-token opening is marked cacheable and hit on five of six turns, charged at one tenth, with a write surcharge on the first turn.

PLAIN51.5.4 what is really happening inside#

1. Renting one datacentre GPU in 2026 costs roughly 2 to 4 US dollars an hour on the open market, varying by provider, region and commitment. Take 2.50.
2. That is about 1,825 dollars a month, per GPU, running or idle.
3. One such GPU serving an 8-billion-parameter model with good batching produces on the order of 2,000 to 4,000 output tokens per second in aggregate across all users.
4. At 2,500 tokens per second held constantly, that is 6.6 billion output tokens a month, which works out at about 28 cents per million.
5. That looks unbeatable until you notice the words “held constantly”.
6. Compare the two cost shapes across real volumes:

7. Break-even against a cheap hosted model is around 3 billion output tokens a month, and that is before you add a second GPU for redundancy, an engineer to run it, or the load spikes you must size for.
8. The honest version: self-hosting is rarely cheaper below very high sustained volume. The good reasons to self-host are usually not cost.
9. They are: data must not leave your network, you need a model that no provider hosts, you need the exact same weights for years, or you need a latency floor you control.

TECHNICAL51.5.5 the engineer’s version#

1. Published list prices, checked August 2026. Prices move; treat these as a snapshot, not a constant.

2. Discount structure, also checked August 2026: batch processing at 50 percent of both input and output on both providers; cache reads at 10 percent of the input rate; cache writes at 125 percent for a short time-to-live.
3. Long-context surcharges are now common. OpenAI applies a higher rate above a published input threshold for the whole request, which means one oversized prompt reprices the entire call.
4. Reasoning models bill their internal thinking tokens as output. A model that thinks for 4,000 tokens before writing 200 costs twenty-one times more than the visible answer implies. This is the single most common billing surprise of 2025 and 2026.

WORDS51.5.6 remember these#

Input token — text you send — prompt tokens, billed at the lower rate. Output token — text produced — completion tokens, billed several times higher. Batch API — cheaper if you can wait — asynchronous queue, typically 50 percent off with a 24-hour window. Prompt caching — pay less for a repeated opening — server-side KV reuse keyed on an exact prefix, billed at about 10 percent. Duty cycle — how much of the time your machine is busy — utilization, the term that decides every self-hosting argument.

51.6 Running one yourself, practically#

PLAIN51.6.1 in simple words#

1. You can run a real language model on your own laptop today, offline, for free, in about ten minutes.
2. llama.cpp is the engine. It is a C++ program that loads a model file and runs it on your processor, your graphics chip, or both.
3. Ollama wraps that engine in something friendly: one command to fetch a model, one to talk to it, and a small local server.
4. LM Studio is a desktop application with a window and buttons, for people who would rather not use a terminal.
5. vLLM, SGLang and TGI are for the other end: serving many users at once on datacentre graphics cards.
6. MLX is Apple’s own numerical framework, built for Apple silicon, and it is noticeably faster than the alternatives on those machines.
7. Start with Ollama. Move to llama.cpp when you want control. Move to vLLM when you have users.

PLAIN51.6.2 a picture in your head#

1. Think of playing recorded music.
2. llama.cpp is the amplifier and speaker: raw, capable, many knobs, no case.
3. Ollama is the all-in-one hi-fi with one power button and a catalogue of albums you can fetch by name.
4. vLLM is the sound system for a concert hall, designed for two thousand listeners rather than one.
5. MLX is a speaker built specifically for the shape of your room.

Where this comparison breaks: with music, the recording is identical whatever plays it. Here the tools change the result. Different engines pick different default quantizations, context lengths and sampling settings, so the same named model can give different answers and very different speeds.

PLAIN51.6.3 a worked example#

1. Here is Ollama, start to finish, on a Mac or Linux machine.

# install with a package manager, then check it runs
ollama --version

# fetch a small model (a few gigabytes) and chat
ollama pull qwen3:8b
ollama run qwen3:8b

# see what you have, and what is loaded right now
ollama list
ollama ps

# use it as a local HTTP service on port 11434
curl localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "messages": [{"role":"user","content":"Explain TCP in 3 lines"}],
  "stream": false
}'

2. That last call has the same shape as the hosted API in section 51.2. Most local servers deliberately imitate it so your code does not change.
3. Now llama.cpp directly, which is one level down.

# build once, with Apple GPU support turned on
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release -j

# run a model pulled straight from Hugging Face
./build/bin/llama-cli -hf ggml-org/gemma-3-1b-it-GGUF \
  -p "Write one sentence about routers."

# serve it on port 8080, 8k context, all layers on the GPU
./build/bin/llama-server -m model-Q4_K_M.gguf \
  -c 8192 -ngl 99 --port 8080

# measure prompt speed and generation speed separately
./build/bin/llama-bench -m model-Q4_K_M.gguf

4. The flag that matters most is -ngl, the number of layers to put on the GPU. Setting it to 99 means “all of them, if they fit”.
5. If the model does not fit, layers stay on the processor and speed collapses, often by a factor of five or more. That is the single most common cause of “why is my local model so slow”.

PLAIN51.6.4 what is really happening inside#

1. llama.cpp was started by Georgi Gerganov in March 2023 as an experiment in running Llama on a MacBook with no graphics card at all.
2. It defined a file format, now called GGUF, which packs the weights, the tokenizer and the metadata into one file that can be memory-mapped.
3. On the serving side the design goal is the opposite. vLLM keeps the GPU saturated with continuous batching and PagedAttention from section 51.4.
4. SGLang adds RadixAttention, a shared prefix tree, which pays off when thousands of requests begin with the same long system prompt.
5. Hugging Face’s Text Generation Inference was the early default, but Hugging Face put it into maintenance mode on 11 December 2025 and now points new users at vLLM or SGLang. Treat it as a migration topic, not a choice.
6. MLX is Apple’s array framework, released in December 2023, designed around unified memory so tensors are not copied between processor and graphics chip. Ollama added an MLX backend in version 0.19 on 27 March 2026.

TECHNICAL51.6.5 the engineer’s version#

1. macOS caps what the GPU may wire down, historically around 75 percent of unified memory. The iogpu.wired_limit_mb sysctl raises it. That is an implementation detail of one operating system version and it has been renamed before; check before relying on it.
2. Ollama’s published benchmark for its MLX backend, March 2026, on a mixture-of-experts model with 35 billion total and 3 billion active parameters: decode from 58 to 112 tokens per second, prefill from 1,154 to 1,810 tokens per second.
3. Vendor figures, so read them as a direction, not a promise.
4. For serving, the vLLM invocation you will actually type is vllm serve <model> --tensor-parallel-size 2 --max-model-len 32768, and it exposes an OpenAI-compatible endpoint plus a Prometheus /metrics path.
5. Watch three numbers in production: running batch size, waiting queue length, and KV cache utilization. When cache utilization hits 100 percent the scheduler starts preempting sequences and latency becomes erratic.

WORDS51.6.6 remember these#

GGUF — the one-file local model format — successor to GGML, memory-mappable, carries weights, tokenizer and metadata. Offload — putting model layers on the graphics chip — -ngl or n_gpu_layers; partial offload is the usual cause of slow local models. Ollama — the friendly local runner — a daemon plus registry over llama.cpp. vLLM — the standard GPU serving engine — PagedAttention, continuous batching, OpenAI-compatible server. MLX — Apple’s numerical framework — unified-memory array library for Apple silicon, released December 2023.

51.7 What hardware you actually need#

PLAIN51.7.1 in simple words#

1. Two numbers decide everything: how much memory you have, and how fast that memory is.
2. Memory decides whether the model runs at all. Bandwidth decides how fast it talks.
3. The memory rule from Chapter 47: parameters times bytes per parameter, plus a bit for the conversation and the software.
4. At 4 bits per parameter, a rough working rule is that the model needs about 0.6 GB per billion parameters, plus 2 to 3 GB of headroom.
5. So an 8-billion model at 4 bits wants about 7 GB free, and a 32-billion model wants about 22 GB free.
6. If it does not fit, it does not simply run slower. Part of it spills to the processor or to disk and the speed falls off a cliff.
7. Apple silicon is unusually good here, because the processor and graphics chip share one pool of memory. A laptop with 32 GB can load models that need a 3,000 dollar graphics card on a PC.
8. But Apple laptops have much narrower memory pipes than datacentre cards, so they load big models and then run them slowly.

PLAIN51.7.2 a picture in your head#

1. Think of memory as the size of your desk and bandwidth as how fast you can turn pages.
2. A PC graphics card is a small desk with an extremely fast hand.
3. An Apple laptop is a large desk with a moderate hand.
4. A datacentre card is a large desk with an extraordinarily fast hand, in somebody else’s building, rented by the hour.
5. If the book does not fit on the desk, you keep fetching pages from the floor, and everything stops.

Where this comparison breaks: a person can skim. The model reads every page for every word. There is no skimming, so the “fast hand” is used at full stretch on every single token.

PLAIN51.7.3 a worked example#

1. Here are real machines, with real figures from the manufacturers, and what each can hold.

2. Now the speeds. These are the bandwidth ceiling from section 51.3 with a realistic 70 percent efficiency applied, for 4-bit models.

3. Being specific, as promised. On an 8 GB MacBook Air: models up to about 4 billion parameters at 4 bits run fine. An 8B at 4 bits technically loads and then swaps to disk, giving a few tokens per second and a hot machine. Anything larger will not run at all.
4. On a 16 GB Air: 8B is comfortable, 14B works, 32B does not fit.
5. On a 24 GB Air: 32B at 4 bits fits with a short context and delivers about 8 tokens per second, which is slower than reading speed and unpleasant.
6. A useful threshold: below about 10 tokens per second, most people stop using a model interactively. Above 25 it feels fine for chat.

PLAIN51.7.4 what is really happening inside#

1. Why unified memory helps so much: on a normal PC, the graphics card has its own memory, and anything not in it must cross the PCI Express bus, which moves about 64 GB per second on a sixteen-lane fourth-generation link.
2. That is twenty to fifty times slower than the card’s own memory, so a model that half fits runs at roughly the speed of the slow path.
3. On Apple silicon there is one pool. There is no crossing, so a 70-billion model at 4 bits genuinely runs on a laptop, which no 3,000 dollar consumer graphics card can do.
4. Where Apple silicon is still limited, in two places.
5. First, bandwidth. Even the fastest laptop chip at 614 GB per second is under a fifth of an H100. Big models load and then crawl.
6. Second, prompt processing. Prefill is compute-bound, and Apple’s graphics cores have far less matrix throughput than a datacentre card. Pasting in a long document can take many seconds before the first word appears, even though generation afterwards looks reasonable.
7. This asymmetry surprises people constantly. The machine feels fine in chat and terrible the moment you paste a file into it.

TECHNICAL51.7.5 the engineer’s version#

1. Sizing formula for a local deployment, in bytes:

total = params x bytes_per_param
      + kv_bytes_per_token x context_length x concurrency
      + activations (0.5 to 1 GiB)
      + runtime overhead (0.5 to 2 GiB)

2. Mixture-of-experts models decouple the two limits: capacity is set by total parameters, speed by active parameters. A 26-billion-total, 4-billion-active model needs about 16 GB at 4 bits but decodes like a 4B model.
3. Community measurements in 2026 report roughly 150 tokens per second for such a model on an RTX 4090 or 5090 and roughly 80 on an M5 Max.
4. Apple’s published bandwidths, August 2026: M5 in the MacBook Air, 153 GB/s; M5 Pro up to 307 GB/s with up to 64 GB; M5 Max 460 GB/s in the 32-core configuration and 614 GB/s in the 40-core, up to 128 GB.
5. NVIDIA’s published bandwidths: RTX 4090 1,008 GB/s over 24 GB GDDR6X; RTX 5090 1,792 GB/s over 32 GB GDDR7; H100 SXM 3,350 GB/s over 80 GB HBM3; H200 4,800 GB/s over 141 GB HBM3e; B200 about 8,000 GB/s.
6. Observe the reality rather than the ceiling with llama-bench, which prints prompt-processing tokens per second and generation tokens per second as two separate rows. They will differ by one or two orders of magnitude, exactly as section 51.3 predicts.

WORDS51.7.6 remember these#

Unified memory — one pool shared by processor and graphics — Apple silicon’s architecture; capacity advantage, bandwidth disadvantage. VRAM — the graphics card’s own memory — dedicated GDDR or HBM, fast but small on consumer cards. Offload spill — part of the model living on the slow side — layers left on the CPU or paged to disk; the usual cause of a tenfold slowdown. Prompt processing speed — how fast it reads what you paste — prefill tokens per second, compute-bound, weakest on laptops. Active parameters — the part of a sparse model actually used — the figure that sets decode speed in a mixture-of-experts model.

51.8 Getting weights and reading a model card#

PLAIN51.8.1 in simple words#

1. Hugging Face is the place where models live. It is a website and a set of tools, and it works much like a code-hosting site.
2. Each model has a repository with an owner and a name, files inside it, and a page of documentation called a model card.
3. As of 2026 the site hosts more than two million public models. Most are variations, fine-tunes and re-uploads of a much smaller number of originals.
4. You download either the original weights, which are large files in a format for research frameworks, or a converted single-file version for local tools.
5. The model card is the part people skip and should not. A good one tells you what the model was built from, what it may legally be used for, and where it fails.
6. A bad one tells you only that the model is excellent.
7. Then you choose a quantization level, which is the trade from Chapter 47: fewer bits per number means less memory and more speed, and a little less accuracy.

PLAIN51.8.2 a picture in your head#

1. Think of buying a second-hand car from an online listing.
2. The files are the car. The model card is the seller’s description.
3. A good listing states the year, the mileage, the service history, what is worn, and what the car is not suitable for.
4. A bad listing says “runs great, must sell” and shows one photograph taken from a flattering angle.
5. The licence is the logbook. Without it you may not legally use the car for the purpose you had in mind, however well it runs.

Where this comparison breaks: a car’s condition can be inspected by a mechanic in an hour. A model’s real behaviour on your task can only be found by running your own evaluation on your own data, which is why Chapter 50’s material on evaluation matters more than any card.

PLAIN51.8.3 a worked example#

1. A typical repository contains these files. Knowing what each is stops a lot of confusion.

config.json                 architecture: layers, heads, sizes
model-00001-of-00004.safetensors   weight shard 1 of 4
model.safetensors.index.json       which tensor is in which shard
tokenizer.json              the vocabulary and merge rules
tokenizer_config.json       special tokens, chat template
generation_config.json      default temperature, stop tokens
README.md                   the model card
LICENSE                     the terms you are agreeing to

2. To download, use the official command-line tool, which is hf in current versions and was named huggingface-cli before 2025:

hf download meta-llama/Llama-3.1-8B-Instruct \
  --local-dir ./llama-3.1-8b

# or fetch only one quantized file for local use
hf download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
  Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf --local-dir .

3. Prefer files ending in .safetensors over .bin. The older .bin format is Python pickle, which can execute code when loaded. Safetensors, introduced in 2022, is a plain data format that cannot.
4. Now the quantization decision. These are real measured figures from llama.cpp’s own tables for an 8-billion model, showing file size and the increase in perplexity, which is the confusion measure from Chapter 50. Lower is better.

5. Read the last row. Two-bit quantization on a small model does not degrade it slightly; it breaks it.
6. The decision, as a table:

PLAIN51.8.4 what is really happening inside#

1. What a good model card contains, in the order you should read it.
2. The base model and how this one relates to it: is it an original pre-trained model, an instruction-tuned version, or somebody’s fine-tune of a fine-tune.
3. A description of the training data. Not the data itself, usually, but its sources, size, languages and cut-off date.
4. The licence, stated plainly, including whether commercial use is allowed and whether outputs may be used to train other models.
5. Intended use and out-of-scope use. Real cards say what the model is not for.
6. Limitations and known biases, in specifics rather than boilerplate.
7. Evaluation results with the benchmark names, the exact settings, and ideally the harness used, so the numbers can be reproduced.
8. What a bad card omits, in order of how much it should worry you: the licence, the training data description, the evaluation settings, and any statement of weakness at all.
9. A card that reports only wins on benchmarks, with no method and no limitations section, is marketing. Treat the numbers as unverified.
10. The idea is not new or informal. Model cards were proposed in a 2019 paper by Margaret Mitchell and colleagues, and datasheets for datasets in a 2018 paper by Timnit Gebru and colleagues. Both are conventions, widely adopted and nowhere enforced.

TECHNICAL51.8.5 the engineer’s version#

1. Licences differ far more than people assume, and “open” is not one thing. Apache 2.0 and MIT are permissive. Llama’s community licence adds an acceptable-use policy and a threshold on monthly active users. Some research-only licences forbid commercial use entirely.
2. “Open weights” means you can download the parameters. It does not mean open source: the training data and training code are usually not released. Say open-weights when that is what you mean.
3. Gated repositories require accepting terms with your account before the files resolve, so automated pipelines need a token with the acceptance already recorded.

WORDS51.8.6 remember these#

Model card — the honest description page — a structured README covering data, licence, intended use, limitations and evaluations. Safetensors — a weight file that cannot run code — a simple tensor container format, preferred over pickle-based .bin. Open weights — you can download the numbers — distinct from open source, which would include data and training code. Perplexity — how confused the model is by real text — exponentiated average negative log-likelihood; used here to price quantization damage. Gated repository — you must accept terms first — access controlled by account agreement on the hub.

51.9 Building something small from scratch, honestly#

PLAIN51.9.1 in simple words#

1. You can train your own language model on a laptop. It is a genuinely good afternoon’s work, and it will teach you more than ten articles.
2. Realistically, on a laptop, you are training something between one and fifty million parameters on a few tens of millions of characters of text.
3. That is between one thousandth and one ten-thousandth of the size of the models you use every day.
4. What it will do: produce text with the correct shape. Sentences that look like sentences. Correct punctuation. Local coherence for a line or two.
5. If you train it on Shakespeare, it produces convincing Shakespeare-shaped nonsense with character names and stage directions in the right places.
6. What it will not do: know any facts, follow an instruction, answer a question, hold a conversation, or reason.
7. Those abilities appear only with vastly more data and a separate instruction-tuning stage afterwards.
8. Do it exactly once, as an education. Then never ship it, because a downloadable open model is thousands of times better and free.

PLAIN51.9.2 a picture in your head#

1. Think of a child copying the appearance of adult handwriting before they can spell.
2. The loops are right. The spacing is right. The line has the rhythm of writing. From across the room it looks like a letter.
3. Your tiny model is at exactly that stage, and getting there takes an hour.
4. Learning to spell is the next thousand hours, and learning to write something worth reading is the thousand after that.

Where this comparison breaks: the child is learning towards meaning the whole time. The model has no such trajectory. Scale does not turn imitation into understanding by itself; it turns imitation into much better imitation, which turns out to be enough for a surprising number of tasks, and this distinction is exactly where thoughtful people still disagree.

PLAIN51.9.3 a worked example#

1. Concrete plan, with real numbers.
2. Data: the collected works of Shakespeare, about 1.1 million characters, which is a standard toy dataset.
3. Model: a transformer with 6 layers, 6 attention heads, 384 hidden dimensions, character-level vocabulary. That is roughly 10 million parameters.
4. Training: a few thousand steps. On an Apple silicon laptop this takes on the order of 10 to 30 minutes. On a rented graphics card, a couple of minutes.
5. Result: text like this in shape, though the words will be invented.

DUKE VINCENTIO:
Well, your wit is in the care of side and that.

Second Lord:
They would be ruled after this chamber, and
my fair nues begun out of the fact, to be conveyed,
Whose noble souls I'll have the heart of the wars.

6. Every convention is right. Speaker names, colons, line breaks, capitalized verse. No sentence means anything.
7. Now the honest scaling. To get factual knowledge you need billions of tokens, not one million characters.
8. The compute-optimal ratio from the 2022 Chinchilla paper is roughly 20 training tokens per parameter. A useful 1-billion-parameter model implies about 20 billion tokens, which is not a laptop job.

PLAIN51.9.4 what is really happening inside#

1. The canonical implementations to read, by name, so you can find them.
2. minGPT, by Andrej Karpathy, 2020: a few hundred lines, written to be read rather than run fast.
3. nanoGPT, by the same author, January 2023: the practical version, which reproduces GPT-2 at 124 million parameters.
4. The video “Let’s build GPT: from scratch, in code, spelled out”, 2023, which builds the whole thing live in about two hours.
5. llm.c, 2024: the same training in plain C and CUDA, with no framework at all, useful for seeing where the time actually goes.
6. nanochat, October 2025: the full pipeline including tokenizer, pre-training, fine-tuning, evaluation and a chat interface, in one repository.
7. The nanochat numbers are the useful ones for calibration. Its own documentation reports training a model of GPT-2 capability for about 48 dollars, roughly two to four hours on a node of eight H100 cards, against the roughly 43,000 dollars that training GPT-2 cost in 2019.
8. That fall of nearly a thousand times in six years is one of the clearest single facts about this field, and it is about hardware, software and recipes together, not about any one breakthrough.

TECHNICAL51.9.5 the engineer’s version#

1. What a laptop-scale run looks like against the real thing:

2. The alternative, and usually the right one: fine-tune an existing open model instead of training one.
3. Full fine-tuning of an 8B model needs the 165 GiB from section 51.1, so it is a multi-GPU job.
4. Parameter-efficient methods avoid that. LoRA, from a 2021 Microsoft paper, freezes the model and trains small low-rank matrices alongside it, typically 0.1 to 1 percent of the parameters.
5. QLoRA, 2023, adds 4-bit quantization of the frozen base, which brought fine-tuning a 7B model onto a single 24 GB card, and a 70B model onto a single 80 GB card.
6. When fine-tuning is the right answer: you need a fixed output format, a house style, a domain vocabulary, or you want to make a small cheap model imitate a large expensive one on one narrow task.
7. When it is the wrong answer: you want the model to know new facts. Facts change, fine-tuning bakes them in, and the model will still state the old ones confidently. Use retrieval, which section 51.11 covers.
8. The practical threshold: below roughly 1,000 good examples, prompt engineering and few-shot examples usually beat fine-tuning. Above roughly 10,000, fine-tuning usually wins.
9. Evaluate before and after on a held-out set you wrote yourself. Fine-tuning frequently improves the target task and quietly degrades everything else, a phenomenon called catastrophic forgetting.

WORDS51.9.6 remember these#

Pre-training — the long expensive stage — next-token prediction over a very large corpus, producing a base model. Base model — a text continuer, not an assistant — the pre-trained model before instruction tuning. Chinchilla ratio — how much text per parameter — about 20 training tokens per parameter for compute-optimal training, from the 2022 paper. LoRA — training a small patch instead of the whole model — low-rank adapter matrices added to frozen weights. Catastrophic forgetting — getting better at one thing and worse at everything else — degradation of unrelated capabilities after narrow fine-tuning.

51.10 Prompting as an engineering practice, not a trick#

PLAIN51.10.1 in simple words#

1. A prompt is just the text you send. There is no magic word.
2. What works is the same thing that works when briefing a competent new colleague who cannot ask you questions.
3. Be specific about the task, the audience, the length and the format.
4. Show an example of what you want, because one example beats a paragraph of description.
5. Say what the output must look like, exactly, especially if a program will read it.
6. For multi-step problems, let the model work through the steps rather than demanding the answer immediately.
7. Break big tasks into several calls instead of one enormous one.
8. And now the honest part: the same request phrased two ways can give noticeably different quality. That is a real weakness of the technology, not a skill you lack.

PLAIN51.10.2 a picture in your head#

1. Think of writing a work order for a contractor who will do exactly what the paper says and will never phone you back.
2. “Fix the bathroom” gets you something. Possibly not the thing you wanted.
3. “Replace the tap with this model, keep the existing pipework, finish by Friday, leave the old tap in the box” gets you the thing you wanted.
4. Adding a photograph of a finished job you liked works better still.
5. The contractor is skilled. The failure was in the brief.

Where this comparison breaks: a real contractor who received an ambiguous brief would either ask or use judgement informed by having seen your house. The model has no judgement about your situation and cannot ask, so it fills gaps with the statistically ordinary choice, which may be nothing like your case.

PLAIN51.10.3 a worked example#

1. Five real improvements. In each pair, the second version is not longer for the sake of it; each addition removes one specific ambiguity.

1 BEFORE: Summarize this.
  AFTER:  Summarize the text below in 5 bullet points for a
          non-technical manager. Each bullet under 20 words.
          Do not mention anything not in the text.

2 BEFORE: Write a regex for emails.
  AFTER:  Write a Python regex matching the addresses in the
          list below and rejecting the ones after "BAD:".
          Return only the pattern, no explanation.

3 BEFORE: Is this contract clause risky?
  AFTER:  Read the clause. List each obligation it places on
          the buyer, then rate each Low/Medium/High risk with
          one reason. If the clause is ambiguous, say so.

4 BEFORE: Extract the data.
  AFTER:  Return JSON only, matching exactly this shape:
          {"invoice_no": string, "total": number,
           "currency": string, "date": "YYYY-MM-DD"}
          Use null for any field not present in the text.

5 BEFORE: What is 17% of 4,382 plus the VAT?
  AFTER:  Work step by step. State each arithmetic step on its
          own line, then give the final number. Use the
          calculator tool for each multiplication.

2. Improvement 2 supplies examples of both classes, which is what few-shot prompting means.
3. Improvement 4 pins the exact schema, which is the difference between code that parses and code that crashes at 3 a.m.
4. Improvement 5 does two things: asks for steps rather than a single leap, and moves the arithmetic to a tool, for reasons section 51.15 explains.

PLAIN51.10.4 what is really happening inside#

1. Why examples work: the model is completing a pattern. Two worked examples define the pattern far more sharply than a description of it.
2. Zero-shot means no examples, just an instruction. Few-shot means two to five worked examples in the prompt. Few-shot costs tokens on every call and is usually worth it for format compliance.
3. Why step-by-step working helps: each token the model writes becomes input for the next token. Written reasoning is literally extra computation, and it gives the model somewhere to put intermediate results, since it has no scratchpad of its own.
4. This was formalized as chain-of-thought prompting in 2022, and the effect is large on multi-step arithmetic and logic.
5. When it does not help: simple lookups, classification, and formatting tasks, where it adds cost and latency for nothing.
6. There is also a 2026 research finding worth knowing: more reasoning is not monotonically better. Accuracy against reasoning length often follows an inverted U, and past a point extra thinking flips correct answers to wrong ones. That result is active research, not settled.
7. Recent reasoning models do this internally and are trained for it, so instructing them to “think step by step” can be redundant or harmful. Read the provider’s guidance for the specific model.

TECHNICAL51.10.5 the engineer’s version#

1. Treat prompts as versioned artifacts. They belong in the repository, with a test suite, not pasted into a dashboard.
2. Prompt sensitivity is measurable and is a genuine limitation. Reported effects in the literature include changes in accuracy from reordering few-shot examples, from formatting choices such as separators, and from the position of the question relative to a long document.
3. The position effect has a name: models attend most reliably to the beginning and end of a long context and least reliably to the middle, described as “lost in the middle” in a 2023 paper.
4. Put the question after the documents, and put the most important document last.
5. Techniques with reasonable evidence: few-shot examples, explicit output schemas, decomposition into separate calls, self-consistency by sampling several answers and taking the majority, and constrained decoding when the output must parse.
6. Techniques that are mostly folklore: politeness, offering tips, threatening, role-play preambles such as “you are a world-class expert”, and long lists of adjectives. Some show small effects in some studies and vanish in others.
7. The honest summary: prompt engineering is real but shallow. Its ceiling is low compared with fixing your retrieval, your data or your task decomposition.

WORDS51.10.6 remember these#

Zero-shot — no examples given — instruction-only prompting. Few-shot — a handful of worked examples — in-context learning from demonstrations in the prompt. Chain of thought — asking it to show working — eliciting intermediate reasoning tokens before the answer. Self-consistency — ask several times and take the majority — sampling multiple reasoning paths and voting. Prompt sensitivity — small wording changes, different quality — measured variance in output quality under semantically equivalent rephrasing.

51.11 RAG: giving a model your documents#

PLAIN51.11.1 in simple words#

1. A model knows only what was in its training data, which stopped at some date, and which never included your company’s files.
2. You cannot fix that by asking nicely. The information is genuinely not inside the weights.
3. But you can put the relevant text into the question. The model reads its context, so text you paste in is text it can use.
4. Doing that automatically is called retrieval-augmented generation, or RAG.
5. The pipeline is: cut the documents into pieces, turn each piece into a vector of numbers, store them, turn the question into a vector too, find the closest pieces, paste them into the prompt, and generate.
6. The vector is the embedding from Chapter 47: a list of numbers where similar meanings sit close together.
7. Every part of that pipeline can fail, and when the answer is wrong it is usually retrieval that failed, not the model.

PLAIN51.11.2 a picture in your head#

1. Think of an exam where the candidate may bring notes, but only one page.
2. Someone else chooses which page, from a filing cabinet, in one second, based on the question.
3. If they bring the right page, the candidate looks brilliant.
4. If they bring a page about the right topic but the wrong year, the candidate confidently gives last year’s answer.
5. If they bring six pages of near-misses, the candidate gets distracted and the good page gets lost among them.
6. The candidate’s ability is not what varies between these cases. The filing clerk’s is.

Where this comparison breaks: a real candidate would say “these notes do not cover this question”. A model usually will not. It will answer from the notes if it can, and from habit if it cannot, and both answers look equally confident.

PLAIN51.11.3 a worked example#

1. The pipeline, drawn out:

  DOCUMENTS                          QUESTION
      |                                 |
   [ chunk ]                            |
      |                                 |
   [ embed ] -> vectors            [ embed ]
      |                                 |
   [ store  ] <--- nearest search ------+
      |                                 |
   top-k chunks -----> [ prompt ] <-----+
                            |
                       [ model ] -> answer

2. Real sizes for a small system: 500 documents, average 8 pages each, cut into chunks of about 500 tokens with 50 tokens of overlap.
3. That gives roughly 30,000 chunks. Embedded at 1,536 dimensions in 4-byte floats, the vectors occupy 30,000 x 1,536 x 4 bytes, which is 184 MB.
4. Embedding cost, at a published rate of about 0.02 dollars per million tokens for a small embedding model: 15 million tokens costs 30 cents. Embedding is cheap; that is not where the money goes.
5. At query time you embed one question, find the 5 nearest chunks, and paste about 2,500 tokens into the prompt.
6. That 2,500 tokens is charged on every single question, which is why section 51.5’s cost table had retrieved context in it.

PLAIN51.11.4 what is really happening inside#

1. Chunking first, because it decides more than any other choice.
2. Chunk too small and each piece loses the context that made it meaningful. A sentence saying “this does not apply in India” is useless alone.
3. Chunk too large and the embedding becomes an average of several topics, which is close to nothing in particular, so it matches nothing well.
4. Strategies, in increasing order of effort: fixed size with overlap; splitting on structure such as headings and paragraphs; splitting on semantic shifts; and adding a short summary of the parent document to every chunk so it carries its own context.
5. Overlap exists so that a fact sitting on a boundary appears whole in at least one chunk. Ten percent is a common default.
6. Then embedding. An embedding model is a separate, smaller model that maps text to a fixed-length vector. Both the documents and the query must use the same one, and changing it means re-embedding everything.
7. Then storage and search. With 30,000 chunks you could compare the query to every vector, which is exact and fast enough. With 30 million you cannot.
8. So vector databases use approximate nearest neighbour search, and the dominant method is called HNSW.
9. Here is HNSW in plain words. Build a graph where each vector is a point joined to a few of its neighbours. Then build a second, sparser graph above it containing a random subset, and a third above that, and so on.
10. To search, start at the top layer, walk greedily to the closest point you can reach, then drop a layer and repeat from there.
11. The top layers move you across the whole space in a few hops. The bottom layer does the fine work. It is an express train, then a local train, then walking.
12. It is approximate: it can miss the true nearest neighbour. You trade recall against speed with a parameter, usually called ef_search.

TECHNICAL51.11.5 the engineer’s version#

1. RAG was named in a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research. HNSW is from a 2016 paper by Yury Malkov and Dmitry Yashunin. FAISS, the library that made billion-scale search routine, came from Facebook AI Research in 2017.
2. Vector store choices in 2026, honestly ranked by how often they are the right answer: pgvector inside an existing Postgres for anything up to tens of millions of vectors; Qdrant, Milvus or Weaviate when you need dedicated scale or filtering; Pinecone when you want it fully managed; Chroma or FAISS for prototypes and local work.
3. Pure vector search fails on exact tokens: part numbers, error codes, proper nouns and negations. Keyword search fails on paraphrase. Hybrid search runs both and fuses the results.
4. The standard fusion method is reciprocal rank fusion, from a 2009 paper by Cormack, Clarke and Buettcher: score each document by the sum of one over (a constant plus its rank) in each list.
5. It needs no score calibration, which is why it is used.
6. Re-ranking is a second, more expensive model that scores query-document pairs jointly, called a cross-encoder. Retrieve 50 candidates cheaply, re-rank to the best 5, pass those on. This is usually the single highest return improvement in a RAG system.
7. Evaluate retrieval separately from generation, always. Retrieval metrics: recall at k, mean reciprocal rank, normalized discounted cumulative gain. Generation metrics: groundedness, meaning every claim is supported by a retrieved chunk, and answer relevance.
8. The three honest failure modes:

9. The third one deserves a note. Models do sometimes override retrieved text with training-time knowledge, especially when the retrieved text is surprising. Requiring the model to quote the supporting sentence before answering reduces it measurably and makes the failure visible when it happens.

WORDS51.11.6 remember these#

RAG — fetch the right text and paste it in — retrieval-augmented generation, named in a 2020 paper. Chunk — one searchable piece of a document — a token-bounded span, usually with overlap, that is embedded as a unit. Embedding model — turns text into a vector of meaning — a bi-encoder producing a fixed-length dense representation. HNSW — express train, local train, then walking — hierarchical navigable small world graph for approximate nearest neighbour search. Hybrid search — keywords and meaning together — dense plus sparse retrieval fused, commonly by reciprocal rank fusion. Re-ranker — a second opinion on the shortlist — a cross-encoder scoring query and document jointly.

51.12 Tool use and function calling#

PLAIN51.12.1 in simple words#

1. A model produces text. Only text. It cannot press a button or call a web service.
2. It writes down a request, in a structured form you specified, saying which function it wants and with what arguments.
3. Your program reads that request, decides whether to run it, runs it, and puts the result back into the conversation as a new message.
4. The model then continues, now with the answer in front of it.
5. That is the whole mechanism. The model never executes anything. Your code does.
6. Which means every tool you offer is a door in your system that a piece of text can ask to open. That is a security boundary, and it must be treated like one.

PLAIN51.12.2 a picture in your head#

1. Think of a consultant behind a glass wall with no phone and no keyboard.
2. They can write notes and slide them under the glass.
3. A note might say: look up order 4471 and tell me the status.
4. You read it, decide whether that is a reasonable thing to ask, look it up, and slide the answer back.
5. At no point did the consultant touch your systems. They asked. You acted.
6. If you slide back whatever any note asks for, without checking, the glass wall is decorative.

Where this comparison breaks: a human consultant has intentions of their own. A model has none, but it can be steered by any text that reaches its context, including text inside the very documents it is reading for you. So the note under the glass may have been dictated by a stranger.

PLAIN51.12.3 a worked example#

1. A complete exchange, in four steps. First you declare the tool.

"tools": [{
  "type": "function",
  "function": {
    "name": "get_order_status",
    "description": "Look up the status of one order.",
    "parameters": {
      "type": "object",
      "properties": {
        "order_id": {"type": "string",
                     "description": "Order number"}
      },
      "required": ["order_id"]
    }
  }
}]

2. Second, the user asks “where is order 4471”, and the model replies with a request rather than an answer:

"message": {
  "role": "assistant",
  "content": null,
  "tool_calls": [{
    "id": "call_a1",
    "type": "function",
    "function": {
      "name": "get_order_status",
      "arguments": "{\"order_id\": \"4471\"}"
    }
  }]
},
"finish_reason": "tool_calls"

3. Third, your code validates the arguments, runs the real lookup, and sends the whole conversation back with one extra message:

{"role": "tool",
 "tool_call_id": "call_a1",
 "content": "{\"status\":\"shipped\",
              \"eta\":\"2026-08-15\"}"}

4. Fourth, the model produces the final answer: “Order 4471 shipped and should arrive on 15 August 2026.”

PLAIN51.12.4 what is really happening inside#

1. The tool declarations are inserted into the model’s context as text, usually as a schema in the system section. They cost tokens on every call.
2. The model was trained to emit a particular token pattern when it wants a tool. The serving layer detects that pattern and returns it as structured JSON instead of prose.
3. Many providers use constrained decoding here, so the arguments are guaranteed to match the schema. Others do not, and you must handle malformed JSON.
4. The model chooses the tool by matching your description text against the user’s request. So the description is a prompt, and vague descriptions cause wrong tool choices.
5. Now the security part, stated plainly. The model decides which tool to call based on text in its context. Some of that text came from your user. Some may have come from a document, a web page, or an email.
6. Therefore any text the model can read can attempt to trigger any tool the model can call. There is no separation between instructions and data.
7. So: validate every argument in your own code as if it came from an anonymous internet user, because effectively it did. Enforce permissions on the caller, not on the model.
8. Never pass model output into a shell, a SQL string or a file path without the same checks you would use on a web form.
9. Prefer narrow tools over general ones. get_order_status(order_id) is safe to expose. run_sql(query) is not, however convenient it looks.

TECHNICAL51.12.5 the engineer’s version#

1. History: OpenAI shipped function calling in June 2023. The research precursor was Toolformer, published in February 2023. Anthropic released the Model Context Protocol in November 2024 to standardize how tools are described and discovered; OpenAI adopted it in March 2025, and its specification is dated by release, with 2026-07-28 current in August 2026.
2. Cost note: every tool round trip re-sends the entire conversation. A five-tool-call task sends the context six times. This is the main reason agent workloads are expensive, and the main reason prompt caching matters.
3. Threat model, in the vocabulary that has settled: the dangerous combination is private data access, exposure to untrusted content, and an ability to communicate outward. Simon Willison named this the “lethal trifecta” in June
   2025. Remove any one of the three and exfiltration becomes much harder.
4. Practical controls: allowlist tools per session, require human confirmation for writes, run tools with least privilege, log every call with arguments, and put outbound network egress behind a proxy you control.

WORDS51.12.6 remember these#

Function calling — the model asks, you do it — structured tool-request output matching a declared JSON schema. Tool result — the answer handed back — a message with role “tool” carrying the execution output into the context. Constrained decoding — only valid output can be produced — logit masking against a grammar or schema. MCP — a common plug for tools — Model Context Protocol, from Anthropic, November 2024, adopted broadly in 2025 and 2026. Lethal trifecta — the dangerous combination — private data, untrusted content and outbound communication in one agent.

51.13 Agents: what the word actually means#

PLAIN51.13.1 in simple words#

1. An agent is a loop, not a product and not a personality.
2. The loop is: the model decides what to do next, calls a tool, sees the result, and decides again, until it says it is finished or you stop it.
3. A single API call is one question and one answer. An agent is many calls in a row, where the model chooses what the next call contains.
4. RAG is different again: retrieval happens once, before the model runs, and the model does not choose it.
5. So the ladder is: one call, then one call with tools, then a loop with tools, which is an agent.
6. The problem is arithmetic. Steps multiply. If each step is 95 percent reliable, twenty steps together are 36 percent reliable.
7. That is not a detail to be tuned away. It is the central engineering fact about agents.

PLAIN51.13.2 a picture in your head#

1. Think of a relay race where each runner hands over a written message rather than a baton.
2. Each runner is fast and mostly accurate, but occasionally mishears a word.
3. Over forty, it arrives as something else entirely, and nobody along the way noticed anything wrong.
4. Now add this: every runner is completely confident that what they passed on was correct.
5. The fix is not faster runners. It is checkpoints where the written message is compared against the original.

Where this comparison breaks: a garbled message is obviously garbled at the end. An agent’s wrong result usually looks exactly like a right one, because the same fluency that makes the output readable makes the error invisible.

PLAIN51.13.3 a worked example#

1. The compounding arithmetic, spelled out. This is just multiplication.

2. Read the 95 percent row. A step that fails once in twenty sounds excellent and produces a task that fails four times in five at thirty steps.
3. Now read the 99.9 percent row. To run thirty steps reliably you need per-step reliability that no language model currently offers unaided.
4. So the practical designs all do the same thing: reduce the number of steps, or make each step verifiable so failures are caught rather than compounded.
5. A verifiable step is one where something other than the model can check the result. Code that compiles. Tests that pass. A schema that validates. A number that reconciles.
6. This is exactly why coding agents work better than most other kinds. The compiler and the test suite are an independent judge on every step.

PLAIN51.13.4 what is really happening inside#

1. The common patterns, named, so you recognize them in documentation.
2. ReAct, from a 2022 paper, interleaves reasoning and acting: think, act, observe, repeat. It is the default shape of most agent frameworks.

Thought: I need the order status.
Action: get_order_status("4471")
Observation: {"status":"shipped","eta":"2026-08-15"}
Thought: I now have what I need.
Answer: It shipped and arrives on 15 August.

3. Plan-and-execute makes a full plan first, then carries out the steps. It is cheaper and more predictable, and worse at recovering when reality differs from the plan.
4. Reflection adds a criticism step: the model reviews its own output against the goal and revises. It helps when there is an objective standard to check against, and mostly produces confident agreement when there is not.
5. Multi-agent designs give different roles to different model calls, such as a planner, a coder and a reviewer, with messages between them.
6. The honest position on multi-agent systems as of 2026: they demonstrate well, they are expensive, and independent evidence that they beat one well-prompted model with good tools on ordinary tasks is thin. Some published analyses argue that errors compound faster across agents, not slower.
7. What actually makes agents work in production, in order of importance: narrow scope, verifiable steps, a hard step limit, a budget limit, retries with different phrasing, and a human checkpoint before anything irreversible.

TECHNICAL51.13.5 the engineer’s version#

1. Separate what is established from what is sold.
2. Established: tool-calling loops work well for bounded tasks with checkable outputs; coding assistants that run tests are genuinely useful; retrieval plus a small number of tool calls is a solid production pattern.
3. Active research: long-horizon planning, self-correction without an external verifier, reliable multi-agent coordination, and memory that survives across sessions in a useful way.
4. Marketing claim: that an agent can be given a business objective and left alone. No published evidence supports this for open-ended tasks, and the compounding arithmetic above argues against it directly.
5. Engineering controls that belong in any agent you deploy:

WORDS51.13.6 remember these#

Agent — a loop where the model chooses the next action — iterative plan-act-observe with tools and a stopping condition. ReAct — think, act, observe, repeat — the interleaved reasoning and acting pattern from a 2022 paper. Reflection — the model criticizing its own draft — a self-review step, useful mainly with an external standard to check against. Compounding error — small failures multiplying — end-to-end success equals per-step reliability raised to the number of steps. Human in the loop — a person approves the risky bit — a mandatory confirmation gate before irreversible actions.

51.14 Memory, and why a model with none appears to have some#

PLAIN51.14.1 in simple words#

1. A language model has no memory between calls. None at all.
2. The only state it has is the text currently in its context window.
3. When a chat application seems to remember yesterday, an ordinary database remembered it and pasted it back in.
4. Every turn of a conversation resends the whole conversation so far.
5. That has a cost consequence people miss: turn twenty is not the same price as turn one, because turn twenty carries nineteen turns of history.
6. Everything sold as “memory” is an application feature: store some text, retrieve the relevant bits later, put them in the prompt.
7. So memory is just retrieval, from section 51.11, pointed at your own past conversations instead of your documents.

PLAIN51.14.2 a picture in your head#

1. Imagine a brilliant advisor with complete amnesia every night.
2. Each morning you hand them a folder containing everything relevant from before, and they work perfectly.
3. They are not remembering. You are.
4. If your folder is wrong, they are confidently wrong, and they have no way to notice, because the folder is their entire world.

Where this comparison breaks: the advisor would know they had amnesia. The model has no sense of a gap. Text it was never given simply does not exist for it, and it will fill the space with something plausible rather than reporting the absence.

PLAIN51.14.3 a worked example#

1. A ten-turn conversation with 500 tokens per turn, no caching:

2. Ten turns of 500 tokens each is 5,000 tokens of conversation, but you paid for 27,500 input tokens, which is five and a half times more.
3. The growth is quadratic in the number of turns. Doubling the turns quadruples the input bill.
4. Three fixes, all standard: cache the stable prefix, summarize old turns into a short note, or drop old turns entirely past a window.

PLAIN51.14.4 what is really happening inside#

1. The common designs, and what each is really doing.
2. Full history: send everything. Simple, exact, and quadratic in cost. Fine for short sessions.
3. Sliding window: keep the last N turns. Cheap and predictable; loses anything older, silently.
4. Running summary: after every few turns, ask the model to compress the older part into a paragraph, and send the summary plus recent turns. Cheap, and lossy in ways you cannot predict.
5. Retrieved memory: store every turn in a vector database and retrieve the few most relevant to the current message. Scales, and inherits every failure mode from section 51.11.
6. Structured facts: extract specific fields, such as the user’s name, preferences and account, into an ordinary database, and inject them as text. This is the most reliable and the least fashionable.
7. The limits are the same in every case: whatever is not retrieved does not exist, and nothing about this is the model remembering.

TECHNICAL51.14.5 the engineer’s version#

1. Context is a hard bound, not a soft one. Exceeding it produces an error or silent truncation, and truncation usually removes the oldest tokens, which is where your system prompt might be.
2. Put the system prompt where truncation cannot reach it, and validate token counts before sending, using the model’s own tokenizer.
3. Structured extraction beats free-text memory for anything you will filter or act on. “The user’s plan is enterprise” belongs in a column, not in prose inside a vector store.

WORDS51.14.6 remember these#

Context window — everything the model can see right now — the maximum token sequence, its only state. Conversation history — the transcript resent every turn — the messages array, billed in full on each call. Summarization memory — squash the old part — periodic lossy compression of earlier turns into a short note. Structured memory — facts in a proper database — extracted fields injected as text, the most reliable design.

51.15 The limits, stated plainly#

PLAIN51.15.1 in simple words#

1. Hallucination is when a model states something false with the same confidence it states something true.
2. It is not a bug that was left in. It follows from what the model is.
3. The model was trained to produce likely continuations of text. A confident, fluent, plausible answer is a likely continuation whether or not it is true.
4. There is also no internal flag for “I do not know”. The model always produces a next token, and the machinery for producing a true one and a false one is identical.
5. Asking it to be accurate does not help, because it is not choosing to be inaccurate.
6. What helps is putting truth in front of it, or letting a tool check.
7. And this is the important sentence: confident wrongness is the default failure mode. Errors do not arrive marked as errors.

PLAIN51.15.2 a picture in your head#

1. Think of an extremely well-read person answering at a dinner party, at speed, who considers not answering to be a failure.
2. Ninety percent of the time they are right and it is delightful.
3. Ten percent of the time they produce the same tone, the same fluency, the same specific detail, and it is invented.
4. You cannot tell the two apart from the outside, and neither can they.
5. Now imagine the party is a legal filing, a medical note or a payment.

Where this comparison breaks: the dinner guest could stop and say “I am not sure”. The model can be trained to say that more often, and modern ones do, but it is still producing the phrase because such phrases followed such contexts in training, not because it inspected its own uncertainty and found it high.

PLAIN51.15.3 a worked example#

1. What actually reduces hallucination, and what does not:

2. A concrete pair. Ask for a citation for a claim and you may get a real- looking reference that does not exist, with plausible authors, journal and year.
3. Give the model five real abstracts and require it to quote from one, and the invented citation disappears, because the correct behaviour is now the easy continuation.
4. Now three things the model cannot know, whatever you do.
5. Anything after its training cut-off. It will still answer, using the world as it was.
6. Anything private: your files, your database, this morning’s ticket. Unless you put it in the context, it is not there.
7. Its own weights and internals. A model’s account of why it answered is a plausible story generated after the fact, not a readout. This is established: self-reported reasoning does not reliably match the computation.

PLAIN51.15.4 what is really happening inside#

1. Why counting and arithmetic fail, connecting to Chapter 47.
2. The model never sees letters. It sees tokens, which are chunks of several characters, and the chunks were chosen by compression statistics.
3. So “strawberry” may arrive as two or three tokens, and the letter r is not individually visible in any of them. Counting them is like counting bricks in a photograph of a wall taken from a mile away.
4. Long arithmetic fails for a second, separate reason: the model does a fixed amount of computation per token and has no scratchpad.
5. Multiplying two six-digit numbers requires carrying intermediate results through many steps. With no working memory, the only place to put them is the output itself.
6. That is exactly why writing out the steps helps, and why handing the sum to a calculator tool helps far more.
7. Now prompt injection, which is the unsolved one. The model has a single channel. Instructions and data arrive as the same tokens.
8. So text inside a document, a web page, an email or a code comment can address the model directly, and the model has no reliable way to tell that this text was not from you.
9. The term was coined by Simon Willison in September 2022. As of August 2026 it remains unsolved in the general case, and security guidance from national agencies now assumes it rather than promising a fix.
10. Jailbreaking is a related but different thing: the user of a model trying to get past its own trained refusals. Injection is a third party attacking your application through content it processes.
11. Sycophancy is the tendency to agree with the user, to soften a correct answer under pushback, and to praise a bad idea. It comes from training on human preference, and humans prefer agreement.

TECHNICAL51.15.5 the engineer’s version#

1. Hallucination is best understood as a calibration problem. A well-calibrated model would assign confidence matching its accuracy and abstain below a threshold.
2. A 2025 paper from OpenAI, “Why Language Models Hallucinate”, argues the incentive is structural: both pre-training objectives and benchmark scoring reward a guess over an admission of ignorance, exactly as a multiple-choice exam with no penalty for wrong answers does.
3. That implies the fix is partly in evaluation design: score abstention as better than a wrong answer, and models trained against that scoring abstain more. This is active work, not a solved problem.
4. Uncertainty signals you can actually use: token log-probabilities where the provider exposes them, agreement across several samples at nonzero temperature, and agreement between two different models. All are weak but better than nothing.
5. Prompt injection defenses that reduce risk without solving it: strict separation of trusted and untrusted content with explicit markers, least-privilege tools, breaking the lethal trifecta by removing outbound communication or private data access from the same agent, output filtering, and human confirmation on consequential actions.
6. Defenses that do not work: instructing the model to ignore instructions in documents, and detecting injections with another language model, which is itself injectable.
7. What all of this means if you are building:

WORDS51.15.6 remember these#

Hallucination — confident invention — generation of unsupported content, a calibration failure rather than a discrete bug. Calibration — confidence matching accuracy — the property a model would need to know when to abstain. Prompt injection — instructions smuggled in through data — an architectural consequence of one shared channel; unsolved as of August 2026. Jailbreak — getting past the model’s own rules — user-side circumvention of trained refusals, distinct from injection. Sycophancy — agreeing because you pushed — preference-trained bias toward agreement and flattery. Knowledge cut-off — the date the world stopped for it — the end of the training data, after which nothing is known.

51.16 The gaps you did not ask about but need#

PLAIN51.16.1 in simple words#

1. Six things you will meet in week one that nobody puts in the introduction.
2. Multimodality: how a picture becomes something a text model can read.
3. Reasoning models: models trained to think at length before answering, and billed for the thinking.
4. Long context: how a model that started with 2,000 tokens now takes a million, and why that is less useful than it sounds.
5. Cards: a model card describes one model; a system card describes a whole deployed product including its safety testing.
6. The safety vocabulary: alignment, red-teaming, refusal, guardrails, evals.
7. Distillation: making a small model imitate a big one.

PLAIN51.16.2 a picture in your head#

1. For images, think of a photograph cut into a grid of small squares.
2. Each square is described as a list of numbers, in the same kind of language the model already uses for words.
3. The squares are then laid in a row alongside the text tokens, and the model reads the whole row as one sequence.

Where this comparison breaks: the translation is learned, not defined, and it is lossy in ways nobody chose. Fine print, precise counts and exact spatial relations survive badly, which is why models misread charts and miscount objects far more often than their fluent descriptions suggest.

PLAIN51.16.3 a worked example#

1. Rough token costs of non-text inputs, which is what you will be billed on:

2. So ten screenshots can cost more than a long document, and a reasoning model may bill 4,000 hidden thinking tokens behind a 200-token answer.

PLAIN51.16.4 what is really happening inside#

1. Images: a vision encoder cuts the picture into patches, embeds each patch, and a small projection layer maps those embeddings into the same space as token embeddings. The transformer then treats them as ordinary positions.
2. Audio: either a spectrogram is patched the same way, or a separate codec turns sound into discrete tokens from a learned audio vocabulary.
3. Reasoning models: trained with reinforcement learning to produce long internal chains of thought before answering. Established fact: this raises accuracy substantially on mathematics, competitive programming and multi-step logic.
4. Active research: how far it generalizes, and the 2026 finding that past a point extra thinking reduces accuracy.
5. Long context is achieved by changing how positions are encoded, most commonly by scaling rotary position embeddings, plus attention variants that avoid materializing a full attention matrix.
6. The honest note: a large advertised context is a capacity, not a competence. Retrieval quality inside a long context degrades with distance from the ends, and simple needle-finding benchmarks overstate real performance on tasks needing several facts combined.
7. Distillation, from a 2015 paper by Hinton and colleagues: train a small student model on the outputs of a large teacher. Most small fast models available today were made this way.

TECHNICAL51.16.5 the engineer’s version#

1. Alignment, defined concretely rather than as a slogan: the engineering work of making a model’s outputs conform to a written specification of intended behaviour, implemented through preference training and rule-based methods, and measured by evaluations. It is a practice with artifacts, not an aspiration.
2. Red-teaming: adversarial testing by humans or automated attackers, aimed at producing the failures the model is meant not to produce. Findings feed back into training and into guardrails.
3. Refusal: the trained behaviour of declining a request. Both over-refusal and under-refusal are measured, and the trade between them is a product decision, not a technical constant.
4. Guardrails: checks outside the model, on input and output, such as classifiers, regular expressions, schema validation and allowlists. They are the only part of the safety stack that is deterministic.
5. Evals: automated test suites over fixed inputs with scoring, covering both capability and safety. Chapter 50 covered how to build them; the essential discipline is that a private eval set you wrote beats any public leaderboard.
6. System card: a document about a deployed system rather than a bare model, covering intended use, safety evaluations, red-team results, known failure modes and mitigations. The convention was established by the GPT-4 system card in March 2023 and is now normal for frontier releases.
7. Where the field appears to be heading, explicitly labelled speculation and not fact: more inference-time compute rather than only larger models; sparse mixture-of-experts as the default architecture; agentic use as the main product surface; standardized tool protocols; and small models on devices for the majority of routine work.
8. Any of these could be wrong within a year, and this paragraph should be read with its date attached.

WORDS51.16.6 remember these#

Multimodal — text plus images or audio — inputs encoded into the same embedding space and consumed as one sequence. Reasoning model — thinks before it answers — trained to emit long internal chains of thought, billed as output tokens. System card — the document about the product — deployment-level description of intended use, evaluations and mitigations. Alignment — making behaviour match the specification — preference training and rules, measured by evaluations. Guardrail — a check outside the model — deterministic filtering or validation on inputs and outputs. Distillation — a small model taught by a big one — student trained on teacher outputs, from a 2015 paper.

51.17 A practical decision guide#

PLAIN51.17.1 in simple words#

1. Most projects reach for the most complicated option first. Work the other way.
2. Ask whether the task needs a language model at all. Regular expressions, database queries and existing libraries are cheaper, faster and exact.
3. If it does, start with one API call and a good prompt.
4. Add retrieval when the answer depends on documents the model has not seen.
5. Add tools when the answer depends on live data or on an action.
6. Add a loop, making it an agent, only when the number of steps genuinely cannot be known in advance.
7. Fine-tune when the format or style must be fixed, or when you want a small model to do one narrow job cheaply.
8. Self-host when policy, volume or control demands it, not because it sounds cheaper.

PLAIN51.17.2 a picture in your head#

1. Think of choosing transport for a journey.
2. A taxi is a hosted API: instant, no maintenance, pay per trip.
3. Bringing a map is retrieval.
4. Hiring a driver who chooses the route as they go is an agent.
5. Buying a fleet is self-hosting, and it only pays if you drive constantly.

Where this comparison breaks: transport choices are reversible in an afternoon. Architecture choices lock in data formats, evaluation sets and vendor contracts, so the cost of starting too complicated is higher than the comparison suggests.

PLAIN51.17.3 a worked example#

Does it need judgement about language?
  no  -> write ordinary code. Stop.
  yes -> Is the needed knowledge in the model already?
           yes -> one API call with a good prompt
           no  -> Is it in documents you hold?
                    yes -> RAG
                    no  -> Is it live or an action?
                             yes -> tools
                             no  -> reconsider the task
Is the number of steps known in advance?
  yes -> a fixed chain of calls
  no  -> an agent loop, with step and cost limits
Is the output format or style unstable?
  yes and you have 1,000+ examples -> fine-tune
  yes and you do not             -> few-shot examples
Must data stay inside your network, or is volume
above roughly 3 billion output tokens a month?
  yes -> self-host
  no  -> hosted API

PLAIN51.17.4 what is really happening inside#

1. The same guidance as a table, with the cost and latency notes that decide arguments.

2. Latency notes worth having in mind. A short hosted call is roughly 0.5 to 3 seconds. Adding retrieval adds 50 to 300 milliseconds.
3. Each tool round trip adds a full model call. A reasoning model can take 10 to 60 seconds. An agent is measured in minutes.
4. Cost notes. A simple call is fractions of a cent. RAG multiplies input tokens. An agent multiplies whole calls. Reasoning multiplies output tokens. The multipliers stack.

TECHNICAL51.17.5 the engineer’s version#

1. Build the evaluation set before choosing an approach. Thirty to two hundred real examples with known good answers, held privately, decides every question in this section empirically rather than by argument.
2. Choose the cheapest model that passes your evaluation, then move up only if it fails. Most production traffic does not need a frontier model.
3. Design for model substitution from the first day: keep prompts in one place, keep an adapter layer, and re-run the evaluation set when you change models, because a provider changing an undated model name is a real event.
4. Route by difficulty. A small model handles the common case and escalates on low confidence or explicit triggers. This routinely cuts costs by an order of magnitude while leaving quality on hard cases intact.
5. Instrument tokens per request, cost per task, cache hit rate, p50 and p99 latency, and end-to-end task success. The last one is the only one your users experience.

WORDS51.17.6 remember these#

Model routing — send easy work to the cheap model — confidence or rule-based dispatch across a tier of models, with escalation. Escalation — try harder when the cheap answer looks weak — a second call to a stronger model triggered by a confidence signal. Eval set — your own private exam for the system — a fixed collection of real inputs with expected outputs, versioned with the code. Golden path — the case you optimize for — the dominant traffic pattern that should decide model and architecture choice.

51.98 Common wrong ideas#

1. Wrong: it looked that up when I asked. Right: unless a retrieval or search tool ran, nothing was looked up. The answer came from frozen weights.
2. Wrong: it remembers me. Right: it has no memory between calls. An application stored your text and pasted it back into the context.
3. Wrong: hallucination will be fixed in the next version. Right: it is a property of predicting likely text without verification. Versions reduce the rate; retrieval, tools and checking reduce it much more.
4. Wrong: the system prompt is secure and hidden. Right: it is ordinary tokens at the front of the context, obeyed by training habit, and it can be overridden or revealed by clever input.
5. Wrong: an agent can be trusted to finish the job. Right: per-step errors compound, so a 95 percent reliable step is 21 percent reliable over thirty steps. Cap steps, verify each one, keep a human on writes.
6. Wrong: self-hosting is cheaper. Right: you rent the machine whether you use it or not. Break-even against cheap hosted models is around billions of output tokens a month.
7. Wrong: a bigger context window means it uses all of it well. Right: capacity is not competence. Attention to the middle of a long context is measurably weaker than to the ends.
8. Wrong: more parameters always means faster answers if the hardware is good. Right: decode speed is memory bandwidth divided by model bytes. A bigger model is slower on the same machine, in direct proportion.
9. Wrong: prompt engineering is the main skill. Right: it is real but shallow. Retrieval quality, task decomposition, evaluation and tool design matter more.
10. Wrong: the model runs my code when it calls a function. Right: it emits a structured request; your program decides whether to run it. Every tool is a security boundary you own.

51.99 Chapter summary in 20 lines#

1. Inference is the frozen model answering; training is what produced it, and it needs roughly ten times the memory.
2. An API call is ordinary HTTPS carrying a list of messages plus sampling settings, and the whole conversation is resent every time.
3. The system prompt is only leading tokens. Its authority is a training habit, not a protection.
4. Generation has two phases: prefill, which is parallel and compute-bound, and decode, which is sequential and bandwidth-bound.
5. Decode speed is approximately memory bandwidth divided by model bytes, which explains almost every speed question people ask.
6. Time to first token is network plus queue plus prefill; tokens per second afterwards is a different limit entirely.
7. The KV cache turns quadratic recomputation into linear work and costs 128 KiB per token for an 8B model at 16-bit.
8. Continuous batching, PagedAttention, speculative decoding, quantization, tensor parallelism and prefix caching are the serving toolkit.
9. The same model runs at different speeds at different providers because of hardware, quantization, batching policy, load and distance.
10. Output tokens cost five to six times input tokens because decode is sequential and prefill is parallel.
11. Self-hosting is rarely cheaper below very high sustained volume; the good reasons are privacy, control and stability.
12. llama.cpp is the engine, Ollama and LM Studio the friendly wrappers, vLLM and SGLang the serving engines, MLX the Apple silicon path.
13. Memory decides whether a model runs; bandwidth decides how fast. Apple’s unified memory wins on capacity and loses on bandwidth and prefill.
14. Read the model card for base model, data, licence, limitations and evaluations, and default to Q4_K_M unless you have a reason.
15. Training a tiny model on a laptop teaches shape without knowledge; do it once as education, then fine-tune or use an open model instead.
16. RAG is chunk, embed, store, retrieve, inject, generate; chunking and retrieval quality dominate the outcome.
17. Tool calling means the model emits a request and your code executes it, so every tool is a security boundary.
18. An agent is a loop, and per-step errors compound multiplicatively, so narrow scope, verifiable steps and human checkpoints are what make it work.
19. Hallucination is intrinsic to predicting likely text without verification; retrieval, tools and constrained output reduce it, exhortation does not.
20. Prompt injection remains unsolved as of August 2026 because instructions and data share one channel, so build as though confident wrong output will happen, because it will.