Machine Learning From Scratch - Fitting, Loss, Gradients and Neural Networks

46.0 What this chapter gives you#

1. You will be able to say exactly what machine learning is, in one sentence, without using the word “intelligence”, and explain how it differs from ordinary programming with a concrete example of both.
2. You will be able to tell the honest history: the perceptron, two winters, the support vector machine years, and why 2012 changed everything.
3. You will be able to fit a straight line to five points by hand, define what a parameter is, and compute the error of your fit two different ways.
4. You will be able to explain gradient descent to someone who has never seen a derivative, then work four steps of it on paper with real numbers.
5. You will be able to use the words sample, feature, label, epoch, batch, iteration and step correctly, and never confuse them again.
6. You will be able to read a training curve and say whether the model is underfitting, overfitting, or learning properly.
7. You will be able to name the main classical methods, say when each is the right tool, and explain why neural networks are not always the answer.
8. You will be able to describe one artificial neuron exactly, compute its output by hand, and prove in two lines why a network with no activation function is worthless.
9. You will be able to explain backpropagation in plain language as blame assignment, then do a complete backward pass through a small network with real numbers.
10. You will be able to say what a trained model physically is, and why training costs so much more than using the result.

46.1 What machine learning actually is#

PLAIN46.1.1 in simple words#

1. In ordinary programming you know the rule, and you write it down.
2. You want to convert Celsius to Fahrenheit, so you write: multiply by 1.8, then add 32. Done. The rule was in your head and now it is in the file.
3. Machine learning is for the other case: you do not know the rule, but you have a pile of examples of the rule working.
4. You have ten thousand emails, each already marked “spam” or “not spam”. You cannot write the rule that separates them. Nobody can.
5. So you do something different. You write down a shape for a rule, with blank spaces in it where numbers should go.
6. Then you write a procedure that fills in those numbers, by trying, checking against the examples, and adjusting.
7. When the procedure stops, the blanks are full. Those numbers are the rule.
8. Nobody wrote them. Nobody can read them. But they work on the examples, and usually on new cases too.
9. That is the whole of machine learning. A rule-shaped thing with adjustable numbers, plus a procedure that tunes the numbers to match examples.
10. At heart this is curve fitting: finding the settings that make a formula pass close to a set of points.
11. Saying “it is just curve fitting” is not dismissive. It is accurate, and the accuracy is what lets you reason about it.
12. Curve fitting in a space of a billion adjustable numbers, on data covering most of what humans have written, does astonishing things. It is still curve fitting.

PLAIN46.1.2 a picture in your head#

1. Picture a sound mixing desk with a hundred sliders on it.
2. Each slider changes the sound a little. You do not know what any single one does, and no slider has a label.
3. Someone plays you a recording of how the finished mix should sound.
4. Your job is to move the sliders until what comes out of the speakers matches the recording.
5. You cannot solve this by thinking. You have to move sliders and listen.
6. So you make a small move, listen, and ask one question: closer or further?
7. If closer, keep going that way. If further, go back the other way.
8. Do this over and over, for all hundred sliders, thousands of times, and the mix creeps towards the target.
9. At the end you have a hundred slider positions that produce the right sound. You still cannot say what any one slider does.
10. The sliders are the parameters. The recording is the training data. The listening is the loss. The nudging is the optimizer.

Where this comparison breaks: a real sound engineer knows what each slider does, so they aim. Nothing here aims. Also, real learning does not move one slider and listen. It computes, in one shot, the direction every slider should move, all at once. That computation is called backpropagation and it is the single most important idea in the chapter. Finally, a mix only has to sound right once. A model has to work on sounds it has never heard, which is a much harder demand and the source of nearly every practical problem in the field.

PLAIN46.1.3 a worked example#

1. Take spam detection. First, the hand-written way.

def is_spam_handwritten(email):
    subject = email.subject.lower()
    if "free" in subject and email.link_count > 3:
        return True
    if "viagra" in email.body.lower():
        return True
    if email.sender not in email.user_contacts:
        if email.exclamation_count > 5:
            return True
    return False

2. Every line of that was decided by a person. You can read it, argue with it, and fix it. Those are real advantages and they are often undervalued.
3. Now the failure. A spammer writes “FR33” instead of “free”. The rule misses.
4. You add a rule for “FR33”. They write “F.R.E.E”. You add another rule.
5. This is a race you lose, because there are more spellings than there are hours in your life.
6. Now the learned way. You do not write rules. You write a shape.
7. The shape: give every word a number, called its weight. Add up the weights of the words present. If the total is above zero, call it spam.
8. That shape has one blank per word in the vocabulary. Say 50,000 blanks.
9. You feed in 10,000 emails already marked spam or not, and let the tuning procedure fill the blanks. Afterwards the numbers look like this:

10. Nobody told the system about “fr33”. It appeared in spam in the training data, so it got a large positive weight automatically.
11. That is the real difference. The hand-written rule needed a human to notice each trick. The learned rule needs only examples containing the trick.
12. Test one email with the words “free”, “meeting” and “unsubscribe”, with weights +2.20, -2.60 and +1.35. Total: 2.20 - 2.60 + 1.35 = +0.95.
13. Above zero, so it says spam. It is probably wrong, because “meeting” should have carried more. That is a failure of the training data, not of the idea.
14. Both approaches fail. The difference is how they fail and how you fix them. You fix the first with a code change. You fix the second with more data.

PLAIN46.1.4 what is really happening inside#

1. Every machine learning system has exactly four pieces. Learn these four and nothing later will surprise you.
2. Piece one, the model: the shape of the rule, with blanks. A straight line, a tree of yes-or-no questions, or a network with a billion blanks.
3. Piece two, the parameters: the numbers that go in the blanks. Before training they are random or fixed. After training they are the answer.
4. Piece three, the loss: a single number saying how wrong the current parameters are on the examples. Lower is better. Zero would be perfect.
5. Piece four, the optimizer: the procedure that changes the parameters to make the loss smaller.
6. Training is a loop over these four. Predict with current parameters. Measure the loss. Work out which way each parameter should move. Move them a little. Repeat, often millions of times.
7. Notice what has been swapped. In ordinary programming, the logic is the thing you write and the data is what flows through it.
8. In machine learning, the data is the thing you provide and the logic is what comes out. The data has become the specification.
9. That single swap explains almost everything strange about the field. Bad data gives a bad model, in exactly the way a bad specification gives bad software, and for exactly the same reason.
10. It also explains why debugging feels so different. There is no line to step through. There is only a distribution of examples, and a number.

TECHNICAL46.1.5 the engineer’s version#

1. The term was coined by Arthur Samuel of IBM, in his 1959 paper “Some Studies in Machine Learning Using the Game of Checkers”, in the IBM Journal of Research and Development. His program ran on an IBM 701.
2. The standard formal definition is Tom Mitchell’s, from his 1997 textbook Machine Learning: a program learns from experience E with respect to task T and performance measure P, if its performance at T, measured by P, improves with E.
3. Formally: choose a hypothesis class H, a set of functions parameterized by a vector theta. Choose a loss function L. Find theta that minimizes the average loss over the training data.
4. That last step is called empirical risk minimization. “Empirical” because you minimize measured loss on data you have, not true loss on data you do not have. The gap between the two is the whole subject of generalization.
5. The linear spam model above is logistic regression on a bag-of-words feature vector. It was the industry standard for spam filtering from the late 1990s, alongside naive Bayes, which Paul Graham popularized with his 2002 essay A Plan For Spam.
6. Side by side:

7. Established fact: the “rule-shaped thing plus optimizer” description above is complete and correct for essentially every supervised learning system in production today, including the largest language models.
8. Marketing claim: that these systems reason, understand, or think. Those words have no agreed technical definition here, and vendors use them because they are unfalsifiable. Treat them as advertising until defined.
9. Active research: whether large models build internal structures that deserve to be called world models or algorithms rather than statistics. The field of mechanistic interpretability studies this. It is genuinely open, and serious researchers disagree, so do not let anyone tell you it is settled.

WORDS46.1.6 remember these#

1. Machine learning — filling in blanks in a rule from examples — fitting the parameters of a hypothesis class by minimizing empirical risk.
2. Model — the shape of the rule, with blanks — the hypothesis class, a parameterized family of functions.
3. Parameter — a number the system may change during training — a component of the weight vector theta, learned by optimization.
4. Loss — one number saying how wrong you are — a scalar objective function measuring disagreement between prediction and target.
5. Optimizer — the procedure that changes the numbers — the algorithm that updates theta to reduce the loss, usually a gradient method.
6. Curve fitting — finding settings that make a formula pass near the points — function approximation from finite samples. An accurate description, not an insult.

46.2 A short honest history#

PLAIN46.2.1 in simple words#

1. This field is older than most people think, and it has failed publicly at least twice. Knowing that keeps you sane during the hype.
2. In 1943 two researchers, Warren McCulloch and Walter Pitts, wrote down a simple mathematical model of a nerve cell. That is where it starts.
3. In 1957 and 1958 Frank Rosenblatt built the perceptron, a machine that learned to tell simple pictures apart by adjusting numbers.
4. The press coverage was wild, and far ahead of the reality. That is a pattern you will see again in your lifetime.
5. In 1969 two respected researchers proved the perceptron could not do certain very simple things. Funding collapsed. This is called the first AI winter.
6. The fix, an algorithm for training networks with more layers, was invented several times between 1970 and 1974, and mostly ignored.
7. In 1986 it was published clearly, in a famous journal, and the field came back. Then it faded again in the early 1990s. Second winter.
8. Through the late 1990s and 2000s a different family of methods, based on clean mathematics rather than networks, did better in practice.
9. In 2012 a neural network won a big image contest by such a large margin that the argument ended in a single afternoon.
10. The thing that made 2012 possible was not a new idea. It was cheap parallel arithmetic hardware, plus a very large labelled dataset.

PLAIN46.2.2 a picture in your head#

1. Think of a seed planted in soil that is too poor to grow it.
2. The seed is the idea of learning by adjusting numbers. It was planted in 1958 and it was a good seed.
3. The soil is three things: computing power, data, and algorithms.
4. In 1958 there was almost no computing power, almost no data, and only a one-layer algorithm. The seed sprouted and died.
5. In 1986 the algorithm arrived. Compute was better, data was still thin. The seed sprouted higher and died again.
6. By 2012 all three were rich at once: graphics chips gave the compute, the internet gave the data, and the 1986 algorithm was still sitting there unchanged. The seed grew into a tree.
7. Almost nothing about the seed changed across those fifty-four years.

Where this comparison breaks: the algorithms did improve, and pretending otherwise is unfair to a great deal of careful work. Better activation functions, better initialization, better normalization and better optimizers each moved the field, and without them the 2012 result would not have trained at all. The honest version: compute and data were the necessary conditions, and a decade of unglamorous engineering was what turned “possible in principle” into “trains on a Tuesday”.

PLAIN46.2.3 a worked example#

1. The overclaim of 1958, in detail, because it teaches you how to read news about this field.
2. Rosenblatt built the Mark I Perceptron at the Cornell Aeronautical Laboratory in Buffalo, New York, funded by the Office of Naval Research.
3. It had a 20 by 20 grid of 400 light sensors, so it saw a 400-pixel picture.
4. Its adjustable numbers were physical. They were potentiometers, small dials, turned by electric motors during training. You could watch the machine learn by watching the dials move.
5. It could learn to tell simple shapes apart. It genuinely worked, and that was a real achievement.
6. In July 1958 The New York Times reported the Navy’s expectation that the machine would be able to “walk, talk, see, write, reproduce itself and be conscious of its existence”.
7. It could sort simple shapes. The gap between those two sentences is the entire lesson of this section.
8. Eleven years later, Marvin Minsky and Seymour Papert published the book Perceptrons, in 1969, showing what one layer could not do.
9. Their killer example was XOR: output 1 if exactly one of two inputs is 1. Here is the whole problem.

   x2
    1 |  1        0        1 means "should output 1"
      |
    0 |  0        1
      +-------------
         0        1     x1

  A single straight line must put both 1s on one side.
  No straight line can. That is the entire objection.

10. A one-layer perceptron can only draw a straight dividing line. XOR needs two lines, or a bent one. So one layer cannot do XOR.
11. That was mathematically correct and it was fatal in practice, because nobody yet had a working way to train two layers.
12. Minsky and Papert did say multi-layer networks were more capable. The reception of the book was harsher than its text, which is common.

PLAIN46.2.4 what is really happening inside#

1. Why did the training method take so long to arrive, when the mathematics is just repeated use of the chain rule from school calculus.
2. Because nobody was looking for it in the place it was found. The reverse accumulation method was worked out for a completely different purpose.
3. In 1970 Seppo Linnainmaa, in his master’s thesis at the University of Helsinki, described how to compute the derivative of a whole computation efficiently by walking backwards through it.
4. His subject was rounding error in numerical algorithms. He was not thinking about learning at all. The algorithm was correct and general anyway.
5. In 1974 Paul Werbos, in his Harvard PhD thesis, applied that idea directly to training neural networks. Very few people read it.
6. In 1986 David Rumelhart, Geoffrey Hinton and Ronald Williams published “Learning representations by back-propagating errors” in Nature. Same idea, clearly written, in a journal everyone read, with results.
7. The lesson is not about mathematics. It is that an idea has to arrive in a readable form, in a place the right people look, at a moment when the hardware can demonstrate it.
8. The second winter came after 1987 for practical reasons: networks of the day were slow to train, needed careful hand-tuning, and were beaten on real tasks by simpler methods with better theory.
9. What broke the deadlock in 2012 was throughput. A graphics chip does the same arithmetic operation on thousands of numbers at once, which is exactly the shape of a neural network’s work.

TECHNICAL46.2.5 the engineer’s version#

1. The timeline, with the facts you should be able to quote.

2. The 2012 result, precisely. AlexNet, by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, won the ImageNet Large Scale Visual Recognition Challenge with a top-5 error rate of 15.3 percent. The second-placed entry, which did not use deep learning, scored 26.2 percent.
3. A gap of nearly 11 percentage points in a contest usually decided by one is why the field turned in a single day.
4. The training set was about 1.2 million images across 1,000 categories. The network had roughly 60 million parameters, most of them in its fully connected layers.
5. It was trained on two NVIDIA GTX 580 cards with 3 GB of memory each, for about five to six days. The 3 GB limit is why the network was split across two cards at all. A hardware constraint shaped the architecture.
6. Chapter 22 explained what a graphics chip is and why it multiplies matrices so quickly. The short version, without repeating it: these chips run thousands of identical arithmetic operations in parallel, and a neural network is thousands of identical arithmetic operations. The fit is exact and it was an accident of history.
7. ImageNet itself, led by Fei-Fei Li and published at CVPR in 2009, was arguably the more important contribution. Without a large, clean, labelled dataset there is nothing to fit.
8. The support vector machine era rests on Corinna Cortes and Vladimir Vapnik’s 1995 paper “Support-Vector Networks”, building on the kernel trick from Bernhard Boser, Isabelle Guyon and Vapnik in 1992. From roughly 1995 to 2010 these methods usually beat neural networks on the benchmarks of the day, and had far better theory.
9. Hinton, Yoshua Bengio and Yann LeCun shared the 2018 Turing Award, presented in 2019. In October 2024 John Hopfield and Geoffrey Hinton received the Nobel Prize in Physics for foundational discoveries enabling machine learning with artificial neural networks.
10. Historical note on naming: “backpropagation” is a shortening of “backward propagation of errors”, from the 1986 Nature paper title.

WORDS46.2.6 remember these#

1. Perceptron — the first machine that learned by adjusting dials — a single linear threshold unit with the 1958 update rule, convergent on linearly separable data.
2. AI winter — a period when funding and interest collapsed — the funding contractions of roughly 1974 to 1980 and 1987 to 1993.
3. XOR problem — the simple task one layer cannot do — a function that is not linearly separable, the core objection of Perceptrons, 1969.
4. Backpropagation — the way to find how each number should change — reverse- mode automatic differentiation applied to a layered network.
5. ImageNet — the giant labelled picture collection — the dataset of about 14 million images, and its annual challenge, ILSVRC, run from 2010.
6. AlexNet — the network that won in 2012 and ended the argument — an eight-layer CNN, 60 million parameters, 15.3 percent top-5 error.

46.3 Fitting a line, the foundation of everything#

PLAIN46.3.1 in simple words#

1. Everything in this chapter is an elaboration of one small task: draw the best straight line through some dots on a graph.
2. Understand this properly and the rest is bookkeeping.
3. You have pairs of numbers. For each one, an input and an output.
4. Hours studied and exam mark. Size of a flat and its rent. Anything.
5. Plot them. They do not lie on a perfect line, because reality is messy, but they lean in a direction.
6. You want the line that leans the same way and sits as close to all the dots as it can.
7. A straight line is described completely by two numbers. Just two.
8. The first is how steep it is. Call it w. If w is 2, then every step right makes the line go up two.
9. The second is how high it sits. Call it b. It is the height of the line where the input is zero.
10. So the line is: output equals w times input, plus b. Written short: y = wx + b.
11. Those two numbers, w and b, are the only things you are allowed to change.
12. A number the system is allowed to change to fit the data is called a parameter. This is the definition that carries through the whole book.
13. A straight-line model has two parameters. A large language model has hundreds of billions. There is no difference in kind, only in count.

PLAIN46.3.2 a picture in your head#

1. Picture a long straight stick and a scattering of nails hammered into a board at different heights.
2. You must hold the stick against the board so it passes as close as possible to all the nails at once.
3. You have exactly two freedoms. You can slide the stick up and down, and you can tilt it.
4. Sliding up and down is b. Tilting is w. There is nothing else you can do to a straight stick.
5. Now the useful part. Suppose the nails clearly climb from left to right and your stick is flat. You can see immediately that tilting it will help.
6. Suppose the stick is at the right tilt but all the nails are above it. You can see that sliding it up will help.
7. That “you can see which way to move it” feeling is exactly the gradient. You already have the intuition. Later we just compute it instead of eyeballing.

Where this comparison breaks: with two freedoms you can see the whole board and judge both at once. With a million freedoms you cannot see anything. You can only ask, at your current position, which way is downhill, and take a step. Your entire view of the landscape is the slope under your feet.

PLAIN46.3.3 a worked example#

1. Here is the dataset used for the whole of this chapter. Five points. Learn them, because they come back three more times.

2. Plotted, they rise, but not smoothly. The fourth point dips.

  y
  6 |
  5 |         *     *      (3,5) and (5,5)
  4 |     *       *        (2,4) and (4,4)
  3 |
  2 | *                    (1,2)
  1 |
  0 +----------------------
      1   2   3   4   5   x

3. First, a guess. Try w = 1 and b = 0, the line y = x.
4. Predictions: 1, 2, 3, 4, 5. Actual values: 2, 4, 5, 4, 5.
5. It is too low at the start and too low in the middle. Not good.
6. Now do it properly. The best straight line under the usual error measure has a formula, and we work it by hand.
7. Average of the x values: (1+2+3+4+5) / 5 = 15 / 5 = 3.
8. Average of the y values: (2+4+5+4+5) / 5 = 20 / 5 = 4.
9. Now for each point, take x minus 3, and y minus 4, and multiply them.

10. Sum of the products: 4 + 0 + 0 + 0 + 2 = 6.
11. Sum of the squares: 4 + 1 + 0 + 1 + 4 = 10.
12. The slope is the first divided by the second: w = 6 / 10 = 0.6.
13. The height is the average y minus w times the average x: b = 4 - (0.6 x 3) = 4 - 1.8 = 2.2.
14. So the best line is y = 0.6x + 2.2. Those two numbers are the trained model. That is the whole model. Two numbers.
15. Its predictions on our five inputs: 2.8, 3.4, 4.0, 4.6, 5.2.
16. Compare to the actual values 2, 4, 5, 4, 5. Close, never exact. That is normal and expected.

PLAIN46.3.4 what is really happening inside#

1. Notice what happened there. We did not search. We solved.
2. For a straight line under squared error, there is a formula that lands on the exact best answer in one calculation. No stepping, no guessing.
3. This is called a closed-form solution. It exists for this problem because the problem is simple enough to solve with algebra.
4. Here is the important fact: for almost every model you will ever use, no such formula exists.
5. Add a bend to the curve, add a threshold, add a second layer, and the algebra stops closing. There is no formula to solve.
6. When you cannot solve, you search. You start somewhere, look at which way is downhill, and step. That is section 46.5.
7. So the line is the perfect teaching example, precisely because you can check the search against the exact answer.
8. We know the truth is w = 0.6, b = 2.2. When gradient descent creeps towards 0.6 and 2.2 later in this chapter, you will know it is working.
9. One more thing to notice. The model has no memory of the five points. After training, all that is left is 0.6 and 2.2. The data is thrown away.
10. That is true of every trained model, from this line to the largest network. The data shaped the parameters and then went home.

TECHNICAL46.3.5 the engineer’s version#

1. The method is ordinary least squares. Adrien-Marie Legendre published it in
   1805. Carl Friedrich Gauss published in 1809 and claimed use from 1795. The priority dispute is real and unresolved.
2. The closed form for simple linear regression: w equals the sample covariance of x and y divided by the sample variance of x, and b equals the mean of y minus w times the mean of x.
3. In matrix form for many features, the normal equation is theta equals the inverse of X-transpose-X, times X-transpose-y.
4. Computing that inverse directly is a mistake in practice. Numerically stable implementations use a QR decomposition or a singular value decomposition. The function numpy.linalg.lstsq uses SVD.
5. Cost: the normal equation is order n times d squared, where n is samples and d is features. At d equal to 100,000 features the d-squared term is fatal, which is one practical reason iterative methods win at scale.

import numpy as np
x = np.array([1, 2, 3, 4, 5], dtype=float)
y = np.array([2, 4, 5, 4, 5], dtype=float)
A = np.vstack([x, np.ones_like(x)]).T
w, b = np.linalg.lstsq(A, y, rcond=None)[0]
print(w, b)          # 0.6  2.2

6. Terminology, and be careful here. In statistics these two numbers are “coefficients” and the model is a “regression”. In machine learning they are “weights” and “bias”, and the model is a “linear layer with one output”. Same equation, two vocabularies, two research communities.
7. Parameter has a precise meaning: a value learned from data by the optimization procedure. A hyperparameter is a value you choose before training, which the optimizer never touches. The learning rate is a hyperparameter. The slope is a parameter. Confusing these two is the single most common vocabulary error in the field.
8. Scale check for intuition:

WORDS46.3.6 remember these#

1. Parameter — a number the system may change to fit the data — a learned component of theta, updated by the optimizer.
2. Hyperparameter — a setting you choose before training starts — a value outside theta, fixed during optimization, tuned on validation data.
3. Weight — how strongly an input pushes the output — the multiplicative coefficient applied to an input feature.
4. Bias — the fixed offset added regardless of input — the additive intercept term, allowing the fit to leave the origin.
5. Slope — how steep the line is — the first derivative of output with respect to input, constant for a linear model.
6. Closed form — an exact answer from one formula — an analytic solution, such as the normal equation, available only for simple models.

46.4 Loss: how you measure being wrong#

PLAIN46.4.1 in simple words#

1. To improve something you must first be able to score it.
2. The score here is called the loss. It is one number. It says how badly the current parameters do on the examples.
3. Lower is better. Zero would mean perfect on every example.
4. The obvious idea is to add up how far each prediction is from the truth.
5. But there is a trap. If one prediction is 3 too high and another is 3 too low, they cancel to zero, and the model looks perfect when it is not.
6. So you must remove the sign before adding. There are two natural ways.
7. First way: square each miss. A miss of 3 becomes 9. Squares are never negative, so nothing cancels. Average the squares. That is mean squared error.
8. Second way: drop the minus sign on each miss. A miss of minus 3 becomes 3. Average those. That is mean absolute error.
9. Both are sensible. They are not the same, and the choice changes what the model learns.
10. Squaring punishes big misses much harder. A miss of 10 counts a hundred times more than a miss of 1, not ten times more.
11. So squared error makes the model terrified of large mistakes, and willing to accept many small ones to avoid one large one.
12. Absolute error treats a miss of 10 as exactly ten misses of 1. It does not panic about outliers.
13. The loss is not a description of reality. It is your statement of what you care about. The model will optimize exactly what you asked for, including the parts you did not mean.

PLAIN46.4.2 a picture in your head#

1. Imagine you are a delivery company and you must score your drivers.
2. Scheme one: fine a driver the square of the minutes late. Five minutes late costs 25. Twenty minutes late costs 400.
3. Under that scheme a driver will do almost anything to avoid one very late delivery, and will happily be three minutes late on many.
4. Scheme two: fine one unit per minute late, always. Twenty minutes late costs 20, exactly four times what five minutes costs.
5. Under that scheme a driver optimizes total minutes and will let one delivery run very late if it saves a little time on several others.
6. Neither scheme is wrong. They encode different values.
7. And here is the point: the drivers are not cheating. They are doing exactly what you asked. If the result is bad, your scoring scheme was bad.

Where this comparison breaks: drivers understand the rule and plan around it. A model has no plan and no understanding. It only follows the slope of the number you defined. That is worse in one way, because it will exploit loopholes no honest person would notice, and better in another, because it is not trying to deceive you. The honest version: a model does not “game” the loss. The loss is its entire universe. There is nothing else to game.

PLAIN46.4.3 a worked example#

1. Take the five points again and our fitted line y = 0.6x + 2.2.
2. Compute every step of the mean squared error.

3. Error means predicted minus true. Sign kept for now.
4. Sum of the squared column: 0.64 + 0.36 + 1.00 + 0.36 + 0.04 = 2.40.
5. Divide by the number of points: 2.40 / 5 = 0.48.
6. Mean squared error is 0.48. Written as a formula:

MSE = (1/n) * sum over i of ( (w*x_i + b) - y_i )^2

n = 5
  = (1/5) * (0.64 + 0.36 + 1.00 + 0.36 + 0.04)
  = (1/5) * 2.40
  = 0.48

7. Now mean absolute error on the same fit. Drop the signs: 0.8, 0.6, 1.0, 0.6, 0.2. Sum is 3.2. Divide by 5: 0.64.
8. Note that the errors were not squared, so the answer is in the same units as
   25. An MAE of 0.64 means “typically wrong by about 0.64 marks”. An MSE of 0.48 is in squared marks, which is not a unit anyone can feel.
9. Taking the square root of MSE gives 0.693, the root mean squared error, back in the original units. That is why people quote RMSE in reports.
10. Now the demonstration that the choice matters. Add a sixth point that is wildly wrong: x = 6, y = 20. Perhaps a data entry error.
11. Refit with squared error. The best line becomes y = 2.629x - 2.533.
12. The slope went from 0.6 to 2.63. One bad point out of six dragged the whole line more than four times steeper.
13. Refit with absolute error. The best line becomes y = 1.5x + 0.5.
14. Still moved, but far less. The absolute-error fit largely ignored the outlier while the squared-error fit chased it.
15. That is not a small technical detail. That is the loss deciding what the model believes.

PLAIN46.4.4 what is really happening inside#

1. Now the idea that makes optimization possible: the loss surface.
2. The loss depends on the parameters. Change w, the loss changes. Change b, the loss changes.
3. So imagine a flat map. East-west is the value of w. North-south is the value of b. Every point on that map is one possible model.
4. At every point, compute the loss and use it as a height. Now you have a landscape.
5. High ground means bad parameters. Low ground means good ones.
6. Training is walking downhill on this landscape. That is the entire idea.
7. For our five points, here is the real landscape. Every number is a genuine mean squared error, computed for that pair of w and b.

        w=0.0   w=0.3   w=0.6   w=0.9   w=1.2
 b=4.4  1.36    2.35    5.32   10.27   17.20
 b=3.3  1.69    0.70    1.69    4.66    9.61
 b=2.2  4.44    1.47    0.48*   1.47    4.44
 b=1.1  9.61    4.66    1.69    0.70    1.69
 b=0.0 17.20   10.27    5.32    2.35    1.36
                     * lowest point: w=0.6, b=2.2

8. Read it like a contour map. The 0.48 in the middle is the valley floor. That is exactly the answer we computed by formula in section 46.3.
9. Look at the diagonal. Going from top-left to bottom-right, the values stay small: 1.36, 0.70, 0.48, 0.70, 1.36. That is a long shallow trough.
10. Going the other diagonal, values shoot up fast: 1.36 to 17.20.
11. So this valley is not a round bowl. It is a long narrow ravine, steep across and shallow along. Remember that shape. It causes most of the pain in section 46.5.
12. The reason the ravine is tilted is that w and b are not independent here. Making the line steeper and lowering it can partly cancel out.

TECHNICAL46.4.5 the engineer’s version#

1. The standard losses, with the situation each belongs to.

2. Mean squared error is the maximum likelihood estimator under the assumption that the errors are Gaussian with constant variance. That is where it comes from. It is not an arbitrary choice.
3. Mean absolute error is the maximum likelihood estimator under Laplace- distributed errors, and its minimizer is the conditional median rather than the conditional mean. That is exactly why it resists outliers.
4. MSE has a continuous derivative everywhere. MAE has a kink at zero where the derivative jumps from -1 to +1 and is undefined at the point itself. Implementations return 0 or a subgradient there. This is a real, if minor, optimization nuisance.
5. Huber loss, from Peter Huber’s 1964 paper, is quadratic within a threshold delta of zero and linear outside it. It gets MSE’s smooth gradients near the optimum and MAE’s outlier resistance far away. In object detection the same idea appears as smooth L1 loss.
6. For classification you almost never use MSE. Cross-entropy is standard, because its gradient with respect to the pre-activation of a softmax output is simply predicted probability minus target, which is clean and does not saturate. MSE combined with a sigmoid output produces vanishing gradients when the model is confidently wrong, which is the worst possible time.
7. The loss surface of linear regression under MSE is a convex quadratic, so it has exactly one minimum and no local traps. Its Hessian for our five points is the matrix with rows [22, 6] and [6, 2].
8. That matrix has eigenvalues 23.66 and 0.338. The ratio, 70.0, is the condition number. It is the numerical measure of the ravine we saw above, and it directly bounds how fast gradient descent can converge.
9. Observation tools: log the loss every step and plot it. In PyTorch a scalar loss.item() per step written to TensorBoard, Weights and Biases, or a plain CSV is enough. Anyone who trains without a loss curve is guessing.

WORDS46.4.6 remember these#

1. Loss — one number saying how wrong you are — a scalar objective evaluated on predictions and targets, to be minimized.
2. Mean squared error — average of the squared misses — the L2 loss, maximum likelihood under Gaussian noise, minimized by the conditional mean.
3. Mean absolute error — average of the misses, signs dropped — the L1 loss, minimized by the conditional median, robust to outliers.
4. Loss surface — the landscape of wrongness over parameter settings — the graph of the objective as a function of theta, in as many dimensions as there are parameters.
5. Outlier — a data point far from the rest — an observation with low probability under the assumed noise model, which dominates squared loss.
6. Cross-entropy — the standard score for choosing between categories — the negative log-likelihood of the correct class under the predicted distribution.

46.5 Gradient descent: walking downhill in fog#

PLAIN46.5.1 in simple words#

1. You are standing somewhere on a hillside. Thick fog. You can see nothing beyond your own boots.
2. You want to reach the lowest point in the valley.
3. You cannot see the valley. But you can feel the ground under your feet, and feel which way is downhill.
4. So you do the only sensible thing. You take a small step in the steepest downhill direction. Then you stop and feel again. Then step again.
5. Repeat a few thousand times and you will end up at the bottom of something.
6. That is gradient descent. That is the entire algorithm. Everything else in this section is detail.
7. Translate it. The hillside is the loss surface from section 46.4. Your position is the current values of the parameters.
8. “Feeling which way is downhill” means computing, for each parameter, whether increasing it makes the loss go up or down, and by how much.
9. That per-parameter answer is called the gradient. It is a list with one number per parameter.
10. The size of your step is called the learning rate. You choose it. It is a hyperparameter, and it matters enormously.
11. Every neural network you have heard of was trained this way. There is no more sophisticated principle hiding behind it.

PLAIN46.5.2 a picture in your head#

1. Stay with the fog, but make the ground more interesting.
2. Suppose you are in a long narrow ravine, steep on the sides, almost flat along the bottom, running away into the fog.
3. You feel the ground. The steepest direction is across the ravine, not along it, because the sides are steep and the floor is nearly level.
4. So you step across. Now you are on the other side, slightly lower. You feel again. Steepest is back across.
5. You zig-zag from wall to wall, making painfully slow progress along the valley floor, which is where you actually needed to go.
6. That is the single most common failure of plain gradient descent, and our five-point problem has exactly this shape. Its condition number is 70, which means the ravine is seventy times steeper across than along.
7. The fixes all amount to remembering which way you were already going, so the zig-zags cancel and the along-valley movement adds up. That is momentum.

Where this comparison breaks: a real hiker can see their boots, feel the whole ground under them, and take a step of any size. A model computes the slope at exactly one point and then assumes the ground keeps that slope for the whole step. If the step is long and the ground curves, that assumption is wrong and you can land higher than you started. That is not a metaphor for divergence. It is literally what divergence is.

PLAIN46.5.3 a worked example#

1. First, what “the slope” means for one parameter. Move w a tiny bit and see how much the loss moves.
2. At w = 0.6, b = 2.2, the loss is 0.48. At w = 0.7, b = 2.2, it is 0.58. The loss rose by 0.10 for a rise of 0.1 in w. So the slope is about +1.0.
3. Positive slope means increasing w increases the loss. So to reduce the loss, decrease w. Always move opposite to the slope. That is the rule.
4. The update rule, for every parameter:

new value = old value - (learning rate) * (slope for it)

5. For mean squared error on a line, the two slopes have exact formulas:

slope for w = (2/n) * sum of ( (w*x + b - y) * x )
slope for b = (2/n) * sum of ( (w*x + b - y) )

6. Now run it. Start at w = 0, b = 0, which is the flat line y = 0. Learning rate 0.02. Four full steps, all arithmetic shown.
7. Step 1. Predictions are all 0. Errors, predicted minus true, are -2, -4, -5, -4, -5.
8. Slope for w: (2/5) times [(-2)(1) + (-4)(2) + (-5)(3) + (-4)(4) + (-5)(5)] = (0.4) times [-2 - 8 - 15 - 16 - 25] = 0.4 times -66 = -26.4.
9. Slope for b: (2/5) times [-2 - 4 - 5 - 4 - 5] = 0.4 times -20 = -8.0.
10. Update: w = 0 - 0.02 times (-26.4) = +0.528. b = 0 - 0.02 times (-8.0) = +0.16.
11. Both went up, which is right, because the line started far too low and far too flat.
12. Now the whole run in a table. Every number is exact.

13. The slopes computed at each of those positions were:

14. Read the loss column. 17.20, then 5.58, then 2.35, then 1.45, then 1.19. It falls fast at first and then slows. That shape is normal and you will see it in every real training run.
15. Read the slope table. The slopes shrink towards zero. That is the ground flattening as you approach the valley floor.
16. But look at w. It has overshot. It is 1.02 and the true answer is 0.6. And b is only 0.34 when the answer is 2.2.
17. That is the ravine. The optimizer raced along the steep w direction and is crawling along the shallow b direction.
18. Continue the run and it does get there, slowly:

19. Two thousand steps to find two numbers that a formula gave us instantly in section 46.3. That is the price of not having a formula. For any model worth training, there is no formula, so you pay it.
20. Notice also that the loss stops at 0.48 and does not reach zero. 0.48 is the best a straight line can do on this data. The remaining error is not a failure of the optimizer. The model simply is not a perfect description of the world, and no amount of training fixes that.

PLAIN46.5.4 what is really happening inside#

1. Now the learning rate problem, with real numbers from the same dataset.
2. Too small: with a learning rate of 0.0001 instead of 0.02, every step is two hundred times shorter. You would need roughly 400,000 steps instead of 2,000. It works. It just wastes your week.
3. Too large: here is what happens at 0.09.

4. The loss is going up, and it is going up faster each step. The parameters are flinging from one side of the ravine to the other, further each time.
5. This is divergence. Left alone it ends in numbers too large to represent, and the loss prints as nan, meaning not a number.
6. Seeing nan in a loss log means, in the overwhelming majority of cases, the learning rate was too high. Halve it and restart.
7. For this problem the exact boundary is computable: any learning rate below 0.0845 converges, and anything above diverges. At 0.02 we were safe. At 0.09 we were not.
8. Now a fact worth knowing. That boundary depends on how the input numbers are scaled. Our x values run 1 to 5, so their average is 3, far from zero.
9. Subtract 3 from every x, so they run -2 to +2 with an average of zero. Fit the same model. The condition number drops from 70 to 2, and the safe learning rate rises from 0.0845 to 0.5.
10. Nothing about the problem changed. We only recentred the inputs, and training became about thirty-five times easier.
11. That is why every practical guide tells you to normalize your inputs. It is not superstition. It reshapes the ravine into a bowl.
12. Finally, local minima. In a landscape with hills and dips, you might walk into a small dip that is not the true bottom, and get stuck.
13. In two dimensions this is a serious worry. In a million dimensions it is much less of one, and the reason is worth understanding.
14. To be a local minimum, the ground must curve upward in every single direction at once. With a million directions, that is like flipping a million coins and getting heads every time.
15. What you meet instead are saddle points: up in some directions, down in others. A saddle is not a trap. There is always a way out, though progress near one is slow.

TECHNICAL46.5.5 the engineer’s version#

1. The gradient of a scalar loss L with respect to a parameter vector theta is the vector of partial derivatives, written grad L. It points in the direction of steepest increase, so we step along its negative.
2. The update is theta at time t+1 equals theta at time t minus eta times grad L, where eta is the learning rate.
3. Convergence bound for a quadratic objective: plain gradient descent is stable if and only if eta is less than 2 divided by the largest eigenvalue of the Hessian. For our five-point problem that is 2 / 23.66 = 0.0845, which is exactly the boundary observed above.
4. The number of iterations to a fixed accuracy scales with the condition number, the ratio of largest to smallest eigenvalue. Ours was 70 raw and 2 after centring the inputs.
5. The three ways to choose how much data to use per step:

6. Mini-batch is universal in practice. Batch sizes are usually powers of two, which is a convention rather than a requirement, chosen because it maps cleanly onto hardware. Common values are 32, 64, 128, 256.
7. The gradient noise from small batches is not purely harmful. It acts as a mild regularizer and helps escape sharp regions. Very large batches often generalize slightly worse unless the learning rate is scaled up, an effect documented by Keskar and colleagues in 2017 and by Goyal and colleagues in the same year, whose linear scaling rule trained ImageNet in one hour.
8. The optimizer family, in order of invention:

9. Momentum keeps a running velocity v: v becomes beta times v plus the gradient, then theta moves by minus eta times v. Typical beta is 0.9, which means the step is roughly an average of the last ten gradients. In a ravine the across-valley components alternate in sign and cancel, while the along-valley components agree and accumulate.
10. RMSProp divides each parameter’s step by the square root of a running average of that parameter’s recent squared gradients. Parameters with consistently large gradients get shorter steps. Hinton introduced it in a 2012 Coursera lecture and never published it. It is cited as such.
11. Adam combines both: a running mean of the gradient, a running mean of the squared gradient, and a bias correction for the fact that both averages start at zero. Defaults from the paper: beta1 = 0.9, beta2 = 0.999, epsilon = 1e-8. These defaults are used almost unchanged a decade later.
12. AdamW, from Ilya Loshchilov and Frank Hutter, corrects how Adam applies weight decay. It is the standard for transformer training. Typical settings for large language model pretraining are beta2 = 0.95 and weight decay 0.1.
13. Practical starting learning rates, as of 2026, as rough guidance rather than law:

14. On saddle points: Dauphin and colleagues argued in 2014, in “Identifying and attacking the saddle point problem”, that saddles rather than local minima are the dominant obstacle in high-dimensional non-convex optimization. This is the standard modern view, though the loss landscape of deep networks remains an area of active research rather than settled theory.

WORDS46.5.6 remember these#

1. Gradient — which way is uphill, and how steeply, for every parameter at once — the vector of partial derivatives of the loss with respect to theta.
2. Gradient descent — repeatedly stepping downhill — iterative first-order minimization by moving against the gradient.
3. Learning rate — how big a step you take — the scalar eta multiplying the gradient in the update rule; a hyperparameter.
4. Divergence — the loss climbing instead of falling — instability from a step size above 2 over the largest Hessian eigenvalue, usually ending in NaN.
5. Momentum — remembering the direction you were already going — an exponentially weighted moving average of past gradients, coefficient beta.
6. Adam — the optimizer most people use by default — adaptive moment estimation, combining momentum with per-parameter gradient scaling.
7. Saddle point — flat ground that is up one way and down another — a stationary point whose Hessian has both positive and negative eigenvalues.

46.6 The vocabulary of training#

PLAIN46.6.1 in simple words#

1. This field reuses ordinary words with narrow meanings, and half of all confusion comes from that. Fix the words now.
2. A sample is one example. One email. One photograph. One patient record. Also called an instance, an observation, or a row.
3. A feature is one measured property of a sample. The number of links in the email. The brightness of pixel 47. The patient’s age.
4. A label is the right answer for that sample, provided by whoever made the data. “Spam”. “Cat”. “Recovered”.
5. A dataset is all the samples together, with their labels.
6. Now the split, which is the most important habit in the field. You cut the dataset into three parts and treat them differently.
7. The training set is what the optimizer sees. The parameters are fitted to it. Usually 70 to 80 percent of the data.
8. The validation set is held back. You never train on it. You use it to choose between models and settings. Usually 10 to 15 percent.
9. The test set is held back even harder. You look at it once, at the very end, to report an honest number. Usually 10 to 15 percent.
10. Why three and not two? Because if you tune your settings by watching the validation score, you are indirectly fitting to it, so it stops being honest. The test set stays clean because you never used it to decide anything.
11. Now the counting words. An epoch is one complete pass through the whole training set.
12. A batch is a small group of samples processed together, typically 32 to 512 of them.
13. An iteration, also called a step, is one batch going through the model and producing one parameter update.
14. So: one step handles one batch, and one epoch is many steps.

PLAIN46.6.2 a picture in your head#

1. A student is preparing for an exam and has a book of 500 practice questions with answers at the back.
2. Studying the questions is training. The 500 questions are the training set.
3. The real exam has different questions on the same subject. That is the test set. The student has never seen it.
4. Student A learns the ideas. On the exam, new questions, good marks. That is a model that generalizes.
5. Student B memorizes all 500 questions and their answers word for word. Perfect on practice, lost in the exam. That is overfitting.
6. Student C did not study enough and gets both the practice and the exam wrong. That is underfitting.
7. The tell is the gap. A big gap between practice performance and exam performance means memorization, not learning.

Where this comparison breaks: a student who memorizes knows they are memorizing. A model cannot tell the difference between a real pattern and a coincidence in the data, and does not experience one as different from the other. Worse, the analogy suggests memorization is always bad. It is not that simple. Very large modern models memorize a great deal of their training data and still generalize well, which classical theory did not predict. That gap between theory and observation is real, and it is active research, not settled.

PLAIN46.6.3 a worked example#

1. Take 50,000 labelled photographs. Split 80 / 10 / 10.
2. Training set: 40,000. Validation set: 5,000. Test set: 5,000.
3. Choose a batch size of 100.
4. Steps in one epoch: 40,000 / 100 = 400.
5. Train for 20 epochs. Total steps: 20 x 400 = 8,000 parameter updates.
6. The model has seen each photograph 20 times, and the parameters have been changed 8,000 times.
7. Now read a real training curve. Both numbers are losses. Lower is better.

loss
2.0 |T
    | T
1.5 |  T V
    |   TV
1.0 |    T V
    |     T  V
0.5 |      TT   V
    |        TTT   V     <- validation turns up here
0.2 |           TTTTT       V
    |                TTTT       V
0.0 +--------------------------------------
     0    5   10   15   20   25   30  epoch
     T = training loss    V = validation loss

8. Epochs 0 to 10: both fall together. This is real learning. Nothing to do.
9. Epochs 10 to 18: training keeps falling, validation flattens. The model is starting to fit noise. Watch it.
10. Epoch 18 onwards: training still falls, validation rises. This is overfitting, and it is unambiguous.
11. The correct action is to stop at around epoch 18 and keep those parameters. That is early stopping, and it is the cheapest regularizer there is.
12. Here is the opposite picture, underfitting: both curves flatten early and high, at say 1.4 and 1.45, and stay there. Small gap, bad numbers.
13. That is not a stopping problem. That means the model is too simple, or the features carry no signal, or the learning rate is wrong.
14. Read the two symptoms as a pair. Big gap, low training loss: too much capacity, add regularization. Small gap, high loss on both: too little capacity, make the model bigger.

PLAIN46.6.4 what is really happening inside#

1. Underneath the two symptoms sits one trade-off, and it has a name: bias and variance.
2. Bias, here, does not mean unfairness. It means systematic error from a model too simple to represent the truth.
3. A straight line fitted to data shaped like a curve has high bias. It is wrong in the same way every time, no matter how much data you give it.
4. Variance means sensitivity to the particular data you happened to get.
5. A very flexible model has low bias and high variance. Show it a slightly different sample of data and it produces a noticeably different answer.
6. Total error is roughly bias squared, plus variance, plus noise that nothing can remove.
7. Simple models sit at the high-bias end. Complex models sit at the high-variance end. Classically, the best model is somewhere in between.
8. Regularization is anything you do to push a flexible model back towards simplicity, so you get low bias without paying full variance.
9. The four you will actually use:
10. Weight decay, also called L2: add a penalty proportional to the sum of the squared weights. Big weights become expensive, so the model prefers small ones, and small weights make gentler, smoother functions.
11. L1: penalize the sum of the absolute weights instead. This drives many weights to exactly zero, which selects features and gives a sparse model you can read.
12. Dropout: during training, at every step, randomly switch off a fraction of the units, typically half. No unit can rely on any other being present, so the network cannot build fragile chains.
13. Early stopping: stop when validation loss stops improving, as in the curve above.
14. Separately from all of those, data augmentation: make more training data by altering what you have in ways that do not change the label. Flip the photograph left to right, crop it, rotate it slightly, change its brightness. A flipped cat is still a cat.
15. Augmentation is often the most effective of the lot, because the real problem is usually not enough data, and this is the cheapest way to get more.

TECHNICAL46.6.5 the engineer’s version#

1. Precise definitions, so the words never slip again:

2. Real scale: ImageNet’s ILSVRC training set is 1,281,167 images. At batch size 256 that is 5,004 iterations per epoch. The classic ResNet recipe runs 90 epochs, so about 450,000 parameter updates.
3. Splitting must respect structure. If several rows come from the same patient, the same user or the same day, they must not be split across train and test, or information leaks and your test number is a fantasy. Use grouped or time-based splits when the data has groups or an ordering.
4. K-fold cross-validation splits the data into k parts, trains k times with a different part held out each time, and averages. It gives a far more stable estimate on small datasets, at k times the cost. Common k is 5 or 10. It is rarely used for large deep networks purely because of cost.
5. L2 regularization adds lambda times the sum of squared weights to the loss. Its gradient contribution is 2 lambda w, so each step multiplies every weight by a factor slightly under one. With lambda 0.01 and learning rate 0.02, that factor is 0.99960, and 1,000 steps of pure decay would shrink a weight of 0.6 to 0.402.
6. In Adam, adding L2 to the loss and applying weight decay directly are not equivalent, because Adam rescales the gradient per parameter. AdamW, from Loshchilov and Hutter in 2017, decouples them. This is why AdamW replaced Adam for transformer training, and it is a real bug fix, not a preference.
7. Dropout, from Srivastava, Hinton and colleagues, published in JMLR in 2014, with an earlier 2012 preprint. At training time each unit is kept with probability 1-p. At inference nothing is dropped. Standard implementations use inverted dropout: divide activations by 1-p during training so the expected value matches, and inference needs no adjustment. Typical p is 0.5 for fully connected layers and 0.1 for transformer residual paths.
8. Batch normalization interacts with dropout badly, and modern convolutional networks often use batch normalization with little or no dropout.
9. Standard augmentations by field:

10. The classical bias-variance curve is U-shaped: error falls, then rises with model size. Established fact for classical models.
11. Active research, and important: Belkin and colleagues showed in 2019 that for very over-parameterized models the test error falls again past the interpolation point. This is called double descent, and it means the classical U-shape is only the left half of the real picture. The theory is incomplete and the practical implications are still debated.
12. Practically, as of 2026, the standard recipe for very large models is not “find the sweet spot on the U”. It is “make the model large, train on far more data, regularize lightly, and stop before the data runs out”. That recipe is empirical, and it works.

WORDS46.6.6 remember these#

1. Sample — one example — one row of the dataset, one point in feature space.
2. Feature — one measured property — one input dimension of the model.
3. Label — the right answer for a sample — the supervision target y.
4. Epoch — one full pass over the training data — ceil(n / batch size) optimizer steps.
5. Batch — the group processed together — the set of samples whose gradients are averaged into one update.
6. Overfitting — memorizing the practice questions — low training error with a large gap to validation error; fitting sample noise.
7. Underfitting — not learning enough — high error on both sets; insufficient model capacity or bad optimization.
8. Regularization — anything that pushes towards simpler solutions — a penalty or constraint that reduces effective capacity, trading bias for variance.
9. Data augmentation — making more examples by changing existing ones — label- preserving transformations that expand the effective training distribution.

46.7 Classical machine learning before neural networks#

PLAIN46.7.1 in simple words#

1. Neural networks are not the field. They are one family in it, and for a large amount of real work they are the wrong choice.
2. Linear regression predicts a number by adding up weighted inputs. That is section 46.3, with more than one input.
3. Logistic regression predicts a yes-or-no answer. It adds up weighted inputs the same way, then squashes the total into a probability between 0 and 1. Despite the name, it does classification.
4. A decision tree asks a series of yes-or-no questions and follows the answers to a leaf. “Is the income above 40,000? Then, is the age under 25?” You can read it out loud and a person will understand it.
5. One tree is easy to read and usually not very accurate. It clings to accidents in the data.
6. A random forest grows hundreds of different trees on different random parts of the data, and lets them vote. The mistakes are different, so they partly cancel.
7. Gradient boosting grows trees one after another, where each new tree is trained to fix the errors the current set makes. This is usually the most accurate method available for table-shaped data.
8. k-nearest neighbours does not train at all. To classify a new point, it finds the k most similar stored points and takes the majority answer.
9. k-means clustering takes unlabelled data and sorts it into k groups by similarity. No right answers are provided.
10. A support vector machine draws the dividing line that leaves the widest possible empty corridor between the two classes.
11. Every one of these is still in production somewhere, today, doing useful work.

PLAIN46.7.2 a picture in your head#

1. Think of a toolbox with a hammer, a screwdriver, a spanner and a saw.
2. Linear regression is the tape measure. Cheap, fast, always the first thing you reach for, and it tells you a lot before you do anything clever.
3. A decision tree is a flowchart pinned to the wall. Anybody can follow it and anybody can argue with it.
4. A random forest is asking three hundred people the same question and taking the majority. No individual is reliable. The crowd is.
5. Gradient boosting is one person answering, then a second person who only corrects the first one’s mistakes, then a third who corrects what is left.
6. k-nearest neighbours is asking your five closest neighbours what they did.
7. A neural network is a machine tool. Enormously capable, expensive to set up, and absurd for hanging a picture.

Where this comparison breaks: real tools have obvious jobs and you can see when you are using the wrong one. Here you cannot, because every method returns a number for every input, and a badly chosen method still returns confident answers. The only way to know is to try several and measure honestly on held- out data. The honest version: the choice is empirical, and anyone who tells you which method wins before seeing your data is guessing.

PLAIN46.7.3 a worked example#

1. Logistic regression, worked fully. Predict whether a student passes, from hours studied.
2. The model is: z = 0.8 times hours, minus 1.2. Then squash z with the sigmoid function, which is 1 divided by (1 plus e to the minus z).
3. Student studied 3 hours. z = (0.8 x 3) - 1.2 = 2.4 - 1.2 = 1.2.
4. Sigmoid of 1.2 = 1 / (1 + e^-1.2) = 1 / (1 + 0.3012) = 0.7685.
5. So the model says 76.85 percent chance of passing.
6. If the student did pass, the loss is minus the natural log of 0.7685, which is 0.263. Small loss, because the model was mostly right.
7. If the student failed, the loss is minus the log of (1 - 0.7685) = minus log of 0.2315 = 1.463. Much bigger loss. Correctly punished.
8. Now k-nearest neighbours, worked fully. New point at (3, 4). Five stored points with known classes:

9. With k = 3, the three closest are all B, so the answer is B.
10. With k = 5, the vote is three B against two A, so still B, but less confidently.
11. Notice there was no training. The distances were computed at question time, against every stored point. That is why it is slow to use and instant to “train”.
12. Notice also that the distances mix the two axes as if they were comparable. If the first axis were rupees and the second were years, this would be nonsense. k-nearest neighbours demands that you scale your features first.

PLAIN46.7.4 what is really happening inside#

1. Now the four kinds of learning, distinguished by what you give the system.
2. Supervised learning: every sample comes with the right answer. The system learns to map input to answer. Spam or not. Cat or dog. House price. This is most of the industry.
3. Unsupervised learning: no answers at all. The system finds structure in the data itself. Grouping customers. Compressing images. Spotting anomalies.
4. Self-supervised learning: the answers are generated automatically from the data by hiding part of it. Take a sentence, hide a word, and the label is the hidden word.
5. That last one deserves attention, because it is how every large language model is trained. Nobody labelled the internet. The data labels itself by the trick of prediction.
6. Self-supervised learning is why scale became possible. Human labelling costs money per example. Hiding a word costs nothing, so the amount of training data is limited only by how much text exists.
7. Reinforcement learning: no labels, only rewards. The system acts, receives a score, and must work out which of its actions caused the score.
8. The hard part of reinforcement learning is that the reward arrives late. You lose a game of chess at move 60. Which of the 60 moves was the bad one? That is the credit assignment problem, and it makes reinforcement learning far harder and far less stable than the other three.
9. One clean way to keep them straight: supervised has answers, unsupervised has none, self-supervised manufactures its own, reinforcement has only scores.

TECHNICAL46.7.5 the engineer’s version#

1. When each method is right:

2. Origins, with dates: decision trees as CART, from Breiman, Friedman, Olshen and Stone in 1984, and ID3 from Ross Quinlan in 1986. Random forests from Leo Breiman in 2001. Gradient boosting from Jerome Friedman in 2001. k-means from Stuart Lloyd in 1957, published 1982, named by James MacQueen in 1967. k-nearest neighbours from Evelyn Fix and Joseph Hodges in 1951, with the classic analysis by Cover and Hart in 1967.
3. The practical boosting libraries: XGBoost, from Tianqi Chen and Carlos Guestrin in 2016; LightGBM from Microsoft in 2017; CatBoost from Yandex in
   2017. Between them they won a very large share of Kaggle competitions on tabular data from 2016 onwards.
4. The honest note on tabular data, which is what most companies actually have. Established fact: for table-shaped data with mixed numeric and categorical columns, gradient-boosted trees have been the strongest default for a decade, and remain a leading choice in 2026. The 2022 paper by Grinsztajn, Oyallon and Varoquaux, “Why do tree-based models still outperform deep learning on typical tabular data”, is the standard citation, and gives the reasons: trees handle uninformative features, irregular target functions and unnormalized columns well, and neural networks handle none of them well.
5. Active research, and genuinely moving: tabular foundation models, starting with TabPFN in 2022 and TabPFN v2 published in Nature in January 2025, are pretrained transformers that make predictions on a new small table without fitting it in the usual sense. On benchmarks of small datasets, roughly under a few tens of thousands of rows, they now often beat tuned gradient boosting. On larger tables and heavily categorical tables, boosted trees still generally lead. This boundary is moving, so check the current benchmarks rather than trusting a book on this point.
6. SVMs use the hinge loss and maximize the margin, the distance from the decision boundary to the closest points, which are the support vectors. The kernel trick, from Boser, Guyon and Vapnik in 1992, computes inner products in a high-dimensional space without ever building the coordinates. The radial basis function kernel is the usual default.
7. Cost matters: the standard SVM solver is between quadratic and cubic in the number of samples, which is why SVMs are rare above roughly 100,000 samples and were displaced partly for that reason.
8. k-means minimizes within-cluster sum of squares. Lloyd’s algorithm is not guaranteed to find the global optimum, so you restart it several times. The k-means++ initialization of Arthur and Vassilvitskii, 2007, is the standard default and is what sklearn.cluster.KMeans uses.
9. Reinforcement learning landmarks: Q-learning from Chris Watkins in 1989, temporal difference learning from Richard Sutton in 1988, Deep Q-Networks from DeepMind in Nature in 2015, AlphaGo defeating Lee Sedol 4-1 in March 2016, and reinforcement learning from human feedback, which is how chat models are aligned after pretraining and is covered later in this book.

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split

Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2,
                                      random_state=0)
clf = GradientBoostingClassifier(n_estimators=200,
                                 learning_rate=0.05,
                                 max_depth=3).fit(Xtr, ytr)
print(clf.score(Xte, yte))

10. Note the hyperparameters in that snippet: number of trees, learning rate and tree depth. Boosting has a learning rate too, for the same reason gradient descent does, and 0.05 with 200 trees is a reasonable default.

WORDS46.7.6 remember these#

1. Logistic regression — a linear model that outputs a probability — a generalized linear model with a sigmoid link, fitted by minimizing binary cross-entropy.
2. Decision tree — a readable flowchart of yes-or-no questions — a recursive partition of feature space, split by information gain or Gini impurity.
3. Random forest — many different trees voting — bagged trees with random feature subsets at each split, reducing variance.
4. Gradient boosting — each new tree fixes the last one’s mistakes — stage-wise additive modelling fitting each learner to the current negative gradient.
5. Support vector machine — the widest empty corridor between two classes — a maximum-margin classifier with hinge loss, optionally kernelized.
6. Supervised learning — learning from labelled answers — estimating a mapping from inputs to targets given paired examples.
7. Self-supervised learning — the data provides its own answers — training on a pretext task built by hiding part of the input, such as next-token prediction.
8. Reinforcement learning — learning from scores, not answers — optimizing a policy to maximize expected cumulative reward from delayed feedback.

46.8 The neuron#

PLAIN46.8.1 in simple words#

1. An artificial neuron is a tiny arithmetic unit. It does exactly three things and then it stops.
2. One: it takes several numbers in, and multiplies each by its own weight.
3. Two: it adds all those products together, then adds one more number called the bias.
4. Three: it passes that single total through a fixed non-straight function, and that answer is its output.
5. That is all. Multiply, add, bend. There is no memory, no decision, no state.
6. Steps one and two are just the line from section 46.3 with more inputs: output = w1 x1 + w2 x2 + w3 x3 + b.
7. Step three, the bend, is the only genuinely new part, and section 46.9 explains why it is essential.
8. A weight says how much that input matters, and in which direction. Large positive means “this pushes the answer up hard”. Negative means it pushes down. Near zero means the neuron ignores it.
9. The bias is the neuron’s own starting opinion, applied before any input is considered.
10. A network is thousands to billions of these, arranged in rows, each row feeding the next.

PLAIN46.8.2 a picture in your head#

1. Picture a small committee vote where the members have unequal power.
2. Each member reports a number. Each member’s report is multiplied by their voting power, which is the weight.
3. The chair has a fixed opinion held before anyone speaks. That is the bias.
4. Everything is added into one total, and a rule converts the total into the committee’s answer. That rule is the activation function.
5. A member with a negative weight is one whose enthusiasm counts against the proposal. That is normal and useful.
6. A member with weight zero has been muted. Their report changes nothing.

Where this comparison breaks: committee members reason about the proposal and about each other. Nothing here does. The weights are fixed numbers that were tuned by an optimizer, not opinions held by anyone. And a real committee can deadlock or refuse to answer. A neuron always produces exactly one number, for every input, instantly, forever.

PLAIN46.8.3 a worked example#

1. One neuron, three inputs. Weights 0.5, -0.2 and 0.1. Bias 0.3.
2. Inputs arrive: 2.0, 3.0 and -1.0.

  x1 = 2.0 ---[ w1 = 0.5 ]---\
                              \
  x2 = 3.0 ---[ w2 = -0.2 ]----> (+) -> z -> f(z) -> output
                              /    ^
  x3 = -1.0 --[ w3 = 0.1 ]---/     |
                              b = 0.3

3. Multiply each input by its weight:
4. 2.0 x 0.5 = 1.0
5. 3.0 x -0.2 = -0.6
6. -1.0 x 0.1 = -0.1
7. Add them: 1.0 - 0.6 - 0.1 = 0.3
8. Add the bias: 0.3 + 0.3 = 0.6. Call this z, the pre-activation.
9. Now apply the bend. Three common choices, all on the same z = 0.6:

10. That is a complete neuron evaluation. Six multiplications and additions, then one function call.
11. Read the weights. Input one matters most and pushes up. Input two pushes down, less strongly. Input three barely matters at all.
12. And notice: input two had the largest raw value, 3.0, but contributed less than input one, because its weight was smaller. Size of input and importance of input are different things.

PLAIN46.8.4 what is really happening inside#

1. Geometrically, the weighted sum plus bias defines a plane cutting through the input space.
2. The neuron is measuring which side of that plane the input lies on, and how far from it.
3. The weights set the direction the plane faces. The bias sets how far the plane sits from the origin.
4. So a single neuron with a threshold is a straight-line separator, and nothing more. It is exactly the 1958 perceptron.
5. This is why one neuron cannot do XOR: one plane, two regions, and XOR needs the two “1” corners on the same side, which no plane achieves.
6. Layers fix it, because the second layer separates a space that the first layer has already bent.
7. Now the honest part about the brain. A neuron in your head is a living cell. It fires discrete spikes whose timing carries information. It has thousands of synapses with their own chemistry, it changes its own structure, it is regulated by hormones, and it is affected by nearby cells that are not even neurons.
8. The artificial version is a weighted sum and a bend. It does not spike, has no timing, no chemistry and no internal state.
9. The name came from the 1943 McCulloch and Pitts paper, which was genuinely an attempt to model nerve cells. The name stuck long after the modelling ambition was dropped.
10. The honest version: the brain analogy is historical and, today, mostly marketing. Calling it a neuron tells you where the idea came from. It tells you nothing about how brains work, and nothing about how networks work. Real computational neuroscience uses far more detailed models and does not use these.

TECHNICAL46.8.5 the engineer’s version#

1. Formally, for input vector x in R^n, weight vector w in R^n and bias b in R: z = w-transpose x + b, and a = f(z), where f is the activation.
2. z is the pre-activation or logit. a is the activation or unit output.
3. Cost per neuron: n multiply-accumulate operations, counted as 2n floating-point operations, plus one activation evaluation.
4. Storage per neuron: n + 1 parameters. In float32 that is 4(n+1) bytes; in bfloat16, 2(n+1) bytes.
5. The set of points where z equals zero is a hyperplane of dimension n-1. The vector w is its normal, and the perpendicular distance from the origin is the absolute value of b divided by the Euclidean norm of w.
6. The original 1958 perceptron used the Heaviside step as f, which has a derivative of zero everywhere it is defined and is undefined at zero. That is precisely why perceptrons could not be trained by gradient descent, and why the switch to smooth activations mattered so much.
7. Biological comparison, with real figures:

8. The human brain has roughly 86 billion neurons, a figure from Suzana Herculano-Houzel’s 2009 work, with of the order of 100 trillion synapses. Parameter counts of large models are sometimes compared to synapse counts. That comparison is not meaningful, because the units are not comparable, and you should treat it as a marketing claim.
9. Spiking neural networks and neuromorphic hardware, such as Intel’s Loihi 2 and IBM’s TrueNorth, do model timing and spikes. They are active research and are not what runs any deployed large model today.

WORDS46.8.6 remember these#

1. Neuron / unit — a weighted sum followed by a bend — a parameterized affine map composed with a fixed non-linearity.
2. Weight — how much an input matters and in which direction — a learned coefficient in the affine map.
3. Bias — the unit’s fixed starting offset — the additive constant, shifting the hyperplane off the origin.
4. Pre-activation — the total before the bend — z, the logit; the affine output before f is applied.
5. Activation — the neuron’s final output — a = f(z), the value passed to the next layer.
6. Hyperplane — the flat surface a neuron measures distance from — the set where w-transpose x + b equals zero, of dimension n-1.

46.9 Activation functions#

PLAIN46.9.1 in simple words#

1. Every neuron ends with a bend, and without the bend the whole network is worthless. Here is the proof, in two lines of algebra.
2. Take two layers with no bend. Layer one computes W1 x + b1. Layer two takes that and computes W2 (W1 x + b1) + b2.
3. Multiply out: W2 W1 x + W2 b1 + b2. Now let W’ be W2 W1 and let b’ be W2 b1 + b2. Both are just fixed matrices of numbers.
4. The result is W’ x + b’. That is one single layer.
5. So a hundred layers with no bend collapse into one layer. Depth buys nothing at all. Not less than expected. Exactly nothing.
6. The bend is what stops the collapse. Once you put a non-straight function between the layers, the multiplication cannot be folded up, and the second layer sees a genuinely reshaped space.
7. The function that provides the bend is called the activation function.
8. There are only a handful in common use, and almost all of modern practice is one of them: ReLU, which is “if the number is negative, make it zero; otherwise leave it alone”.
9. That is genuinely the whole of ReLU. One comparison and one selection.

PLAIN46.9.2 a picture in your head#

1. Picture folding a flat sheet of paper.
2. A linear layer stretches, rotates and slides the sheet. However many times you do that, the sheet stays flat, and you could have done it in one move.
3. ReLU folds it. Everything below the crease is flattened onto the crease line, and everything above is untouched.
4. Fold, stretch, fold, stretch, twenty times, and the sheet is a complicated crumpled shape that no single stretch could produce.
5. Now a straight cut through the crumpled sheet is, when you unfold it, a complicated curved boundary in the original space.
6. That is exactly what a deep network with ReLU is: many folds, then one straight cut at the end.

Where this comparison breaks: paper folds along a line you choose by hand. Here the crease positions are the weights and biases, and they are learned. Also, paper is two-dimensional, so you can see the folds. In 512 dimensions with thousands of folds there is nothing to see, and the intuition stops helping long before the mathematics does.

PLAIN46.9.3 a worked example#

1. The same input z, through four different activations.

2. Notice sigmoid never leaves the range 0 to 1, and tanh never leaves -1 to 1. Both are squashers.
3. Notice ReLU has no upper limit. Feed it 1000 and it returns 1000.
4. Notice GELU is almost ReLU for large inputs but is smooth near zero and allows a small negative output. That smoothness is why it is preferred in transformers.
5. Now the vanishing gradient, worked with numbers. The steepness of sigmoid at any point is s times (1 - s), where s is its output.
6. At z = 0, s = 0.5, so the steepness is 0.5 x 0.5 = 0.25. That is the maximum it ever reaches.
7. At z = 6, s = 0.9975, so the steepness is 0.9975 x 0.0025 = 0.0025. A hundred times flatter.
8. Now recall that backpropagation multiplies these steepness values together, once per layer. Best case, every layer contributes 0.25.

9. After ten sigmoid layers, the gradient reaching the first layer is at best about one millionth of the gradient at the output, and in practice far less.
10. So the early layers barely move. They are effectively frozen at their random starting values. The network trains its last two layers and nothing else.
11. That is the vanishing gradient problem, and it is the single technical reason deep networks did not work before roughly 2010.
12. ReLU’s steepness is exactly 1 for every positive input. Multiply 1 by itself fifty times and you still have 1. The problem disappears.

PLAIN46.9.4 what is really happening inside#

1. So why did ReLU win, when it looks crude next to a smooth curve?
2. Reason one, the gradient. Slope of exactly 1 on the positive side, so gradients pass through many layers undamaged.
3. Reason two, speed. ReLU is one comparison. Sigmoid needs an exponential, which is many times more expensive, and it is evaluated billions of times.
4. Reason three, sparsity. Roughly half the units output exactly zero for a given input. Zeros propagate nothing and cost nothing, and the resulting sparse representations seem to help.
5. ReLU has one real failure, called the dying ReLU. If a unit’s weights land such that its pre-activation is negative for every input in the data, its output is always zero, its gradient is always zero, and it can never recover. It is dead for the rest of training.
6. Leaky ReLU fixes this by giving the negative side a small slope, typically 0.01, instead of zero. A dead unit still receives a trickle of gradient.
7. In practice, plain ReLU with sensible initialization dies rarely enough that leaky ReLU is optional rather than standard, which is a somewhat surprising empirical result.
8. GELU and SiLU are the modern smooth alternatives. Both look like ReLU from a distance, but curve gently through zero and dip slightly negative just below it.
9. There is no strong theory for why that small dip helps. The honest answer is that it measured better on transformer benchmarks and the field adopted it.

TECHNICAL46.9.5 the engineer’s version#

1. The reference table:

2. Where each is used today, as of 2026:

3. Derivatives, which is what backpropagation actually needs. Sigmoid: s(1-s), maximum 0.25 at z = 0. Tanh: 1 - tanh squared, maximum 1.0 at z = 0. ReLU: 1 for z above 0, 0 below, undefined at exactly 0, where implementations return 0 by convention.
4. Tanh’s maximum derivative of 1.0 versus sigmoid’s 0.25 is why tanh was preferred over sigmoid through the 1990s and 2000s. It is zero-centred, which also keeps the mean activation near zero. It still saturates at both ends, so it only postpones the problem.
5. GELU, from Dan Hendrycks and Kevin Gimpel in 2016, is z times the standard normal cumulative distribution function. The common tanh approximation is 0.5z(1 + tanh(sqrt(2/pi)(z + 0.044715 z^3))). Both forms are shipped in PyTorch, selected by the approximate argument.
6. SiLU, also called Swish, appeared in Elfwing, Uchibe and Doya in 2017 and in Ramachandran, Zoph and Le’s neural architecture search paper the same year.
7. SwiGLU, from Noam Shazeer in 2020, is a gated variant using SiLU. It is the feedforward activation in the Llama family and much of the current open model ecosystem. It uses three weight matrices rather than two, so implementations shrink the hidden dimension to keep the parameter count equal.
8. Softmax is not a per-neuron activation. It operates on a whole output vector, turning logits into a probability distribution. Implementations always subtract the maximum logit before exponentiating, to avoid overflow. This is called the log-sum-exp trick and it is an implementation detail every library handles for you.
9. The vanishing gradient problem was identified in Sepp Hochreiter’s 1991 diploma thesis and analysed by Bengio, Simard and Frasconi in 1994. The move to ReLU came through Nair and Hinton in 2010 and Glorot, Bordes and Bengio in 2011, and reached everyone through AlexNet in 2012.

WORDS46.9.6 remember these#

1. Activation function — the bend after the weighted sum — the elementwise non-linearity f applied to the pre-activation.
2. ReLU — negative becomes zero, positive passes through — the rectified linear unit, max(0, z), with derivative 1 on the positive side.
3. Vanishing gradient — the learning signal fading out before it reaches the early layers — the product of many derivatives below 1 decaying towards zero exponentially in depth.
4. Saturation — the region where a squashing function stops responding — where the derivative is near zero because the input is far from zero.
5. Dying ReLU — a unit stuck outputting zero forever — a unit whose pre-activation is negative across the entire data distribution, so its gradient is permanently zero.
6. Softmax — turning scores into probabilities that add to one — the normalized exponential over a logit vector, used with cross-entropy.

46.10 Layers and depth#

PLAIN46.10.1 in simple words#

1. A layer is a row of neurons that all look at the same inputs and all produce outputs at the same time.
2. Neurons in one layer do not talk to each other. They work in parallel, which is exactly why this runs so well on the hardware from Chapter 22.
3. Stack layers so each one’s outputs are the next one’s inputs, and you have a network.
4. The input layer is not really a layer of neurons. It is just the numbers you feed in. Counting it is a convention some books follow and some do not.
5. Hidden layers are the middle ones. They are called hidden because nothing outside the network ever sees their values.
6. The output layer is the last one, and its size is set by the task. One unit for a single number. Ten units for ten categories.
7. Width is how many neurons are in a layer. Depth is how many layers there are.
8. “Deep learning” means, roughly, more than two or three hidden layers. There is no official threshold. It is a marketing term that stuck.
9. What depth actually buys you is composition. Early layers find simple patterns, later layers combine those into complicated ones.
10. In a picture network, the first layer finds edges, the third finds corners and textures, the tenth finds eyes and wheels, the last finds faces and cars. Nobody designed that. It emerges from training.

PLAIN46.10.2 a picture in your head#

1. Think of a factory production line rather than one craftsman.
2. Station one takes raw metal and cuts flat pieces. That is all it does, and it does it to everything.
3. Station two takes flat pieces and bends them into brackets.
4. Station three joins brackets into frames. Station four fits frames into cabinets.
5. No station understands the cabinet. Each does one narrow transformation of whatever arrives.
6. Depth is the number of stations. Width is how many workers stand at each station doing that station’s job in parallel.
7. Adding stations lets you build more complicated products. Adding workers per station lets you build more different things at each stage.

Where this comparison breaks: a factory line is designed, and someone knows what each station is for. Here nobody assigns jobs. The stations arrive at their own division of labour during training, and often it does not decompose neatly at all. Some units respond to combinations of things that have no name. The tidy edges-to-faces story is real for convolutional image networks and was demonstrated by visualization work in the 2010s, but it is a summary of typical behaviour, not a law.

PLAIN46.10.3 a worked example#

1. A network for handwritten digits. Input is a 28 by 28 greyscale image, which is 784 numbers. Output is 10 scores, one per digit.
2. Shape: 784 inputs, then a hidden layer of 128, then a hidden layer of 64, then 10 outputs.

  [784 inputs] -> [128 hidden] -> [64 hidden] -> [10 outputs]
       |               |               |              |
    pixels          ReLU            ReLU          softmax

3. Parameter count, layer by layer. Each layer has (inputs x outputs) weights plus (outputs) biases.
4. Layer 1: 784 x 128 = 100,352 weights, plus 128 biases = 100,480.
5. Layer 2: 128 x 64 = 8,192 weights, plus 64 biases = 8,256.
6. Layer 3: 64 x 10 = 640 weights, plus 10 biases = 650.
7. Total: 100,480 + 8,256 + 650 = 109,386 parameters.
8. Note how lopsided that is. The first layer holds 92 percent of the parameters, purely because 784 inputs is a lot to connect to.
9. This is a real network. Trained for a few minutes it reaches roughly 97 to 98 percent accuracy on the MNIST test set.
10. Now change the shape and see the trade. Width 512 in one hidden layer: 784 x 512 + 512, then 512 x 10 + 10 = 401,920 + 5,130 = 407,050 parameters.
11. Four times the parameters, one layer fewer, and typically slightly worse accuracy on this task. Depth is usually the more efficient way to spend parameters, though not without limit.

PLAIN46.10.4 what is really happening inside#

1. The universal approximation theorem is the famous result here, and it is almost always quoted wrongly. Here it is stated correctly.
2. A feedforward network with a single hidden layer, containing enough neurons, and a suitable non-polynomial activation, can approximate any continuous function on a closed bounded region, to any accuracy you like.
3. Read the conditions again. Continuous. Closed and bounded. Any accuracy you like, but “enough neurons” is not bounded.
4. Now the three things it does not say, which is where the misuse happens.
5. It does not say how many neurons. “Enough” can mean exponentially many in the number of inputs. The theorem is silent on the count.
6. It does not say you can find those weights. It proves such weights exist. It gives no procedure, and gradient descent may never reach them.
7. It does not say anything about new data. Approximating a function on the region you sampled says nothing about behaviour outside it.
8. So the theorem is an existence result. It tells you the model family is rich enough. It does not tell you the model is learnable, efficient or useful.
9. The practical answer to why we use depth at all: for many functions, a deep network needs exponentially fewer units than a shallow one to reach the same accuracy. Depth is about efficiency, not possibility.
10. That is not folklore. There are proven separation results, for example Telgarsky in 2016, exhibiting functions a deep network represents with few units that any shallow network needs exponentially many units to match.

TECHNICAL46.10.5 the engineer’s version#

1. A fully connected layer with n inputs and m outputs has n x m weights and m biases, and costs 2nm floating-point operations per sample.
2. Layer terminology in code: nn.Linear(in, out) in PyTorch, Dense(units) in Keras, nn.Dense(features) in Flax. All the same object.
3. Depth in real architectures, with dates:

4. Depth beyond about 20 plain layers did not train at all before residual connections. ResNet, from Kaiming He and colleagues at Microsoft Research in 2015, adds the input of a block to its output, so the block only has to learn a correction. That single change took usable depth from about 20 to over 1,000 and is the most important architectural idea of the decade.
5. Approximation theory: George Cybenko proved the sigmoid case in 1989. Kurt Hornik generalized it in 1991, showing that the multilayer structure, not the choice of activation, is what gives universality. Leshno and colleagues showed in 1993 that a non-polynomial activation is the precise condition.
6. There are also dual results for width. Lu and colleagues showed in 2017 that networks of bounded width and unbounded depth are universal, with a width threshold around the input dimension plus four.
7. Established fact: depth gives exponential representational efficiency for some function classes, and residual connections make great depth trainable.
8. Active research: exactly why depth helps optimization, not just representation, and why over-parameterized networks generalize at all. There is no complete theory. Anyone claiming otherwise is overselling.

WORDS46.10.6 remember these#

1. Layer — a row of neurons computed together — an affine transform followed by an elementwise non-linearity.
2. Hidden layer — a middle layer nothing outside ever sees — any layer that is neither the input nor the output.
3. Width — how many neurons in one layer — the output dimension of that layer.
4. Depth — how many layers deep the network is — the number of composed non-linear transformations.
5. Universal approximation theorem — one wide layer can copy any smooth function — for any continuous f on a compact set and any epsilon, a sufficiently wide single-hidden-layer network exists within epsilon. An existence result only.
6. Residual connection — adding a block’s input to its output — a skip connection giving gradients a direct path, from ResNet in 2015.

46.11 The forward pass as matrix multiplication#

PLAIN46.11.1 in simple words#

1. Running a network on an input is called the forward pass. Numbers go in at one end and come out the other.
2. Doing it one neuron at a time, as in section 46.8, is correct but slow to describe and slower to compute.
3. Every neuron in a layer does the same thing to the same inputs, with different weights. That is exactly what a matrix multiplication is.
4. So stack the weights of all the neurons in a layer as rows of a grid. Then one grid-times-list operation computes the whole layer at once.
5. That grid is the weight matrix. Its height is the number of neurons and its width is the number of inputs.
6. The whole forward pass becomes: multiply by a matrix, add a bias list, apply the bend. Repeat once per layer. That is all a network is.
7. Chapter 22 explained the matrix multiplication and the chip that does it. This is the same operation, unchanged, on bigger grids.
8. That is not a coincidence or an analogy. It is the same arithmetic, and it is the entire reason graphics hardware runs this field.

PLAIN46.11.2 a picture in your head#

1. Imagine a spreadsheet where a whole column of results is computed by one formula dragged down.
2. You do not calculate each cell separately. You describe the pattern once and the machine fills the column.
3. A matrix multiplication is that idea in two directions at once, and on hardware built to do thousands of the multiplications simultaneously.

Where this comparison breaks: a spreadsheet computes cells one after another and just hides it from you. A graphics chip genuinely computes thousands at the same instant, in separate arithmetic units. The parallelism is real, not presentational, and that difference is worth roughly a thousandfold in speed.

PLAIN46.11.3 a worked example#

1. A two-layer network. Two inputs, three hidden units with ReLU, one output.
2. Input x = [1.0, 2.0].
3. Layer 1 weight matrix W1, three rows of two, and bias b1:

W1 = [  0.5  -0.2 ]        b1 = [  0.1 ]
     [  0.3   0.8 ]             [ -0.1 ]
     [ -0.4   0.1 ]             [  0.2 ]

4. Multiply row by row. Row 1: (0.5 x 1.0) + (-0.2 x 2.0) = 0.5 - 0.4 = 0.1. Add bias 0.1, giving 0.2.
5. Row 2: (0.3 x 1.0) + (0.8 x 2.0) = 0.3 + 1.6 = 1.9. Add bias -0.1, giving 1.8.
6. Row 3: (-0.4 x 1.0) + (0.1 x 2.0) = -0.4 + 0.2 = -0.2. Add bias 0.2, giving 0.0.
7. So z1 = [0.2, 1.8, 0.0]. Apply ReLU: nothing is negative, so a1 is unchanged at [0.2, 1.8, 0.0].
8. Layer 2 is one row of three: W2 = [1.0, -0.5, 0.25], and b2 = [0.05].
9. Multiply: (1.0 x 0.2) + (-0.5 x 1.8) + (0.25 x 0.0) = 0.2 - 0.9 + 0 = -0.7. Add bias: -0.7 + 0.05 = -0.65.
10. Output is -0.65. If this were a two-class problem, apply sigmoid: 0.343, so 34.3 percent.
11. Total parameters: (3 x 2 + 3) + (1 x 3 + 1) = 9 + 4 = 13. Total arithmetic: 9 multiply-adds. That is the whole network.
12. Written compactly, the entire forward pass is two lines:

a1  = ReLU( W1 @ x  + b1 )
out = sigmoid( W2 @ a1 + b2 )

PLAIN46.11.4 what is really happening inside#

1. Now count the work for a real network: the 784 to 128 to 64 to 10 digit classifier from section 46.10.
2. Each weight is used exactly once per input, in one multiply and one add. So count multiply-accumulate operations, then double for floating-point operations.

3. About 218,000 floating-point operations to classify one digit. A modern laptop core does billions per second, so this takes microseconds.
4. Useful rule to memorize: a forward pass costs roughly two floating-point operations per parameter. Here, 218,368 divided by 109,386 is 1.996.
5. That rule holds for dense layers and for transformers, so it lets you estimate the cost of any model you know the size of.
6. Now batching. Instead of one input vector, stack 64 inputs as columns of a matrix. One matrix-times-matrix computes all 64 forward passes.
7. Same total arithmetic, 64 times 218,368 which is about 14 million operations, but each weight is loaded from memory once and used 64 times.
8. Chapter 22 made the point that memory bandwidth, not arithmetic, is usually the limit. Batching is how you escape it, and it is why batch size is a performance knob and not only a statistical one.

TECHNICAL46.11.5 the engineer’s version#

1. For a layer with weight matrix W of shape (m, n), input X of shape (n, B) where B is the batch size, and bias b of shape (m, 1): Z = W X + b, broadcast over the batch dimension, then A = f(Z).
2. This is a GEMM, a general matrix-matrix multiply, the operation every BLAS library optimizes above all others. On NVIDIA hardware it runs through cuBLAS or CUTLASS, and on tensor cores where precision allows.
3. Cost: 2 m n B FLOPs. Memory traffic if unbatched and uncached: m n reads for the weights per sample. The arithmetic intensity, operations per byte, rises linearly with B until the compute units saturate.
4. Convention warning: mathematics texts write W x with x as a column vector. PyTorch’s nn.Linear stores weight of shape (out_features, in_features) but computes x @ W.T + b with x of shape (batch, in_features), so the batch dimension is first. Both conventions are in daily use and mixing them up is a common source of shape errors.
5. Real forward-pass costs, per single sample:

6. Note ResNet-50 breaks the two-FLOPs-per-parameter rule badly, at about 320 FLOPs per parameter. That is because convolution reuses each weight at every spatial position. The rule applies to dense layers, and transformers are mostly dense layers, which is why it works there.
7. The training-cost rule of thumb, from Kaplan and colleagues in 2020, is about 6 N D floating-point operations to train a model of N parameters on D tokens: roughly 2 N for the forward pass and 4 N for the backward pass.
8. Precision matters enormously here, and Chapter 22 gave the per-chip figures. The short summary without repeating them: halving the bits per number roughly doubles the achievable rate on the same silicon, which is why training moved from FP32 to mixed BF16 and inference is moving to FP8 and 4-bit integers.

WORDS46.11.6 remember these#

1. Forward pass — running the input through to the output — evaluating the composed function, layer by layer, with no gradient computation.
2. Weight matrix — the grid holding a layer’s weights — a matrix of shape (out_features, in_features).
3. GEMM — the matrix multiply that dominates the work — general matrix-matrix multiply, the core BLAS level-3 routine.
4. Batch dimension — many inputs processed at once — the axis over which samples are stacked to raise arithmetic intensity.
5. FLOP — one floating-point operation — one multiply or one add; a forward pass through dense layers costs about 2 per parameter.

46.12 Backpropagation#

PLAIN46.12.1 in simple words#

1. The network produced an answer. The answer was wrong by some amount. Now what?
2. Every weight in the network contributed something to that wrong answer. Some contributed a lot. Some contributed almost nothing. Some pushed in the right direction and were outvoted.
3. What we need is, for each weight, its share of the blame: if this one weight were slightly larger, would the error get better or worse, and by how much?
4. Backpropagation is the procedure that works out every weight’s share.
5. It works backwards. It starts at the output, where the error is directly measurable, and moves towards the input, one layer at a time.
6. At the output layer, the blame is obvious. Each output weight fed directly into the answer, so its share is easy to compute.
7. Then the trick. Having computed how much blame belongs to each unit in the last hidden layer, that blame is passed further back, split among the weights that fed those units, in proportion to how strongly each fed them.
8. Repeat until you reach the input. Now every weight in the network has a number attached: its share of the error.
9. Then every weight is nudged a little, against its share. Weights that made things worse move down. Weights that helped move up.
10. That is it. Compute forwards, assign blame backwards, nudge everything. Millions of times.
11. The reason this matters so much is efficiency. It computes the share for every weight in one backward sweep, costing about twice one forward pass, regardless of how many weights there are.

PLAIN46.12.2 a picture in your head#

1. A large company delivers a project three weeks late. The chief executive wants to know who is responsible.
2. She does not interview every one of ten thousand employees. That would take longer than the project.
3. Instead she asks four department heads: how much of the delay came through your department? They can answer, because they sit right next to the outcome.
4. Manufacturing says 60 percent, logistics 30 percent, design 10 percent.
5. Now each head asks their own team leads the same question, splitting only their own share. The manufacturing head splits her 60 percent among five teams.
6. Each team lead splits their portion among their engineers. In four rounds, every one of ten thousand employees has a number.
7. Four conversations per level, not ten thousand interviews. The saving is the whole point.
8. And the blame is proportional to influence. Someone whose work barely touched the delivery date receives a share near zero automatically.

Where this comparison breaks, and it matters. First, blame in a company is a judgement, and people argue. Here it is an exact derivative with a single correct value. Second, real blame implies fault. Here a “share” can be negative, which simply means that increasing that weight would reduce the error, so the weight should grow. Third, and most important, the shares are only valid for a tiny nudge. They tell you the effect of changing one weight by a hair while everything else stays fixed. Change all the weights at once by a large amount and the shares no longer describe reality. That is exactly why the learning rate must be small.

PLAIN46.12.3 a worked example#

1. Now the chain rule, which is the exact rule behind the blame splitting.
2. If a affects b, and b affects c, then the effect of a on c is the effect of a on b multiplied by the effect of b on c.
3. If turning a dial doubles a pressure, and doubling the pressure triples a flow, then turning the dial multiplies the flow by six. Effects along a chain multiply.
4. A network is a long chain, so a weight’s effect on the loss is the product of the effects at every step between that weight and the loss.
5. Here is the network we will do completely. Two inputs, two hidden units with sigmoid, one output unit with sigmoid, and the loss is half the squared error.

  x1=0.5 --w11=0.8--> h1 --v1=0.5--\
         \--w21=0.2-> h2 --v2=-0.3--> out -> loss
  x2=-1.0-/w12=-0.4, w22=0.6/
  biases: b1=0.1, b2=-0.2, c=0.2     target y = 1.0

6. Forward pass. Hidden unit 1: z = (0.8 x 0.5) + (-0.4 x -1.0) + 0.1 = 0.4 + 0.4 + 0.1 = 0.9. Sigmoid of 0.9 = 0.710950.
7. Hidden unit 2: z = (0.2 x 0.5) + (0.6 x -1.0) + (-0.2) = 0.1 - 0.6 - 0.2 = -0.7. Sigmoid of -0.7 = 0.331812.
8. Output: z = (0.5 x 0.710950) + (-0.3 x 0.331812) + 0.2 = 0.355475 - 0.099544 + 0.2 = 0.455931. Sigmoid = 0.612048.
9. Loss = 0.5 x (0.612048 - 1.0)^2 = 0.5 x 0.150506 = 0.075253.
10. Backward pass, output layer first.
11. Effect of the output on the loss: predicted minus target = 0.612048 - 1.0 = -0.387952.
12. Effect of the pre-activation on the output, which is the sigmoid derivative: 0.612048 x (1 - 0.612048) = 0.237445.
13. Multiply them. This product is the output unit’s blame share, called delta: -0.387952 x 0.237445 = -0.092117.
14. Now each output weight’s share is that delta times the activation it multiplied:
15. Share for v1 = -0.092117 x 0.710950 = -0.065491.
16. Share for v2 = -0.092117 x 0.331812 = -0.030566.
17. Share for the output bias c = -0.092117, because a bias multiplies 1.
18. Now push the blame back to the hidden units. Each hidden unit’s blame is the output delta times the weight connecting them, times that unit’s own sigmoid derivative.
19. Hidden unit 1: -0.092117 x 0.5 = -0.046059. Its sigmoid derivative is 0.710950 x 0.289050 = 0.205500. Product: -0.009465.
20. Hidden unit 2: -0.092117 x -0.3 = +0.027635. Its sigmoid derivative is 0.331812 x 0.668188 = 0.221713. Product: +0.006127.
21. Note that hidden unit 2 got a positive share while unit 1 got a negative one, because its outgoing weight was negative. The sign carried through the multiplication automatically.
22. Now the input weights. Each is its unit’s delta times the input it multiplied:
23. w11 = -0.009465 x 0.5 = -0.004733. w12 = -0.009465 x -1.0 = +0.009465.
24. w21 = +0.006127 x 0.5 = +0.003064. w22 = +0.006127 x -1.0 = -0.006127.
25. Biases b1 = -0.009465 and b2 = +0.006127.
26. Every parameter now has a share. Update with a learning rate of 0.5, using new value = old value minus rate times share:

27. Run the forward pass again with the new numbers. Output is now 0.629935, up from 0.612048, and the loss is 0.068474, down from 0.075253.
28. It moved towards the target of 1.0. One step, and it worked.
29. Continue: after 5 steps the output is 0.688, after 20 steps 0.801, after 50 steps 0.874. Loss falls from 0.0753 to 0.0079.
30. That is training. Everything else is this, at scale.

PLAIN46.12.4 what is really happening inside#

1. Notice something about the arithmetic above. The value 0.710950 was used twice: once going forward, and once coming back.
2. That is the reason the forward pass stores its intermediate values. Every activation computed on the way in is kept, because the backward pass needs it.
3. This is why training uses far more memory than inference. Inference can discard each layer’s output as soon as the next layer has consumed it. Training cannot discard anything until the backward pass has been.
4. For large models this stored activation memory frequently exceeds the memory taken by the weights themselves.
5. The other thing to notice: at no point did anyone write down a derivative formula for the whole network.
6. Each small step knew only its own local rule. The sigmoid knows its own derivative. Multiplication knows its own. Addition knows its own.
7. The overall answer came from chaining those local rules together, mechanically.
8. That mechanical chaining is called automatic differentiation, and it is what PyTorch, JAX and TensorFlow actually implement. Backpropagation is one specific use of it.
9. Here is how the frameworks do it. As the forward pass runs, every operation records itself and its inputs onto a graph. Multiply here, add there, ReLU there. That is the computation graph.
10. When you ask for gradients, the framework walks that graph backwards. At each node it applies that operation’s known local derivative rule and multiplies it into what arrived from downstream.
11. The graph is built while the code runs, so an if statement or a loop in your model creates a different graph on different inputs, and it still works. That style is called define-by-run, and PyTorch popularized it.
12. So you never write a gradient. You write the forward computation in ordinary code, call .backward(), and every parameter’s share appears in its .grad field.

TECHNICAL46.12.5 the engineer’s version#

1. Notation for a network of L layers, with z the pre-activation, a the activation, and delta the error signal at a layer.
2. Output layer: delta_L = grad_a L, elementwise times f’(z_L). For the common pairing of softmax with cross-entropy, this simplifies exactly to predicted probability minus one-hot target, with no activation derivative left over.
3. Recurrence for earlier layers: delta_l = (W_{l+1}-transpose delta_{l+1}), elementwise times f’(z_l).
4. Parameter gradients: dL/dW_l = delta_l a_{l-1}-transpose, and dL/db_l = delta_l. Summed over the batch.
5. Read those four lines again. The forward pass multiplies by W. The backward pass multiplies by W-transpose. That is the whole structural symmetry, and it is why backward costs about the same as forward.
6. Cost: the backward pass is about twice the forward, because each layer computes both the gradient with respect to its input and the gradient with respect to its weights. Total training step is about three times a forward pass, giving the 2 N forward plus 4 N backward that produces the 6 N D rule.
7. Memory: activations for all L layers must be retained. Gradient checkpointing, from Chen and colleagues in 2016, stores only some layers and recomputes the rest during the backward pass, trading roughly 30 percent more compute for a large memory saving. It is standard for large models.
8. Automatic differentiation has two modes. Reverse mode, used here, costs one sweep per output and is efficient when outputs are few and inputs are many. Forward mode costs one sweep per input and is efficient in the opposite case. Training has one scalar loss and billions of parameters, so reverse mode wins by an enormous margin.
9. Reverse mode is Linnainmaa’s 1970 result. Applying it to networks is Werbos in 1974 and Rumelhart, Hinton and Williams in 1986.
10. Backpropagation is not finite differences. Nudging each weight and re-running would cost one forward pass per parameter, so 175 billion forward passes for GPT-3 instead of one backward pass. It is also not symbolic differentiation, which would produce an expression that grows explosively with depth. Automatic differentiation is a third thing: numerically exact, and linear in the size of the computation.

import torch
x = torch.tensor([0.5, -1.0])
W1 = torch.tensor([[0.8, -0.4], [0.2, 0.6]], requires_grad=True)
b1 = torch.tensor([0.1, -0.2], requires_grad=True)
h = torch.sigmoid(W1 @ x + b1)
v = torch.tensor([0.5, -0.3], requires_grad=True)
c = torch.tensor(0.2, requires_grad=True)
out = torch.sigmoid(v @ h + c)
loss = 0.5 * (out - 1.0) ** 2
loss.backward()
print(v.grad)     # tensor([-0.0655, -0.0306])
print(W1.grad)    # tensor([[-0.0047, 0.0095], [0.0031, -0.0061]])

11. Those printed gradients are exactly the shares computed by hand above. The hand calculation and the framework agree to the digits shown.
12. Practical detail: gradients accumulate by default in PyTorch, so optimizer.zero_grad() must be called each step. Forgetting it is one of the most common bugs in the field, and it shows up as training that starts fine and then degrades.

WORDS46.12.6 remember these#

1. Backpropagation — passing the blame backwards through the layers — reverse- mode automatic differentiation applied to a layered network.
2. Chain rule — effects along a chain multiply — the derivative of a composition is the product of the derivatives of its parts.
3. Delta / error signal — one unit’s share of the blame — the partial derivative of the loss with respect to that unit’s pre-activation.
4. Computation graph — the record of every operation performed — a directed acyclic graph of operations and tensors, built during the forward pass.
5. Automatic differentiation — chaining known local derivative rules mechanically — exact derivative evaluation at machine precision, linear in the cost of the original computation.
6. Gradient checkpointing — recomputing instead of storing — trading extra forward compute for reduced activation memory during backpropagation.

46.13 Training in practice#

PLAIN46.13.1 in simple words#

1. The theory so far says: start somewhere, compute gradients, step downhill. In practice, five extra things decide whether it works at all.
2. Initialization: what the weights are before training starts. They cannot all be zero, because then every neuron in a layer computes the same thing, gets the same gradient, and stays identical forever. Symmetry must be broken by randomness.
3. But the random numbers cannot be any size. Too large and each layer amplifies its input, so after twenty layers the numbers explode. Too small and each layer shrinks its input, so after twenty layers the signal is gone.
4. The fix is to scale the random numbers by the size of the layer, so each layer roughly preserves the scale of what passes through it.
5. Input normalization: rescale your input data so each feature has an average near zero and a similar spread. Section 46.5 showed this making the safe learning rate thirty-five times larger on a two-parameter problem.
6. Normalization inside the network: do the same rescaling to the values flowing between layers, not just at the entrance.
7. Gradient clipping: if the gradient is enormous, shrink it before taking the step, so one bad batch cannot destroy the model.
8. Learning rate schedules: do not use one step size for the whole run. Start small, grow, then shrink towards the end.
9. None of these change what the model can represent. All of them change whether you can find the weights.

PLAIN46.13.2 a picture in your head#

1. Think of a chain of loudspeakers, each feeding the next one’s microphone.
2. If every stage amplifies by 1.5, then after twenty stages the signal is 3,300 times louder and everything clips into noise.
3. If every stage attenuates by 0.7, after twenty stages the signal is one thousandth of its original size and is lost in the hiss.
4. What you want is every stage having a gain of almost exactly 1, so the signal arrives at the end intact.
5. Careful initialization is setting each amplifier’s gain to about 1 at the start. Normalization layers are automatic gain control, resetting the level at each stage while the system runs.

Where this comparison breaks: an audio chain carries one signal and the right gain is obviously 1. A network carries hundreds of dimensions, and the useful quantity is the variance across them, not the level of one. Also, the gradient flows in the opposite direction to the signal, and it has its own gain that can explode or vanish independently. Initialization schemes have to compromise between keeping the forward pass and the backward pass both well scaled, and in general you cannot have both exactly.

PLAIN46.13.3 a worked example#

1. Xavier initialization, from Glorot and Bengio in 2010: draw weights from a distribution with standard deviation equal to the square root of 2 divided by (inputs plus outputs).
2. He initialization, from He and colleagues in 2015: standard deviation equal to the square root of 2 divided by inputs. It is larger, to compensate for ReLU throwing away half the signal.

3. Rule: Xavier with tanh or sigmoid, He with ReLU and its relatives. Both ship in every framework and are usually the default already.
4. Gradient clipping by norm, worked. Suppose three parameters have gradients 3.0, 4.0 and 12.0.
5. The norm is the square root of (9 + 16 + 144) = square root of 169 = 13.0.
6. Clip threshold is 1.0. Since 13.0 exceeds 1.0, multiply every component by 1.0 / 13.0 = 0.07692.
7. New gradients: 0.2308, 0.3077, 0.9231. Their norm is exactly 1.0.
8. The direction is unchanged. Only the length was cut. That is the point: you still step the right way, just not off a cliff.
9. A warmup-plus-cosine schedule, with peak rate 3e-4, 2,000 warmup steps and 100,000 total steps:

10. Warmup exists because at step zero the weights are random, the gradients are large and meaningless, and a full-size step would throw the model somewhere useless before it has learned anything. Starting from zero and ramping up avoids that.

PLAIN46.13.4 what is really happening inside#

1. The two normalization layers, and exactly what each normalizes. People confuse these constantly.
2. Batch normalization, from Ioffe and Szegedy in 2015: for each feature, compute the average and spread across all the samples in the batch, then rescale that feature so it has mean 0 and variance 1.
3. It normalizes down the batch. Sample 5’s value for feature 3 depends on what samples 1 to 64 happened to be.
4. That has a strange consequence: a sample’s output depends on which other samples it travelled with. At inference you have no batch, so batch norm keeps a running average from training and uses that instead. This train versus inference difference is a genuine source of bugs.
5. Layer normalization, from Ba, Kiros and Hinton in 2016: for each sample, compute the average and spread across that sample’s own features, then rescale.
6. It normalizes across the features. Each sample is handled entirely alone, so the batch size does not matter and training and inference behave identically.
7. That independence is why layer normalization won in transformers and sequence models, where sequences have different lengths and batch statistics are unreliable. Batch normalization remains standard in convolutional image networks.
8. One sentence to remember: batch norm normalizes each feature over the batch; layer norm normalizes each sample over its features.
9. Both add two learned parameters per feature, a scale and a shift, so the network can undo the normalization where it is unhelpful.

TECHNICAL46.13.5 the engineer’s version#

1. Troubleshooting, which is the most useful table in this chapter:

2. The single best debugging habit: take one batch of eight samples and train until the loss reaches near zero. If a model cannot memorize eight examples, the bug is in your code, not your hyperparameters. This takes two minutes and saves days.
3. Schedules in common use: step decay, which multiplies by 0.1 at fixed epochs, the classic ImageNet recipe; cosine annealing, from Loshchilov and Hutter in 2016, now the default for transformers; and inverse square root, used in the original 2017 transformer paper with 4,000 warmup steps.
4. Typical warmup lengths for large language models are 1 to 2 percent of total steps, or a fixed 2,000 steps. This is a convention, not a standard.
5. Gradient clipping: clip by global norm, not per-parameter, so the update direction is preserved. A threshold of 1.0 is the near-universal default for transformer training.
6. Mixed precision is standard practice: keep a master copy of weights in FP32, compute in BF16 or FP16, and with FP16 scale the loss up before the backward pass to stop small gradients underflowing to zero. BF16 has the same exponent range as FP32 so it usually needs no loss scaling, which is why it is preferred on hardware that supports it.
7. Observation tools: TensorBoard or Weights and Biases for curves, torch.cuda.max_memory_allocated() for memory, nvidia-smi for utilization, and torch.profiler or Nsight Systems for finding the real bottleneck. Low GPU utilization almost always means the data loader, not the model.
8. Reproducibility caveat: seeding with torch.manual_seed is not sufficient for bit-identical runs, because some GPU kernels are non-deterministic by design. torch.use_deterministic_algorithms(True) forces determinism where possible, at some cost in speed.

WORDS46.13.6 remember these#

1. Initialization — the random starting values of the weights — a distribution chosen to keep activation and gradient variance stable across depth.
2. Xavier / Glorot initialization — scaling by inputs plus outputs — variance 2/(fan_in + fan_out), suited to tanh and sigmoid.
3. He initialization — scaling by inputs only, larger — variance 2/fan_in, compensating for ReLU zeroing half its input.
4. Batch normalization — rescaling each feature across the batch — normalizing over the batch dimension, with running statistics used at inference.
5. Layer normalization — rescaling each sample across its own features — normalizing over the feature dimension, identical in training and inference.
6. Gradient clipping — cutting the step length without changing direction — rescaling the gradient vector when its global norm exceeds a threshold.
7. Warmup — starting with tiny steps and ramping up — a linear increase of the learning rate over the first steps, before the main decay schedule.

46.14 The families of neural network#

PLAIN46.14.1 in simple words#

1. Everything so far described the feedforward network: layers in a line, every unit connected to every unit in the next layer. It is the default and the fallback.
2. Its weakness is that it treats every input as unrelated to every other. Move a picture one pixel to the right and it is a completely different input.
3. A convolutional network fixes this for images. Instead of one weight per pixel, it learns a small patch of weights and slides that same patch over the whole image.
4. So a pattern learned in one corner is recognized everywhere, and the number of weights collapses.
5. A recurrent network handles sequences, such as words or sound. It reads one item at a time and keeps a running summary of what it has read.
6. An autoencoder learns to squeeze data into a small representation and rebuild it. Useful for compression, denoising and finding structure.
7. A generative adversarial network trains two networks against each other: one makes fakes, one detects fakes, and both improve.
8. A diffusion model learns to remove noise, and generates by starting from pure noise and removing it repeatedly until an image appears.
9. A graph neural network works on data with connections, such as molecules or social networks, by passing messages along the edges.
10. The transformer replaced recurrent networks for almost all sequence work after 2017. Chapter 48 covers it properly.

PLAIN46.14.2 a picture in your head#

1. Think of a rubber stamp and an ink pad.
2. A convolutional filter is a stamp with a small pattern cut into it. You press it at every position on the page and record how well the pattern matched there.
3. One stamp finds one thing: a vertical edge, a spot of red, a diagonal line.
4. A layer has many stamps, so you get many maps of where each pattern occurred.
5. The next layer stamps patterns onto those maps, so it finds combinations of patterns. A corner is where a vertical edge map and a horizontal edge map both light up.
6. Layer after layer, the patterns become larger and more specific.

Where this comparison breaks: a stamp is a fixed pattern you carved. These are learned from data, and typically nobody can say in words what most of them detect. The early ones do look like edge detectors and colour blobs, reliably and across independently trained networks, which is a genuine and repeatable finding. The middle ones are much harder to describe, and the neat story that each unit detects one nameable thing is often false.

PLAIN46.14.3 a worked example#

1. One convolution, worked completely. A 5 by 5 image with a vertical edge in it: two dark columns then three bright ones.

image                     kernel (vertical edge)
  0   0  10  10  10        -1   0  +1
  0   0  10  10  10        -2   0  +2
  0   0  10  10  10        -1   0  +1
  0   0  10  10  10
  0   0  10  10  10

2. The kernel is 3 by 3, so it fits in the image at 3 x 3 = 9 positions.
3. Take the top-left position. The window covers rows 1 to 3, columns 1 to 3: all values are 0, 0, 10 on each row.
4. Multiply each window value by the kernel value in the same place and add:
5. Row 1: (0 x -1) + (0 x 0) + (10 x +1) = 10.
6. Row 2: (0 x -2) + (0 x 0) + (10 x +2) = 20.
7. Row 3: (0 x -1) + (0 x 0) + (10 x +1) = 10.
8. Total: 10 + 20 + 10 = 40. That is one output value.
9. Slide right by one and repeat. Do this for all nine positions:

output (3x3)
  40  40   0
  40  40   0
  40  40   0

10. Read it. The left columns are strongly positive, marking the edge. The right column is exactly 0, because that window was uniformly bright, and a uniform region produces no edge response.
11. Nine parameters found a vertical edge anywhere in the image. A fully connected layer would have needed 25 weights per output unit and would have had to learn the pattern separately at every position.
12. Now pooling, which shrinks the map. Take a 4 by 4 grid and keep the largest value in each 2 by 2 block:

  1  3 | 2  4          max-pool 2x2, stride 2
  5  6 | 1  2   ---->        6   4
  ------+------               7   9
  7  2 | 8  0
  1  4 | 3  9

13. The result is half the size in each direction, so a quarter of the values. It keeps the strongest response and discards where exactly it occurred, which makes the network tolerant of small shifts.

PLAIN46.14.4 what is really happening inside#

1. Recurrent networks and their fundamental limit, which is worth understanding properly because it explains why transformers exist.
2. A recurrent network reads word one, updates a hidden summary, reads word two, updates it again, and so on. The same weights are used at every step.
3. So the summary after 50 words has been through 50 multiplications by the same matrix.
4. If that matrix shrinks things slightly, the influence of word one is multiplied by a number below 1 fifty times and vanishes. If it grows things slightly, the influence explodes.
5. That is the vanishing and exploding gradient problem again, this time through time rather than through layers, and it is why plain recurrent networks cannot remember much past about ten steps.
6. The LSTM, from Hochreiter and Schmidhuber in 1997, fixes it with a separate memory line that information can travel along unchanged, plus three learned gates that decide what to write, what to keep and what to read. That memory path is additive rather than multiplicative, so gradients survive along it. LSTMs handle hundreds of steps.
7. But there is a second limit that no gate fixes: a recurrent network must process step 1 before step 2. It is sequential by construction.
8. That means it cannot use the parallel hardware from Chapter 22 across the sequence. Doubling your chips does not halve your training time.
9. The transformer’s central trick is to let every position look at every other position at once, in one matrix multiplication, with no sequential dependency. That is why it took over. Chapter 48 explains how.
10. Established fact: transformers replaced recurrent networks for nearly all language work from about 2018 onwards. Active research: state-space models such as Mamba, from 2023, revisit the recurrent idea with a formulation that can be parallelized during training. Whether they displace attention at scale is genuinely open as of 2026.

TECHNICAL46.14.5 the engineer’s version#

1. Summary of the families:

2. Convolution parameter count: a layer with C_in input channels, C_out output channels and a k by k kernel has C_in x C_out x k x k weights plus C_out biases, independent of image size. A 3x3 layer with 64 in and 128 out has 73,728 weights and is applied at every spatial position.
3. Convolution FLOPs: 2 x H_out x W_out x C_in x C_out x k x k. This is why ResNet-50 costs about 8.2 billion operations per 224 by 224 image with only 25.6 million parameters. Weights are reused, not multiplied.
4. Convolutional milestones: Fukushima’s Neocognitron in 1980, LeCun’s backpropagation-trained digit network at Bell Labs in 1989, LeNet-5 in 1998 reading cheques in production, AlexNet in 2012, VGG and GoogLeNet in 2014, ResNet in 2015.
5. What is actually implemented: most convolutional layers are lowered to a matrix multiply, either by the im2col transformation or by an implicit GEMM in cuDNN, so the underlying arithmetic is the same GEMM from Chapter 22. Winograd algorithms are used for small kernels to cut the multiply count.
6. Autoencoders: an encoder maps to a bottleneck of lower dimension, a decoder rebuilds, and the loss is reconstruction error. The variational autoencoder, from Kingma and Welling in 2013, adds a probabilistic latent distribution and enables generation. Autoencoder latents are used inside modern image diffusion models to work in a compressed space rather than pixel space.
7. GANs, from Ian Goodfellow and colleagues in 2014, train a generator and a discriminator in a minimax game. They produce images in a single forward pass, which is fast, but training is notoriously unstable and can collapse to producing a few outputs repeatedly. Diffusion models displaced them for quality from about 2021.
8. Diffusion models come from Sohl-Dickstein and colleagues in 2015, made practical by Ho, Jain and Abbeel’s denoising diffusion probabilistic models in 2020. A forward process adds Gaussian noise in steps; a network learns to predict and remove it; generation runs the process backwards from noise. Sampling originally took hundreds of steps; distilled and flow-matching variants now reach single-digit steps. This area is moving fast.
9. Graph neural networks aggregate each node’s neighbours’ features, transform the result, and repeat. Message passing was formalized by Gilmer and colleagues in 2017. They are used in molecular property prediction, and AlphaFold 2 in 2021 combined graph-like reasoning with attention.

WORDS46.14.6 remember these#

1. Convolution — sliding one small pattern over the whole image — a shared, local, translation-equivariant linear operator, implemented as a GEMM.
2. Kernel / filter — the small patch of learned weights — the k by k by C_in tensor slid across the input.
3. Pooling — shrinking the map by keeping the strongest response — a downsampling operation, usually 2x2 max or average with stride 2.
4. Feature hierarchy — edges first, objects later — successive layers representing progressively larger and more abstract patterns.
5. Recurrent network — reading a sequence while keeping a running summary — a network applying shared weights over time steps with a carried hidden state.
6. LSTM — a recurrent unit with a protected memory line — long short-term memory, with input, forget and output gates over an additive cell state.
7. Autoencoder — squeeze it small, then rebuild it — an encoder-decoder trained on reconstruction loss through a bottleneck.
8. Diffusion model — generating by repeatedly removing noise — a model trained to reverse a gradual noising process.

46.15 What a trained model physically is#

PLAIN46.15.1 in simple words#

1. After all the training, what do you actually have? A file.
2. The file contains numbers. Nothing else. No code, no logic, no rules you could read.
3. It is a long list of the weights and biases, in a known order, with a small index at the front saying which block of numbers belongs to which layer.
4. To use the model you need two things: this file, and the code that knows the architecture, so it can put each block of numbers in the right place.
5. Neither is useful alone. The file without the architecture is noise. The architecture without the file is an untrained network that outputs rubbish.
6. Training is the expensive process of producing the file. Inference is the cheap process of using it to answer one question.
7. Training happens once, on a cluster, for weeks. Inference happens billions of times, on one machine, in milliseconds.
8. The asymmetry between those two is the single most important economic fact about this field. Chapter 49 goes into what is inside the file in detail.

PLAIN46.15.2 a picture in your head#

1. Think of a printed dictionary.
2. Compiling it took a team of lexicographers many years, with arguments, revisions and enormous cost.
3. Looking up one word takes you four seconds and costs nothing.
4. The book is the model file. The compilation is training. The lookup is inference.
5. And note: you cannot recover the arguments, the drafts or the sources from the finished book. It contains the conclusions and nothing else.

Where this comparison breaks: a dictionary can be read by a person and checked entry by entry. A model file cannot. There is no entry for anything. The knowledge, if that word even applies, is spread across all the numbers at once, and no single number means anything on its own. That property is called distributed representation, and it is why interpreting these models is hard research rather than an afternoon of reading.

PLAIN46.15.3 a worked example#

1. File sizes, computed directly. Multiply the parameter count by the bytes per number.

2. The arithmetic is simply: FP32 is 4 bytes per number, BF16 and FP16 are 2, INT8 is 1, INT4 is half a byte.
3. That last row is why quantization matters. The same model drops from 140 GB to 35 GB, which is the difference between needing several datacentre cards and fitting on one.
4. Now the compute asymmetry, with the standard estimates. Training cost is about 6 times parameters times training tokens.
5. For an 8-billion-parameter model trained on 15 trillion tokens: 6 x 8e9 x 15e12 = 7.2e23 floating-point operations.
6. Inference cost for one token is about 2 times parameters: 2 x 8e9 = 1.6e10 operations.
7. The ratio is 4.5e13. Training that model costs about as much arithmetic as generating 45 trillion tokens with it.
8. So if the model serves a billion tokens a day, the training cost is repaid in arithmetic terms after about 45,000 days of service.
9. That is why nobody trains their own foundation model unless they are selling access to it, and why fine-tuning an existing file is the normal path.

PLAIN46.15.4 what is really happening inside#

1. Inside the file, the layout is boring by design.
2. A header holds the metadata: the name of each tensor, its shape, its data type, and the byte range where its numbers start and end.
3. Then a single long region of raw numbers, laid out so a program can memory- map the file and read a tensor without copying it.
4. The safetensors format, now the default for published models, is exactly that: a JSON header giving names, shapes and offsets, then raw bytes.
5. It replaced Python’s pickle format for a security reason. A pickle file can execute arbitrary code when loaded, so downloading a model was equivalent to running a stranger’s program. safetensors cannot execute anything.
6. What is not in the file: the training data, the optimizer state, the code, or any record of how the numbers were reached.
7. The optimizer state is worth noting. During training, Adam keeps two extra numbers per parameter, so the training checkpoint is roughly three times the size of the final model, or four if a master FP32 copy is kept. Those extra numbers are discarded before publication.
8. Inference needs the weights and one small activation buffer. Training needs the weights, the gradients, the optimizer state, and every stored activation. That is why a model you can run on one card may need eight cards to fine-tune.

TECHNICAL46.15.5 the engineer’s version#

1. Memory required, per billion parameters, as a planning table:

2. So full fine-tuning at 1 billion parameters needs roughly 16 GB before activations, against 2 GB to run it. A factor of eight, and this is exactly why parameter-efficient methods such as LoRA, from Hu and colleagues in 2021, exist.
3. Formats you will meet: safetensors for published weights, ONNX for framework-portable graphs, GGUF for quantized local inference in the llama.cpp ecosystem, and TensorRT engines for compiled NVIDIA deployment.
4. Established fact: a trained model is a fixed function. Given identical inputs, identical settings and deterministic kernels, it returns identical outputs. Chat models appear to vary because a random sampler is applied to their output distribution, not because the model changed.
5. Established fact: the model does not learn from your conversation. Its weights are read-only at inference. Anything that looks like learning within a session is context, not weight change.
6. Marketing claim: that a deployed model “improves as you use it”. In almost all cases the file is static and improvements arrive as a new file, trained separately, on data that may include logged interactions. That is a different and much slower loop, and it is worth insisting on the distinction.

WORDS46.15.6 remember these#

1. Model file / checkpoint — the file of trained numbers — a serialized tensor dictionary mapping parameter names to arrays.
2. Inference — using the trained model to answer — a forward pass only, with no gradients, activations discarded as soon as they are consumed.
3. Training — producing the numbers in the first place — repeated forward passes, backward passes and optimizer updates over a dataset.
4. Quantization — storing the numbers in fewer bits — reducing weight precision to INT8 or INT4 to cut memory and bandwidth, at some accuracy cost.
5. Safetensors — the safe modern weight file format — a JSON header of names, shapes and offsets over a raw tensor region, with no code execution.
6. Optimizer state — the extra numbers training needs — momentum and variance estimates, roughly two extra values per parameter for Adam.

46.98 Common wrong ideas#

1. Wrong: the network understands what it is doing. Right: it computes a function. Weighted sums and bends, repeated. “Understanding” has no agreed technical definition here, and using the word imports assumptions nobody has earned. Whether internal structures deserve richer descriptions is active research, not a settled fact.
2. Wrong: more layers is always better. Right: past a point, deeper networks get harder to optimize, overfit more and cost more, without gaining accuracy. Plain networks beyond about 20 layers did not train at all until residual connections in 2015, and even now the right depth is chosen by measurement, not by adding more.
3. Wrong: it learned by itself. Right: a person chose the architecture, the loss, the optimizer, the learning rate, the schedule, the regularization, the data and when to stop. The only thing the machine did was solve, by repeated small steps, the exact optimization problem it was handed.
4. Wrong: neural networks work like brains. Right: the resemblance stopped at the 1943 model of a neuron. Real neurons spike, have timing, chemistry and internal state, and change their own structure. An artificial unit is a weighted sum and a bend. The name is historical, and today it is mostly marketing.
5. Wrong: AI is just a big pile of if-statements. Right: there are no branches at all in a dense network’s forward pass. It is a fixed sequence of multiplications, additions and one elementwise function, executed in the same order for every input. That is the opposite of branching.
6. Wrong: training finds the best possible weights. Right: it finds some set of weights where the gradient is near zero, reached from where it started. A different random start gives different weights and often similar accuracy. There is no guarantee of optimality for any non-convex model.
7. Wrong: a lower training loss means a better model. Right: training loss can always be driven down by memorizing. The only number that means anything is performance on data the model has never seen and that you never used to make a decision.
8. Wrong: the model stores its training data, so it can be asked to recall it. Right: the data shaped the parameters and was then discarded. Verbatim recall of some training examples does happen, and is a real privacy concern, but it is a side effect of over-fitting particular sequences, not a lookup mechanism.
9. Wrong: bigger datasets always help. Right: more of the same distribution has diminishing returns, and more badly labelled data actively hurts. Data quality, coverage and correct labelling usually beat raw volume.
10. Wrong: gradient descent gets stuck in local minima all the time. Right: in very high dimensions the common stationary points are saddles, not minima, because a minimum requires upward curvature in every direction at once. The practical obstacles are ill-conditioning, poor scaling and bad learning rates far more often than local minima.

46.99 Chapter summary in 20 lines#

1. Machine learning writes a rule-shaped thing with adjustable numbers, and a procedure that tunes those numbers until outputs match examples.
2. It is curve fitting at heart, and saying so is accurate rather than dismissive; it is what lets you reason about the system.
3. The four pieces are always the same: a model, its parameters, a loss, and an optimizer that reduces the loss.
4. The history has two public failures. The perceptron of 1957 to 1958 was overclaimed, limited by Minsky and Papert in 1969, and funding collapsed.
5. Backpropagation existed from Linnainmaa in 1970 and Werbos in 1974 and was ignored until Rumelhart, Hinton and Williams published it in Nature in 1986.
6. In 2012 AlexNet won ImageNet with 15.3 percent top-5 error against 26.2 percent, trained on two consumer graphics cards, and the argument ended.
7. A parameter is a number the system may change; a hyperparameter is one you choose. Fitting y = wx + b to five points has exactly two parameters.
8. Loss is one number saying how wrong you are. Squared error punishes large misses hardest; absolute error resists outliers. The choice changes the fit.
9. The loss surface is a landscape over the parameters, and training is walking downhill on it in fog, feeling only the slope under your feet.
10. Gradient descent steps against the gradient by a learning rate. Too small crawls; too large diverges to NaN. Normalizing inputs reshapes the ravine.
11. Momentum, RMSProp and Adam all exist to fix the same problem: plain descent zig-zags across narrow valleys instead of running along them.
12. Split your data into training, validation and test sets, and decide nothing using the test set. A widening gap between training and validation loss is overfitting, and the cure is early stopping, regularization or more data.
13. Classical methods are not obsolete. Gradient-boosted trees remain a leading choice for table-shaped data in 2026, though tabular foundation models are now competitive on small datasets and the boundary is moving.
14. One neuron is a weighted sum plus a bias, then a non-linear bend. It is not a brain cell and the analogy is historical.
15. Without the bend, stacked layers collapse algebraically into a single layer, so depth would buy exactly nothing. ReLU won because its slope is 1.
16. A forward pass is a matrix multiply, a bias add and a bend, once per layer. It is the same matrix multiply as Chapter 22, which is why GPUs run this field.
17. Backpropagation assigns each weight its share of the error by passing blame backwards layer by layer, then nudges every weight against its share.
18. That is the chain rule applied mechanically, which frameworks implement as reverse-mode automatic differentiation over a recorded computation graph.
19. Convolutional networks share small filters across an image, recurrent networks carry a running summary and cannot be parallelized over time, and the transformer of 2017 replaced them for sequences. See Chapter 48.
20. A trained model is a file of numbers, useless without the architecture that reads it. Training costs roughly 6 N D operations once; inference costs about 2 N per token, forever. See Chapter 49.