The History of the Mathematics behind AI

First version 23 October 2026, last changed 24 October 2026 · Andreas Stockter

The mathematics behind AI fits into one sentence: A language model is a computational rule with a few hundred billion adjustable numbers that computes, from numbers standing for words, the probability of the next word; training sets these numbers so that the error of the prediction becomes smaller: measure the error, find the direction in which it decreases, take a small step, and do that many billions of times. Every word in this sentence, computational rule, probability, error, direction, step, has a history, and the tools that stand behind them were invented between 1614 and 1986, by people who measured stars, calculated games of chance, reckoned interest or described nerves and had no inkling of language models. The history of AI tells who built the models; this text tells what they calculate with.

It assumes of the reader only the four basic operations of arithmetic and nothing else. Each section introduces a mathematical idea where it arose, explains it with a small numerical example that can be worked through with pencil and paper, and shows where in a language model of 2026 it sits. Like the other chronicles on this site, the text follows a chain: each idea solves a problem that the previous one left open, and creates the next. The unknown calls for the coordinate system, the straight line for the curve, the curve for the derivative, the derivative for the descent, the descent for the matrix, and at the end stands the one formula with which the Transformer was presented in 2017. Whoever arrives there can read it and knows what a model does when it answers. What mathematics to this day does not explain about the models is in the last section.

Contents: 1 The unknown · 2 The straight line · 3 Logarithm and e · 4 The derivative · 5 Probability · 6 Least squares · 7 Gradient descent · 8 Vectors and matrices · 9 Logic and the neuron · 10 Perceptron · 11 Backpropagation · 12 Entropy · 13 Words as numbers · 14 Transformer · 15 Open questions

1. The unknown: Numbers, equations and the algorithm (to 1637)

The problem. Calculating means, to begin with, making a new number out of known ones: three baskets of four apples are twelve apples. The mathematics that language models need begins with the reverse question. One knows the result and seeks the number that produced it. A Babylonian clay tablet from about 1800 BC sets the problem that the area of a square and its side length together come to three quarters, how long the side is, and works out the answer, one half, in a sequence of steps that a pupil can still follow today. Another tablet from the same period gives the length of the diagonal of a square with side one to six decimal places, 1.414213, without knowing that this number can never be written as a fraction.¹ The calculation backwards, from result to cause, is the oldest tool in this story, and it remains its core to the end: to train a language model means to find, from billions of known sentences, the numbers that could have produced them.

The numbers. For that one first needs the numbers themselves, and for a long time they were incomplete. The Greeks calculated with ratios of line segments and knew neither zero nor numbers below zero. The Indian mathematician Brahmagupta gave in 628 the rules by which one calculates with both: a fortune of three and a debt of five together make a debt of two, and the product of two debts is a fortune, minus times minus is plus.² Zero as a digit and the nine other Indian digits came to Europe by way of the Arab world, where Leonardo of Pisa, called Fibonacci, recommended them to merchants in his book of calculation in 1202, and there they needed three centuries to displace the Roman numerals. Negative numbers are indispensable for everything that follows: the adjustable numbers of a language model are about half of them negative, and a negative weight means nothing other than “subtract that”. The numbers are best imagined as points on a line, zero in the middle, the positive ones to the right, the negative ones to the left; in the next section the number line becomes the axis.

The unknown. The calculation backwards got its language when the unknown received a name. Diophantus of Alexandria used around 250 a symbol of its own for the number sought, and Muhammad ibn Musa al-Khwarizmi wrote around 820 in Baghdad the book from whose title the word algebra comes, “al-jabr”, the restoration: he showed how one transforms an equation by doing the same on both sides until the unknown stands alone.³ An equation is an assertion that two calculations give the same result; the sign for it, two strokes of equal length, was introduced by the Welshman Robert Recorde in 1557, because “no two things can be more equal”. Solving means going through the calculation backwards. Whoever knows that three times a number plus two gives eleven first takes away the two, on both sides, and has: three times the number is nine; then divides by three and has: the number is three. Each operation is undone by its inverse, adding by subtracting, multiplying by dividing, and the order is the reverse of the forward calculation, as when taking off shoes and socks. The letter for the unknown that prevailed comes from René Descartes, who in 1637 fixed the last letters of the alphabet, x, y and z, for unknowns and the first, a, b and c, for known quantities; François Viète had in 1591 been the first to use letters for both, and so made it possible to calculate with general formulas instead of with individual numbers.⁴ Such a stand-in for a number is called a variable. A language model of 2026 is, in this language, an equation with a few hundred billion variables; they are called parameters, and training is the attempt to solve it.

Power and root. Two notations from the same period are needed constantly in the rest of the text. Descartes wrote for a number times itself a small raised number, x² for x·x and x³ for x·x·x; that is called a power, the raised number the exponent. x² is also called the square of x, the operation squaring. 3² is 9, 2³ is 8, 10⁶ is a million, and 2¹⁰ is 1024, a number known to everyone who has ever bought a hard disk. The inverse of squaring is the root: √9 is 3, because 3·3 = 9. Most roots do not come out exactly. √2 begins with 1.414213 and never ends, which the Pythagoreans, when they proved it, felt to be a scandal and the Babylonians a millennium and a half earlier had treated as an exercise in calculation: one approaches the number digit by digit, and the sixth digit suffices for every purpose. This thought, that a number which cannot be written down exactly becomes, by approximation, as accurate as one wishes, later carries the derivative, compound interest and the number e.

The algorithm. The second word that arose from al-Khwarizmi’s name is the most important in the text. The Latin form of his name, “Algorismi”, denoted at first calculating with the Indian digits and later every computational rule that leads to the result in finitely many unambiguous steps. The oldest example is found in Euclid around 300 BC: to find the greatest common divisor of 48 and 18, one subtracts the smaller number from the larger as often as possible, 48 minus twice 18 is 12, then the same rule with 18 and 12, remainder 6, then with 12 and 6, remainder 0; the last number before the zero, 6, is the divisor.⁵ An algorithm is such a procedure: a sequence of steps that can be carried out without thinking and without discretion, says unambiguously at every step what is to be done next, and halts after finitely many steps. The training of a language model is an algorithm with three steps that are repeated billions of times, and it dates, as will be seen, from the year 1847.

The boundary. Al-Khwarizmi’s transforming works as long as the unknown in the equation is only added, multiplied, squared, and the equation is not too tangled. How tangled it is, is measured by its degree, the highest exponent with which the unknown occurs: x² + 10x = 39 has degree two. For equations of the second degree the Babylonians already knew the solution, for the third and fourth degree Italians found it in the 16th century, and in 1824 the Norwegian Niels Henrik Abel proved that for the fifth degree there is no general formula for the solution. With an equation in which billions of unknowns sit in nested calculation steps, calculating backwards gets one no further; one needs a procedure that feels its way towards the solution as the Babylonians did towards √2. Sections 4 to 7 deal with that. Before that, the unknown must be given a picture.

Try it yourself. The programming language Python is a pocket calculator that can handle variables: after starting it, the prompt >>> appears, and 3*4+2 answers with 14, 2**10 with 1024, 9**0.5 with 3.0, (-3)*(-5) with 15. With x = 3 the variable gets a value, and 3*x+2 gives 11; whoever wants to calculate the equation backwards types (11-2)/3. That is enough to follow all the numerical examples in this text, and every further section ends with such an experiment.⁶

2. The straight line: Coordinates, functions and slope (1637–1748)

The problem. An equation describes a number. What a language model is to learn is a relationship: how does the price depend on the distance, the yield on the rain, the next word on the previous ones? To see a relationship one needs a picture, and the picture was supplied by Descartes in the same appendix of 1637 in which he fixed the letters. Every point in the plane gets two numbers, its distance to the right and its distance upwards from a fixed starting point, and every curve becomes an equation between these two numbers. Pierre de Fermat had written down the same idea a year earlier, but not published it.⁷ Since then geometry and calculation have been the same thing: one can answer a question about curves by calculating and check a calculation by a drawing. The two numbers of a point are called coordinates, the horizontal line on which the first is marked off the x-axis, the vertical one the y-axis, both together the coordinate system. The number line from section 1 is the x-axis, and negative coordinates lie to the left or below.

The function. The second concept is a machine. A taxi costs four euros basic fare and two euros per kilometre: for three kilometres ten euros, for five fourteen, for zero four. The rule “take the kilometres, double them, add four” makes exactly one output out of every input, and such a thing has been called a function since Leibniz, who used the word in his papers in 1673. Leonhard Euler wrote in 1734 the abbreviation f(x), “f of x”, the output of the machine f for the input x, and in 1748, in his Introduction to the Analysis of the Infinite, gave the function the leading role in mathematics that it has played ever since.⁸ The taxi function reads, in Descartes’s letters, y = 2·x + 4. If one plots for every input x the output y as a point, all the points lie on a straight line, the graph of the function. The line cuts the y-axis at 4, the basic fare, that is the intercept, and rises by 2 per kilometre, that is the slope: the change in the output divided by the change in the input that brought it about. Between kilometre 1 and kilometre 3 the price grows from 6 to 10, that is by 4 for a change in the input of 2; 4 divided by 2 is 2, the same everywhere on the line. A function of this form, y = a·x + b, is called a linear function; a is the slope, b the intercept, and the two together fix the line.

Several inputs. A taxi that also charges for the waiting time has two inputs: price = 2·kilometres + 0.5·minutes + 4. Each input is multiplied by its own number, the products are added, the constant is added on. That is a weighted sum, and it is the most frequent calculation in this story: the multipliers, 2 and 0.5, are called weights in AI, they say how strongly each input counts, and the constant, 4, is called the bias, the shift that is left over even with nothing but zeros. A weight of zero switches an input off, a negative weight makes it subtract. With a thousand inputs the sum has a thousand weights, and the graph can no longer be drawn, but the calculation remains the same: multiply, add, bias on top. An artificial neuron, as it was described in 1943 and is built to this day, calculates exactly this sum, and a language model consists of a few hundred billion such weights; section 8 shows how they are arranged into tables so that the sums can be worked out in bulk.

The boundary of the straight line. What a straight line cannot do is bend. A relationship that grows slowly at first and then fast, like a balance with compound interest, or that first rises and then falls, like the temperature over the year, fits no straight line, and with weighted sums alone one gets no further, because the sum of straight lines is again a straight line. The first curves with which mathematics learned to calculate were the powers from section 1, y = x², the parabola, and y = x³; Euler added in 1748 the curve that describes growth, and it is the subject of the next section. That straight lines can be made to bend if one inserts a simple distortion between two sums is the insight on which neural networks rest, and it comes in section 9.

Try it yourself. A spreadsheet draws any function. In column A the numbers 0 to 10, in cell B1 the formula =2*A1+4, drag the formula down, select both columns and insert a scatter chart: a straight line that begins at 4 and rises by 2 per step. Whoever replaces the 2 in the formula by 3 sees the line become steeper; whoever replaces the 4 by 0 sees it pass through the origin; and whoever enters =A1*A1 sees the first curve, the parabola.⁹

3. Growth and logarithm: Napier’s tables and Euler’s number (1614–1748)

The problem. Around 1600 astronomers calculated with numbers of seven and more digits, and every multiplication by hand took minutes and easily went wrong. Tycho Brahe’s observations, from which Johannes Kepler derived the orbits of the planets, demanded thousands of such calculations. The Scottish laird John Napier published in 1614, after twenty years of work, tables with which one replaces a multiplication by an addition: one looks up an auxiliary number for each of the two numbers, adds the auxiliary numbers and looks up the number that belongs to the sum. Henry Briggs set the tables up for the base ten in 1617, Kepler used them for his Rudolphine Tables of 1627, and Laplace said later that they had doubled the life of the astronomer. The slide rule of 1622, two sliding scales with Napier’s numbers, remained the tool of engineers until the pocket calculator of 1972.¹⁰ The auxiliary number is called the logarithm, and to understand it one needs the powers from section 1 once more, this time with all exponents.

Powers with any exponent. 2³ is 2·2·2 = 8 and 2⁴ is 16: each step in the exponent doubles. If one multiplies 2³ by 2⁴, one counts the twos together, 2⁷ = 128, so exponents add when powers are multiplied. If one goes along the series backwards, each step halves: 2² = 4, 2¹ = 2, 2⁰ = 1, 2⁻¹ = ½, 2⁻² = ¼. A power with exponent zero is one, with a negative exponent a fraction. And what is 2 to the power of one half? The number that multiplied by itself gives 2, that is √2 ≈ 1.414, for ½ + ½ = 1. In this way every number acquires a meaning as an exponent, and the function y = 2ˣ becomes a smooth curve that creeps flat along the zero on the left and climbs ever more steeply on the right: exponential growth, the curve of compound interest, of the bacterial culture and of computing power since 1965. The logarithm is the question of the exponent: the logarithm of 8 to the base 2 is 3, because 2³ = 8, that of 1024 is 10, that of 1 is 0, and that of ½ is −1. Because exponents add when numbers are multiplied, the logarithm of a product is the sum of the logarithms, and that is Napier’s trick. For numbers between 0 and 1 the logarithm is negative, and the closer the number moves to zero, the deeper it falls, without bound: the logarithm of one thousandth to the base 10 is −3, that of one millionth −6. This property, of making small numbers large and negative, is needed in section 12, when it comes to how surprised a model may be by a word.

The number e. Jacob Bernoulli posed a question about interest in 1683. A thaler at a hundred per cent interest a year becomes two thalers after a year. If the bank pays half of it every half-year, one gets 1.5 thalers after half a year and on that another half, 2.25 thalers. Quarterly: 2.44. Monthly: 2.61. Daily: 2.7146. Hourly: 2.7181. The number grows, but ever more slowly, and it stays below a limit that it never reaches, 2.718281828…, a number that like √2 never ends. Euler gave it the letter e in 1731 and calculated it to 23 places in 1748; it has been called Euler’s number ever since.¹¹ The function y = eˣ is the curve of growth that is compounded at every instant, and it has a property that all other bases lack and that section 4 explains: its slope at every point is equal to its value. That is why mathematicians build their formulas from eˣ rather than from 2ˣ or 10ˣ, and that is why e turns up billions of times a second in every language model. The logarithm to the base e is called the natural logarithm, written ln: ln e = 1, ln 1 = 0, ln 2 ≈ 0.693, ln ½ ≈ −0.693, ln 0.01 ≈ −4.6.

The sigmoid function. With eˣ and a division one can build a curve that squeezes every number, however large or small it is, onto the interval between 0 and 1: σ(z) = 1 divided by (1 + e⁻ᶻ). For z = 0, e⁰ = 1, so σ = 1/(1+1) = 0.5. For z = 2, e⁻² ≈ 0.135, so σ ≈ 1/1.135 ≈ 0.88. For z = −2, e² ≈ 7.39, so σ ≈ 1/8.39 ≈ 0.12. For z = 10, σ lies at 0.99995, for z = −10 at 0.00005. The curve looks like an S drawn out lengthways, rises steeply in the middle and flattens off to both sides; the Belgian mathematician Pierre-François Verhulst found it in 1838 as the curve of a population that first grows and then runs up against a limit, and called it logistic.¹² In AI it is called the sigmoid function, and it has two tasks: it turns an arbitrary number into something that can be read like a probability, and it is, as section 9 shows, the distortion that bends a straight line.

Softmax. If instead of one number one has several, say three scores 2, 1 and 0 for three possible next words, and wants to make from them probabilities that add up to 1, one takes e to the power of each number and divides by the sum: e² ≈ 7.39, e¹ ≈ 2.72, e⁰ = 1, sum 11.11; dividing gives 0.665, 0.245 and 0.090. All three are positive, they sum to 1, the order is preserved, and the gaps are stretched: a lead of one point becomes almost three times the probability. The procedure is called softmax, a soft form of the maximum, because it gives the largest value the most weight but does not set the others to zero; the formula comes from the physics of the 19th century, where it describes the distribution of particles over energy states, and was adopted for neural networks in 1989.¹³ If one divides the scores by a number T before exponentiating, the distribution becomes sharper or flatter: with T = 0.5 the numbers 2, 1, 0 become the values 4, 2, 0 and the probabilities 0.867, 0.117 and 0.016; with T = 2 they become 0.507, 0.307 and 0.186. This number is called, after physics, temperature, and it is the setting with which one makes a chatbot more predictable or more inventive. The raw scores before the softmax are called logits. A language model ends with exactly this calculation: for each of the roughly hundred thousand words it knows, it calculates a score, and softmax turns these into the probabilities from which the next word is drawn.

Try it yourself. A pocket calculator with the key xʸ shows Bernoulli’s limit: (1 + 1/12)¹² gives 2.613, (1 + 1/365)³⁶⁵ gives 2.7146, (1 + 1/1000000)¹⁰⁰⁰⁰⁰⁰ gives 2.71828. In Python, import math and then math.exp(1) give the number e, math.log(2) the natural logarithm of 2 and math.log(0.01) that of one hundredth. The softmax of the scores 2, 1, 0 is calculated with [math.exp(z)/(math.exp(2)+math.exp(1)+1) for z in (2,1,0)]; whoever halves or doubles all three numbers beforehand sees the temperature at work.¹⁴

4. The infinitely small: The derivative and the chain rule (1665–1821)

The problem. The slope of a straight line is the same everywhere, that of a curve changes from point to point. A falling stone gets faster at every instant; how fast is it right now? The question of the slope of a curve at a single point, and the related question of where a curve has its highest or lowest point, occupied the 17th century, and two men answered it independently of each other. Isaac Newton found the method in 1665 and 1666 as a student in Cambridge, when the university was closed because of the plague, let it circulate in copies and published it only decades later. Gottfried Wilhelm Leibniz found it in 1675 in Paris, invented the notation that is used to this day, and published in 1684 in the Leipzig Acta Eruditorum seven pages with the title “A new method for maxima and minima”.¹⁵ The dispute over priority, which the Royal Society decided in 1712 with a report that Newton as its president had written himself, poisoned mathematics on one side of the Channel for a century.¹⁶ The method itself is simpler than its reputation.

The slope at a point. Take the parabola y = x² and ask for the slope at the point x = 3, where y = 9. A slope needs two points. From x = 3 to x = 4, y grows from 9 to 16, by 7 for a step of 1: slope 7. That is the slope of the chord between the two points, not that of the curve at 3. So a smaller step: from 3 to 3.1, y grows from 9 to 9.61, by 0.61 for a step of 0.1, slope 6.1. From 3 to 3.01: y grows by 0.0601, slope 6.01. From 3 to 3.001: slope 6.001. The smaller the step, the closer the slope comes to the number 6, and it comes as close to it as one wishes without the step ever having to be zero. This number, the limit of the chord slopes, is the slope of the curve at the point 3, the slope of the straight line that just touches the curve there, the tangent. It is called the derivative of the function at this point. At x = 3 it is 6, at x = 1 it is 2, at x = 5 it is 10: always twice x. The derivative of x² is the function 2x, just as the derivative of x³ is the function 3x² and the derivative of the straight line a·x + b is the constant a, for the line has the same slope everywhere and b only raises it without tilting it. What Newton and Leibniz found were such rules, with which one can write down the derivative of any function built from powers, sums, products and chained functions without calculating limits. That Leibniz in doing so calculated with “infinitely small” steps, which were at one moment unequal to zero and at the next equal to zero, Bishop George Berkeley criticised in 1734 as “ghosts of departed quantities”; only Augustin-Louis Cauchy placed the method on the limit in 1821, as it is explained here, and the derivative got its name from Lagrange, who in 1797 spoke of the “derived function”.¹⁷

What the derivative says. The derivative is a number that answers a question: if I move the input a little way, how far does the output move, in proportion? A derivative of 6 means: one thousandth more input, about six thousandths more output. A derivative of −2 means: more input, less output, twice as fast. A derivative of 0 means: the output does not react to a small nudge at all, the curve is horizontal at this point, and that is the case exactly at the places where it has a peak or a valley. That was Leibniz’s title: whoever seeks the minimum of a function takes its derivative, sets the derivative equal to zero and solves the equation with al-Khwarizmi’s transforming. The parabola y = (x − 3)² has the derivative 2·(x − 3), which is zero at x = 3, and there lies its valley. Section 6 finds in this way the best straight line through a cloud of points, and section 7 shows what one does when the equation “derivative equal to zero” can no longer be solved.

The derivative of e to the power x. Euler proved what section 3 announced: the derivative of eˣ is eˣ itself. The growth curve to the base e climbs at every point as steeply as it is high; with every other base a troublesome factor comes in, with 2ˣ about 0.693, the natural logarithm of 2. That makes e the most convenient base for everything that is to be differentiated, and in AI that is everything. The derivative of ln x is 1/x, and the derivative of the sigmoid function from section 3 follows from its own value: σ·(1 − σ). At z = 0.5, σ ≈ 0.6225 and the derivative 0.6225·0.3775 ≈ 0.235; at z = 10 it is almost zero, for there the S-curve is flat. These two numbers are needed in section 11, when a network learns for the first time.

The chain rule. The rule on which the training of all today’s models rests is already found in Leibniz, and it deals with nested functions. If y depends on u and u on x, how does y depend on x? Imagine gearwheels: if u turns three times as fast as x and y twice as fast as u, then y turns six times as fast as x. Slopes of chained functions multiply. A numerical example: u = 2·x + 1 and y = u². At x = 1, u = 3 and y = 9. The derivative of u with respect to x is 2, the derivative of y with respect to u is 2u = 6, so the derivative of y with respect to x is the product, 12. The check with a small step: at x = 1.01, u = 3.02 and y = 9.1204, the output has grown by 0.1204 for a step of 0.01, ratio 12.04, close enough to 12. Leibniz’s notation turns the rule into the cancelling of fractions: dy/dx = dy/du · du/dx, the du seemingly cancels out. That is the chain rule, and it holds for chains of any length: ten nested functions, ten factors. A language model is a chain of a few hundred nested calculation steps, and the question of how its error depends on a weight deep in its interior is a chain rule with a few hundred factors. How one calculates it without calculating it a hundred billion times is the discovery of section 11.

Several inputs. The weighted sum from section 2 has many inputs, and its output has a slope of its own for each of them. The derivative with respect to one input while all the others are held fixed is called a partial derivative, written with a curly ∂ instead of the d. For the taxi price 2·kilometres + 0.5·minutes + 4 the partial derivative with respect to the kilometres is 2 and with respect to the minutes 0.5, the weights themselves. In training the opposite direction is of interest: how much does the error of the model change if one adjusts a single weight a little and leaves all the others alone? That is a partial derivative with respect to the weight, one per weight, with a large model a few hundred billion numbers, and the list of all these numbers gets a name in section 7.

Try it yourself. The slope of x³ at the point 2 can be closed in on with chords as above: in Python ((2+0.1)**3-8)/0.1 gives the value 12.61, ((2+0.001)**3-8)/0.001 the value 12.006, and the rule “derivative of x³ is 3x²” says 3·2² = 12. Whoever wants to check the chain rule calculates ((2*1.001+1)**2-9)/0.001 and gets 12.004.¹⁸

5. Calculable chance: Probability and Bayes (1654–1933)

The problem. A language model does not know which word comes next; it knows only which words are how probable. That one can calculate with chance at all is a late discovery, and it began at the gaming table. The Chevalier de Méré, a gambler at the court of Louis XIV, asked Blaise Pascal in 1654 how the stake was to be divided fairly if a game is broken off early, and Pascal solved the question in correspondence with Pierre de Fermat: one counts how the rounds still outstanding could turn out, and divides according to the share of the cases in which each wins. Christiaan Huygens made the first textbook of it in 1657; Girolamo Cardano, himself a gambler, had written down the rules a century earlier, but his book appeared only in 1663.¹⁹ The probability of an event is, when all cases are equally possible, the number of favourable cases divided by the number of all cases: a six with a die 1/6, an even number 3/6 = ½, a number below seven 6/6 = 1, a seven 0. It always lies between 0 and 1, and the probabilities of all possible outcomes together make 1.

Calculating with probabilities. Two rules suffice for almost everything. If two outcomes exclude each other, one adds: one or two with a die 1/6 + 1/6 = 1/3. If two events are independent of each other, one multiplies: two sixes with two dice 1/6 · 1/6 = 1/36. The list of the probabilities of all outcomes is called the distribution: with a fair die six times 1/6, with a loaded one perhaps 0.3 for the six and 0.14 each for the other five. If one event depends on another, knowledge of the one changes the probability of the other: if one knows that the die shows an even number, the probability of a six is no longer 1/6 but 1/3, one of three remaining cases. That is a conditional probability, written P(six | even), “probability of six, given even”. With it the multiplication rule can be extended to dependent events: the probability that first A and then B occurs is that of A times that of B, given A, and for a chain of three events P(A)·P(B | A)·P(C | A and B). This product rule is the form in which a language model rates a sentence: the probability of the first word, times that of the second, given the first, times that of the third, given the first two, and so on. A model that supplies a distribution for every next word thereby rates every possible text.

Expectation and large numbers. Huygens introduced the expected value, the average one has to expect in the long run: if a game pays twelve euros on a six and otherwise nothing, the expected value is 1/6 · 12 + 5/6 · 0 = 2 euros; whoever pays more than two euros as a stake loses in the long run. Each outcome is weighted with its probability and the products are added, a weighted sum as in section 2, in which the weights add up to 1. That the long run actually comes about was proved by Jacob Bernoulli in his “Art of Conjecturing”, which appeared in 1713 after his death: the more often one throws the die, the closer the share of sixes comes to 1/6, with as much certainty as one wishes. In 600 throws there are around a hundred sixes, seldom under 80 or over 120.²⁰ This law of large numbers is the reason why probabilities can be estimated from data, by counting: whoever counts in millions of texts how often the word “morning” follows “good” has a usable estimate of how probable it is. Abraham de Moivre found in 1733 that the deviations from the expected value in many trials always form the same bell-shaped curve, high in the middle, flat at the edges; Laplace proved in general in 1810 that the sum of many small chance influences gives this curve, and Gauss had used it in 1809 as the distribution of errors of measurement. It is called the normal distribution, and it is the shape one assumes when one knows nothing about errors except that many small causes make them.²¹

Bayes. The clergyman Thomas Bayes answered, in an essay that his friend Richard Price published in 1763 after Bayes’s death, the question that applies the calculation backwards to chance: one knows the probability of an observation, given a cause, and seeks the probability of the cause, given the observation. Laplace found the rule independently in 1774 and made it into a tool.²² It becomes clearest with counts instead of fractions. A disease strikes one person in a hundred; a test detects nine out of ten of the sick and gives a false alarm for one in ten of the healthy. What does a positive test mean? Of a thousand people ten are sick, the test finds nine of them; of the 990 healthy it gives an alarm for 99. So 108 people are tested positive, of whom 9 are sick: 9/108 ≈ 8 per cent, not 90. The rule in letters, P(sick | positive) = P(positive | sick) · P(sick) / P(positive) = 0.9 · 0.01 / 0.108 ≈ 0.083, says the same, and it is called Bayes’s theorem. Its content: what one held to be probable beforehand, the rarity of the disease, is not replaced by the observation but reckoned together with it; doctors who are supposed to do this reckoning fail regularly with fractions and seldom with counts.²³ In AI the theorem is present twice: as the principle that learning means reckoning a prior assumption together with data, and as the first method with which software judged language statistically, the spam filters since 2002, which calculate from the words of an email the probability that it is advertising.²⁴

The axioms. What a probability actually is, whether a share in the long run or a degree of confidence, remained disputed until Andrei Kolmogorov in 1933, in a book of 62 pages, pushed the question aside and laid down only which rules the numbers must satisfy: not negative, summing to 1, additive for incompatible events. Everything else follows from that, and both readings can use the rules.²⁵ A language model uses them like this: its last calculation is the softmax from section 3, which turns scores into a distribution over all the words it knows. Then a word is drawn, as one draws from an urn in which the slips are distributed according to the probabilities. That is why a chatbot answers the same question differently, and that is why it can also, with temperature zero, always take the most probable word, and then becomes predictable and monotonous. What the model does not do is look things up: it throws dice, with probabilities that it has learned. How one learns them, from data, with a straight line, is the next section.

Try it yourself. Any spreadsheet shows the law of large numbers. In cell A1 the formula =RANDBETWEEN(1,6), drag down to row 600, and in B1 =COUNTIF(A1:A600,6): around a hundred sixes. With the F9 key the dice are thrown afresh, and the number fluctuates around a hundred, but seldom far. Whoever takes only six rows sees that with small numbers anything can happen, even no six or three. The calculation with the test can be repeated with other numbers: if the disease strikes one in ten, of a thousand a hundred are sick, the test finds 90, of 900 healthy it gives an alarm for 90, and a positive test means one half.²⁶

6. The best line: Least squares and regression (1795–1922)

The problem. On 1 January 1801 Giuseppe Piazzi in Palermo discovered a celestial body that was no known star, followed it for six weeks across a small patch of sky and then lost it in the light of the Sun. The 24-year-old Carl Friedrich Gauss succeeded in computing, from a few observations beset with measurement errors, the orbit on which it would be found again, and in December 1801 and on New Year’s night 1802 the minor planet Ceres was found where his calculation predicted it.²⁷ The method behind it was published by Adrien-Marie Legendre in 1805 as an appendix to a treatise on the orbits of comets, under the name it bears to this day, “method of least squares”; Gauss wrote in 1809 that he had been using the principle since 1795, which Legendre held against him and which could never be entirely cleared up.²⁸ The question that both of them answered is the question of every training run: given measurements that do not fit a curve exactly, which curve fits best?

The data. Five pupils study for one, two, three, four and five hours and score 3, 5, 4, 7 and 8 points. The points do not lie on a straight line, but they rise, and one would like a line that describes the relationship and allows a prediction for six hours. Such a line, and in general any computational rule with adjustable numbers that is meant to describe data, is called a model. For each pupil the model makes a prediction, and the difference between prediction and measurement is called the error. The line y = x + 2 predicts 3, 4, 5, 6 and 7 points; the errors are 0, −1, 1, −1 and −1. To combine the errors into a single number one must not simply add them, for then −1 and 1 cancel out as if nothing had happened. Legendre and Gauss squared them: 0 + 1 + 1 + 1 + 1 = 4. The square makes every error positive, it punishes large errors disproportionately, an error of 2 counts four times as much as one of 1, and it yields a smooth function that can be differentiated, which the absolute value would not. The sum of the squared errors is the loss function of the model, in AI simply the loss, and the line with the smallest loss is the best one. For y = 2·x, without intercept, the predictions are 2, 4, 6, 8, 10 and the loss 1 + 1 + 4 + 1 + 4 = 11, worse. For y = x + 2.5 it is 3.25, better. Is there a still better line?

The solution. The loss depends on two numbers, the slope a and the intercept b, and is a number for every pair: a function with two inputs whose graph is a bowl that has a lowest point somewhere. At the lowest point both partial derivatives are zero, the one with respect to a and the one with respect to b, that is Leibniz’s rule from section 4. Writing down both derivatives and setting them equal to zero gives two equations for a and b, which can be solved with al-Khwarizmi’s rearranging, and the result has a form that can be learnt by heart. One computes the mean of the inputs, (1 + 2 + 3 + 4 + 5)/5 = 3, and that of the measurements, 27/5 = 5.4, then for each pupil the deviations from both: for the hours −2, −1, 0, 1, 2, for the points −2.4, −0.4, −1.4, 1.6, 2.6. The slope is the sum of the products of the two deviations, 4.8 + 0.4 + 0 + 1.6 + 5.2 = 12, divided by the sum of the squared deviations of the hours, 4 + 1 + 0 + 1 + 4 = 10: a = 1.2. The intercept follows from it, because the best line passes through the point of the two means: b = 5.4 − 1.2 · 3 = 1.8. The line y = 1.2·x + 1.8 predicts 3, 4.2, 5.4, 6.6 and 7.8, the errors are 0, −0.8, 1.4, −0.4 and −0.2, and the loss is 0 + 0.64 + 1.96 + 0.16 + 0.04 = 2.8, smaller than for any other line. That is the method of least squares: square the errors, add them up, differentiate, set to zero, solve. For six hours it predicts 9 points.

Regression and correlation. The method owes the name under which it goes in statistics and in AI to an error. Francis Galton measured the heights of parents and children in 1886 and found that the children of tall parents are indeed tall, but less tall than the parents, and the children of short parents less short: a “regression to the mean” that he took for a law of heredity and that in truth every line through a cloud of points shows whenever the points scatter.²⁹ The name regression stuck to the method of laying a line or another function through data in such a way that the squared error is smallest. Karl Pearson in 1896 gave the strength of a relationship a number between −1 and 1, the correlation, which is 1 when all points lie on a rising line, −1 for a falling one and 0 when the line has no slope.³⁰ With several inputs, hours of study, sleep, prior knowledge, the line becomes the weighted sum from section 2, and the method finds all the weights and the bias at once: the multiple regression of 1900 is, number for number, the artificial neuron, except that back then one solved the equations instead of feeling one’s way towards the solution.

The most probable explanation. Gauss gave the reason in 1809 why the squares, of all things: if the measurement errors are normally distributed, section 5, then the line with the smallest squared error is the one under which the observed measurements are most probable.³¹ Ronald Fisher made a general principle of this idea in 1922 and gave it its name in 1921: of all the settings of a model, choose the one that gives the observed data the greatest probability, the greatest likelihood.³² The principle holds for every model that outputs probabilities, hence also for a language model: the best setting is the one under which the texts one trains with are most probable. Because the probability of a text is, by the product rule, a product of very many small numbers, one takes the logarithm, which turns the product into a sum, section 3, and flips the sign so that the largest value becomes a smallest: the loss of a language model is minus the logarithm of the probability it gives to the correct words. Section 12 gives this number its name from communications engineering.

Learning by heart. A danger shows itself even with five points. Through five points one can always lay a curve with five adjustable numbers that hits all of them exactly, with loss zero; it snakes its way between the points and predicts minus twelve points for six hours. The model has learnt the data by heart instead of the relationship, and that is called overfitting. Against it, an old precaution of statistics helps: one holds back part of the data, trains on the rest and checks on the held-back part, the test data, whether the model also hits the unknown. That models with more adjustable numbers than data points can nevertheless predict well contradicts this experience, but is the normal case with language models; section 15 returns to it.

The limit. The method of least squares delivers the solution in one go, because the derivatives of the loss are straight lines and “derivative equals zero” leads to solvable equations. That holds for every model that consists of weighted sums, and for no other. As soon as a sigmoid curve sits between the sums, which according to section 2 is necessary for the model to be able to bend, the equations are no longer solvable, and for a bowl with a hundred billion directions, which on top of that is no longer a bowl but a mountain range, one needs the method that Cauchy described on three pages in 1847.

Try it yourself. Every spreadsheet computes the regression. The hours in column A, the points in column B, select both columns, insert a scatter chart, then right-click on a point, “Add trendline”, linear, with “Display equation”: y = 1.2x + 1.8. Whoever changes the fifth pupil’s 8 points into 20 sees the line tip and the influence that a single outlier gains through the squaring. In Python, import numpy and numpy.polyfit([1,2,3,4,5],[3,5,4,7,8],1) deliver the two numbers 1.2 and 1.8; with the 1 at the end replaced by 4 one gets the curve that hits all five points, and with numpy.polyval(..., 6) its prediction for six hours.³³

7. Downhill in small steps: Gradient descent (1847–1952)

The problem. When “derivative equals zero” yields no solvable equation, what remains is the way of the Babylonians: feeling one’s way. Augustin-Louis Cauchy, who had put the derivative on the footing of the limit in 1821, presented to the Paris Academy on 18 October 1847 a communication of three pages, “General method for the solution of systems of simultaneous equations”, which was motivated by the orbits of celestial bodies and contains the method with which every neural network has been trained since: to find the minimum of a function, compute at the current position the direction in which it falls most steeply, go a small distance in that direction and repeat.³⁴ Whoever stands on a mountain in fog and wants to get down to the valley does the same: he feels under his feet which way leads downhill, takes a step and feels again. He does not see the valley, and he does not need to see it.

One direction. With a single adjustable number the method can be worked through by hand. Let the loss function be L = (w − 3)², a parabola with its valley at w = 3, which one can see here and cannot see with a real model. Its derivative is 2·(w − 3), section 4. The rule is: new w = old w minus learning rate times derivative. One starts at w = 0, where the derivative is −6, so the curve falls towards the right, and chooses 0.25 as the learning rate: new w = 0 − 0.25 · (−6) = 1.5. There the derivative is −3, the next step leads to 2.25, then to 2.625, 2.8125, 2.906, 2.953, and every step halves the distance to the valley without ever quite reaching it, just as Bernoulli’s interest never quite reaches the number e. The learning rate determines the step size, and it is an art. With 0.5 one lands here, by chance, exactly at 3 in a single step. With 1 one jumps from 0 to 6, from 6 to 0 and so on, across the valley and back, for ever. With 1.1 every jump becomes larger than the one before, 6.6, −1.32, 8.18, −3.22, and the loss grows instead of falling. Steps that are too small cost time, steps that are too large destroy everything, and a good part of the experience in training large models is experience with this one number.

Many directions. A model has not one adjustable number but billions, and the loss is a function with billions of inputs. For each of them there is a partial derivative, section 4: how does the loss change if I adjust this one weight a little? The list of all these partial derivatives, one per weight, is called the gradient, and it points in the direction in which the loss rises most steeply; the opposite direction is Cauchy’s steepest descent. The rule stays the same, only for all the weights at once: each weight is reduced by the learning rate times its own partial derivative. Weights on which the loss depends strongly move a lot, the others little, and those with derivative zero stand still. The method is called gradient descent, or the gradient method, and its loop is the algorithm that section 1 announced: compute the loss, compute the gradient, take a step. What was a parabola in one dimension is, in billions of dimensions, a mountain range that nobody can see, with valleys that are not the deepest (local minima), in which the method can get stuck, with plateaus on which the gradient is almost zero and the steps become tiny, and with saddles where it goes uphill in one direction and downhill in another. That the method nevertheless finds usable valleys in these mountain ranges is a fact of experience, not a proven one, and section 15 returns to it. Jacques Hadamard described the method again in 1908, and Haskell Curry gave it its English name in 1944, steepest descent.³⁵

With randomness. Computing the loss over all the training data before taking a step is unaffordable with trillions of words. Herbert Robbins and Sutton Monro showed in 1951 that one need not know the derivative exactly: an estimate that is right on average and noisy in detail also leads to the minimum if the steps become smaller over time.³⁶ In practice one takes for each step a small random selection of the data, a mini-batch of a few hundred or thousand examples, computes the gradient only for these and takes the step; the next step takes the next selection. That is called stochastic gradient descent, and it is not only cheaper, the noise evidently also helps to get out of shallow valleys. One pass through all the data is called an epoch; large language models see most of their data only a single time. The step size is not chosen once and for all but varied: slowly increased at the start, then lowered over the course of the training, and since the 2010s moreover adapted for each weight individually. The method Adam, which Diederik Kingma and Jimmy Ba presented in 2014, keeps for each weight a moving average of its past gradients and of their squares and makes the steps smaller where the gradients fluctuate widely; it is to this day the method with which almost all large models are trained.³⁷ The basic idea remains Cauchy’s: downhill, step by step, in the fog.

What is still missing. Gradient descent presupposes that one has the gradient, and for a model with billions of weights in nested layers that is the real difficulty: every partial derivative is, by section 4, a chain rule with many factors, and computing billions of such chains one by one would be as expensive as the feeling one’s way itself. The solution that computes all the chains at once is backpropagation, section 11. Before that, the weights need an order in which billions of them can be written down and computed in one go.

Try it yourself. Feeling one’s way fits into a single column. In cell A1 the starting value 0, in A2 the formula =A1-0.25*2*(A1-3), drag downwards: 1.5, 2.25, 2.625, and after twenty rows 2.999997. Then replace the 0.25 by 0.5, 1 and 1.1. Whoever likes can change the loss function: for L = (w − 3)⁴ the derivative is 4·(w − 3)³, and the steps become very small near the valley, the plateau in miniature.³⁸

8. Numbers in rows: Vectors, matrices and graphics cards (1843–2012)

The problem. A taxi fare has two inputs, an image has a million, and a word in a language model is described by several thousand numbers. To compute with such quantities without naming every number individually, mathematics needs a notation in which a list of numbers is a single thing, and rules by which one computes with such things. Both arose in the middle of the 19th century, from questions that had nothing to do with data. William Rowan Hamilton was looking in 1843 for a way of multiplying rotations in space, found the quaternions and in 1846 gave the part that is a direction with a length the name vector, the carrier. Hermann Grassmann, a grammar-school teacher in Stettin, published in 1844 an “Ausdehnungslehre” (theory of extension), which contained the calculation with vectors in arbitrarily many dimensions and was written so incomprehensibly that nobody read it for thirty years.³⁹ A vector is, for this text, simply a list of numbers, (3, 4), and one can picture it as a point in the coordinate system or as an arrow from the origin to that point. Two vectors are added number by number, (3, 4) + (1, −1) = (4, 3); multiplying by a number multiplies each component, 2·(3, 4) = (6, 8), the arrow becomes twice as long. Its length is, by Pythagoras, the square root of the sum of the squares, √(3² + 4²) = √25 = 5, and that holds in a thousand dimensions just as in two, even if one can no longer draw it there. The number of components is called the dimension of the vector.

The dot product. The most important calculation with two vectors is the weighted sum from section 2, read anew: the weights (2, 0.5) and the inputs (3 kilometres, 10 minutes) are two vectors, and 2·3 + 0.5·10 = 11 is their dot product: component times component, everything added up, the result a single number, a scalar. The dot product has a geometric meaning that the rest of the text needs: it is large when two arrows point in the same direction, zero when they are perpendicular to each other, and negative when they point in opposite directions. (3, 4) and (4, 3) point almost the same way: 3·4 + 4·3 = 24. (3, 4) and (−4, 3) are perpendicular: −12 + 12 = 0. If one divides the dot product by the two lengths, one gets a number between −1 and 1 that depends only on the direction, not on the length: 24 divided by 5·5 is 0.96, close to 1, almost the same direction. This number is called the cosine similarity, because it is the cosine of the angle between the arrows, and it is the measure with which language models decide whether two words, two sentences or a word and its surroundings fit together, sections 13 and 14.

The matrix. A table of numbers, with rows and columns, was named matrix by James Joseph Sylvester in 1850, after the Latin word for womb, because from it emerge the determinants with which systems of equations had been solved since Leibniz. His friend Arthur Cayley made the table into an object of calculation in 1858 in “A Memoir on the Theory of Matrices”: he explained how two matrices are added and multiplied and when one can divide by one, and remarked that the product depends on the order, A times B is not B times A.⁴⁰ The rule that counts is matrix times vector, and it is nothing but several dot products at once. The matrix

has two rows and three columns; multiplied by the vector (2, 1, 1), each row yields a dot product: 1·2 + 2·1 + 0·1 = 4 and 0·2 − 1·1 + 3·1 = 2, the result is the vector (4, 2). Each row is a set of weights, each row is a neuron, and the matrix times the input vector computes the weighted sums of all the neurons in one go. Such a row of neurons that all see the same input is called a layer, and a layer is a matrix. The product of two matrices, Cayley’s rule, is then the chaining of two layers: first the one matrix on the vector, then the other on the result, and that can be combined into a single matrix, which is the reason why something non-linear has to go between the layers, section 9. If one swaps the rows and columns of a matrix, tips it over at the diagonal, one is said to transpose it, written with a superscript T; the notation turns up in the formula of the transformer, and there it means only that dot products are formed between all the rows of two tables.

Counting. A layer that makes 12,288 outputs from 12,288 inputs is a matrix with 12,288 rows and 12,288 columns, that is 151 million weights, and for a single input vector it needs 151 million multiplications and as many additions. GPT-3 of 2020 has 96 layers with a dozen matrices of this order of magnitude each, together 175 billion weights, and for every word generated all of them are multiplied once: around 350 billion arithmetic operations per word.⁴¹ That this can be done in fractions of a second is owed to a property of the matrix product: all the dot products are independent of one another and can be computed simultaneously. The numerical mathematics of the 20th century, from John von Neumann’s study on inverting large matrices in 1947 to the standard routines BLAS of 1979 and LAPACK of 1992, had already made the matrix product the most optimised calculation in the world when graphics cards came along.⁴² A graphics card computes the same small calculation for every pixel, millions of times simultaneously; Nvidia called the GeForce 256 of 1999 the first GPU, opened the cards to arbitrary calculations in 2007 with CUDA, and in 2009 a group around Andrew Ng showed that neural networks train seventy times faster on them than on the main processor. Three years later AlexNet, trained on two graphics cards for gamers, won the image competition with which the deep-learning wave began, section 5 of the history of AI.⁴³ Since then, computing power for matrix products has been the currency in which models and companies are valued. A matrix with more than two directions, a stack of matrices, is called a tensor, and the word is in TensorFlow, the tool with which Google released computing with them in 2015.

What the matrix cannot do. A matrix turns vectors into vectors, but only through weighted sums, and the result of several matrices one after another is again only a matrix. A model made of nothing but matrices remains a straight line, however many layers it has; it can do nothing that the multiple regression of 1900 could not. The distortion between the layers that bends the line comes from another corner of mathematics, from logic, and from a model of the nerve cell from 1943.

Try it yourself. The library NumPy, which is supplied with Python or can be installed afterwards, computes with vectors and matrices as with numbers: import numpy as np, then W = np.array([[1,2,0],[0,-1,3]]) and x = np.array([2,1,1]), and W @ x yields [4 2]. np.dot([3,4],[4,3]) gives 24, np.linalg.norm([3,4]) the length 5, and W.T the transposed matrix with three rows and two columns. A matrix of a thousand by a thousand random numbers, A = np.random.rand(1000,1000), is multiplied by A @ A in a few hundredths of a second, a billion arithmetic operations.⁴⁴

9. Logic becomes arithmetic: Boole, Shannon and the neuron (1847–1949)

The problem. Everything said so far computes with quantities: prices, lengths, probabilities. Thinking, however, seems to work with yes and no, with “if this and that, then the other”, and Leibniz had dreamt in 1685 of settling disputes by calculation: “Let us calculate.” George Boole, son of a shoemaker from Lincoln, who had taught himself mathematics, redeemed the dream in 1847 in a treatise of 82 pages and in 1854 in the “Laws of Thought”: write 1 for true and 0 for false, and logic becomes algebra.⁴⁵ “A and B” is the product A·B, which is 1 only if both are 1. “Not A” is 1 − A. “A or B” is A + B − A·B, for 1 + 1 − 1 is 1, not 2. With these three calculations every statement composed of yes and no can be written as a formula and its truth value computed, and Boolean algebra remained a hobby-horse of logicians until a student translated it into copper wire in 1937.

Switches and machines. Claude Shannon, 21 years old, showed in his master’s thesis at MIT that a relay, an electrical switch that is closed by another current, carries out Boole’s calculation: two switches one after the other are “and”, two side by side are “or”, a switch that opens when current flows is “not”. Every Boolean formula becomes a circuit and every circuit a formula, and with that the design of calculating machines could be reduced to algebra.⁴⁶ A year earlier Alan Turing in Cambridge had described what a machine that only reads symbols, writes them and changes its state can compute, namely everything that can be computed with an algorithm at all, and that a single such machine can imitate every other if one gives it the other’s description.⁴⁷ The history of AI tells what became of both. For this text what counts is that yes and no have been numbers ever since, and that a calculation of zeros and ones can do everything that a machine can do.

The neuron as a threshold. In 1943 the neurophysiologist Warren McCulloch and the 20-year-old Walter Pitts in Chicago joined Boole’s logic to what was known about nerve cells: a cell receives signals from others, some excitatory, some inhibitory, and fires when the excitation exceeds a threshold. Their paper “A Logical Calculus of the Ideas Immanent in Nervous Activity” made a computing element of it.⁴⁸ The threshold neuron forms the weighted sum of its inputs, section 2, and outputs 1 if the sum reaches a threshold, otherwise 0. With the inputs A and B, both weights 1 and the threshold 2 that is “and”: only 1 + 1 reaches 2. With the threshold 1 it is “or”: 1 + 0 suffices. With one input, the weight −1 and the threshold 0 it is “not”: −0 reaches 0, −1 does not. Boole’s three calculations are three settings of the same component, and McCulloch and Pitts proved that networks of such neurons can compute every Boolean formula and, with feedback, can do what Turing’s machine does. John von Neumann read the paper and adopted its notation in 1945 in the design of the first stored-program computer.

The bend. The jump from 0 to 1 at the threshold is the point at which this text solves the problem from sections 2 and 8. The calculation “sum, then jump” is called the activation function: a function that is applied to the result of the weighted sum and is not itself a straight line. The step of McCulloch and Pitts was the first. A layer of neurons with an activation function, followed by a second, can no longer be combined into a single matrix, because the step stands in between, and that is why the network, unlike the regression of 1900, can describe relationships that bend, have corners, go up and down. The threshold can be written as a bias: “sum reaches 2” is the same as “sum minus 2 reaches 0”, the bias is −2, and the neuron is then exactly the linear function from section 2, followed by the decision whether it lies above zero. In the plane with the inputs A and B, “sum equals threshold” is a straight line, and the neuron says on which side a point lies. That remains the core of every artificial neuron to this day: weights, sum, bias, activation. Only the step was replaced, in 1986 by the sigmoid curve from section 3, which does the same service but has a slope, and since 2011 by an even simpler curve, section 11.

Learning through use. What McCulloch and Pitts did not say was where the weights come from. They set them by hand, as Shannon wired his switches. The Canadian psychologist Donald Hebb proposed in 1949 in “The Organization of Behavior” that connections between nerve cells grow stronger when both cells are active at the same time, a sentence that became famous in the short form “cells that fire together wire together” and that was actually phrased that way by Carla Shatz in 1992.⁴⁹ The Hebbian rule was the first learning rule: a prescription by which a network changes its weights itself from experience. It could imprint patterns, but not learn from errors. The neuron that adjusts its weights from its own mistakes was built eight years later by a psychologist in Buffalo, and with it the history of AI in the narrower sense began.

Try it yourself. The and-neuron fits into one cell: in A1 and B1 the inputs 0 or 1, in C1 the formula =IF(1*A1+1*B1>=2,1,0). With >=1 it becomes “or”. Whoever adds a third input and a negative weight, =IF(1*A1+1*B1-2*C1>=1,1,0), has “A or B, but not if C”, and can read off the truth table with eight rows for all the input combinations. There is one combination of two inputs that no choice of the two weights and the threshold can manage; which one, the next section reveals.⁵⁰

10. The machine learns: The perceptron and its limits (1957–1969)

The problem. Frank Rosenblatt, a psychologist at the Cornell Aeronautical Laboratory in Buffalo, wanted a machine that recognises patterns without anyone setting its weights for it. He described it in 1957 in a report and in 1958 in a paper and built it in 1960: the Mark I Perceptron had a camera of 400 photocells, weights in the form of potentiometers that small motors adjusted, and learnt in hours to distinguish letters.⁵¹ The Navy, which paid for the project, presented it to the press, and the New York Times wrote on 8 July 1958 of “the embryo of an electronic computer that [the Navy] expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence”, the first sentence of a genre that has not broken off since.⁵² The machine itself was McCulloch and Pitts’ neuron with a learning rule, and the rule fits into a single sentence.

The learning rule. The perceptron computes the weighted sum of its inputs plus bias and outputs 1 if it lies above zero, otherwise 0, as in section 9. Then it compares its output with the correct answer. If the two agree, everything stays as it is. If it said 0 where 1 was correct, it adds to each weight the corresponding input and to the bias one, so that the sum comes out larger the next time. If it said 1 where 0 was correct, it subtracts the input. In a formula: new weight = old weight + (target − output) · input, where target − output is either 0, 1 or −1. Let us work through the “or”, with the four examples (0,0) → 0, (0,1) → 1, (1,0) → 1, (1,1) → 1 and all the weights, bias included, zero at the start. At the example (0,1) the sum is 0, the output 0, correct would be 1: the weight of the second input becomes 1, the bias becomes 1. In the second pass the perceptron answers (0,0) with 1, because the bias is 1, correct would be 0: the bias falls to 0. To (1,0) it answers 0, correct would be 1: the first weight becomes 1, the bias is 1 again. In the third pass (0,0) is wrong again, the bias falls to 0, and after that everything is correct: bias 0, both weights 1, the sum is above zero exactly when at least one input is 1. Four corrections, and the machine has learnt the “or” that McCulloch and Pitts had set by hand. Bernard Widrow and Ted Hoff at Stanford gave the same idea in 1960 a form that computes with the slope instead of the jump: their Adaline adjusted the weights by the learning rate times the error of the sum itself, times the input, and that is, number for number, Cauchy’s descent on the squared error from sections 6 and 7, the first neuron that learnt with gradient descent.⁵³ Albert Novikoff proved in 1962 that Rosenblatt’s rule always finishes after finitely many corrections if there are any weights at all that separate all the examples correctly.⁵⁴

The line in the plane. The proof has a condition, and it is the end of the first enthusiasm. By section 9 the weights of a neuron are a straight line in the plane of the inputs, and the neuron says on which side a point lies. With the “or” the point (0,0) lies on one side and the three others on the other, and a line between them is easy to find; examples that can be separated in this way are called linearly separable. Now the “either or”, the exclusive one, in logic XOR: (0,0) → 0, (0,1) → 1, (1,0) → 1, (1,1) → 0. The two ones lie on the corners (0,1) and (1,0), the two zeros on (0,0) and (1,1), crosswise, and no line in the world puts the ones on one side and the others on the other. A neuron cannot learn XOR, not in a thousand passes, because the weights it is looking for do not exist. Marvin Minsky and Seymour Papert made this and harder examples the subject of a book in 1969, “Perceptrons”, which showed with mathematical care what a single neuron cannot do, and the funders withdrew from research on networks, section 2 of the history of AI.⁵⁵ Rosenblatt drowned in a sailing accident in 1971, on his 43rd birthday.

The second layer. And yet the solution was known, to Minsky and Papert as well. Two neurons, one for “A or B” and one for “A and B”, lay two lines in the plane, and a third neuron, which receives the outputs of the two as inputs, with the weights 1 and −1 and the threshold 1, says “or, but not and”: exactly XOR. The first two neurons form a hidden layer, hidden because neither the input nor the output of the network shows them directly, and with enough hidden neurons arbitrarily many lines can be laid and regions of any shape fenced in. A network of an input layer, one or more hidden layers and an output layer is called a multilayer perceptron, and it is to this day a component of every large model; in the transformer there is one in every layer. The problem was not building it but training it. Rosenblatt’s rule adjusts the weights of a neuron according to its own error, but what error does a hidden neuron have? Nobody tells it what it should have output. That is the question of credit assignment: which of the many weights inside bears how much of the blame for the error at the end? The answer is the partial derivative of the error with respect to each weight, section 4, computed with the chain rule through all the layers. Only, the step of McCulloch and Pitts has derivative zero everywhere except at the jump, where it is infinite, and by the chain rule nothing gets through a derivative of zero; the error does not find its way to the hidden weights. The solution is to replace the step by the sigmoid curve, which does the same service and has a slope at every point, σ·(1 − σ), section 4. Then the whole network is a chain of functions that can be differentiated, from the first weight to the error, and the chain rule delivers for each weight its share of the blame. That is backpropagation, and it was invented several times before anyone noticed what they had got.

Try it yourself. The perceptron is ten lines long in Python:

data = [((0,0),0), ((0,1),1), ((1,0),1), ((1,1),1)]   # or
w1 = w2 = b = 0
for epoch in range(10):
    for (x1, x2), target in data:
        out = 1 if w1*x1 + w2*x2 + b > 0 else 0
        w1 += (target-out)*x1; w2 += (target-out)*x2; b += (target-out)
    print(epoch, w1, w2, b)

After the third pass nothing changes any more: 1, 1, 0. If one replaces the data by XOR, ((0,0),0), ((0,1),1), ((1,0),1), ((1,1),0), the weights run in circles, even with range(10000).⁵⁶

11. Backwards through the chain: Backpropagation (1960–1989)

The problem. A network with hidden layers needs, for every weight, the partial derivative of the loss, and by the chain rule each of these is a product of as many factors as there are layers between the weight and the output. Computing the factors separately for every weight would mean going through billions of chains for billions of weights. The insight that avoids this came from rocket guidance: Henry Kelley in 1960 and Arthur Bryson in 1961 computed how the final error of a trajectory depends on every control decision along the way by calculating backwards from the end, and Stuart Dreyfus wrote that down as the chain rule in 1962.⁵⁷ Seppo Linnainmaa showed the general procedure in 1970 in his master’s thesis in Helsinki: for any computation, however nested, all partial derivatives with respect to all inputs can be determined in a single pass backwards, and that pass costs hardly more than the computation itself.⁵⁸ Paul Werbos applied this to neural networks in 1974 in his doctoral thesis at Harvard, and nobody took notice.⁵⁹ Only when David Rumelhart, Geoffrey Hinton and Ronald Williams showed in Nature on 9 October 1986, on four pages, that networks trained this way develop usable representations of the task in their hidden layers, XOR included, did the procedure get its name and its effect: backpropagation, the passing back of the error.⁶⁰ Yann LeCun had found it independently in 1985, and so had David Parker; the history of AI tells what it set off.

Forwards. The smallest network on which everything can be shown has one input x, one hidden neuron and one output neuron, without bias, to keep the calculation short. The hidden neuron forms z = w₁·x and outputs h = σ(z), the sigmoid curve from section 3. The output neuron forms y = w₂·h, and the loss is the squared error L = (y − t)², where t is the right answer. With x = 1, the target t = 0.25 and the starting weights w₁ = 0.5 and w₂ = 1, the forward pass gives: z = 0.5; h = σ(0.5) = 0.6225; y = 1 · 0.6225 = 0.6225; L = (0.6225 − 0.25)² = 0.3725² = 0.1387. The network says 0.62 and should say 0.25. Which weight is to blame, and by how much?

Backwards. One begins at the end and goes back along the chain, and at every step one multiplies by the local slope, section 4. The derivative of the loss with respect to y is 2·(y − t) = 2 · 0.3725 = 0.745: one thousandth more output, 0.745 thousandths more loss. One step back to w₂: y is w₂·h, the derivative of y with respect to w₂ is h = 0.6225, so the derivative of the loss with respect to w₂ is the product 0.745 · 0.6225 = 0.464. Now to the hidden neuron: the derivative of y with respect to h is w₂ = 1, so the loss depends on h with 0.745 · 1 = 0.745. The derivative of h with respect to z is σ·(1 − σ) = 0.6225 · 0.3775 = 0.235, so the loss depends on z with 0.745 · 0.235 = 0.175. And the derivative of z with respect to w₁ is x = 1, so the derivative of the loss with respect to w₁ is 0.175 · 1 = 0.175. That is the backward pass, and it has the form Linnainmaa meant: the number 0.745 was computed once and used for both weights, and in a network with a thousand hidden neurons the number that arises at the output would likewise be computed once and passed on to all of them. Every neuron receives from behind a single number, how strongly the loss depends on its output, multiplies it by its own slope and passes the result, weighted by the weights, on to the neurons before it. The backward pass costs about as much as the forward pass, for all weights together, and that is the whole reason why networks with billions of weights can be trained.

The step. With the two derivatives, Cauchy’s method from section 7 takes its step, learning rate 0.5: w₂ = 1 − 0.5 · 0.464 = 0.768; w₁ = 0.5 − 0.5 · 0.175 = 0.412. Forwards once more: z = 0.412; h = 0.602; y = 0.768 · 0.602 = 0.462; L = (0.462 − 0.25)² = 0.045. The loss has fallen from 0.139 to 0.045, to a third, with one step. The second step brings it to 0.017, and after a few dozen steps the output is at 0.25. A large model does nothing else, only with matrices instead of single numbers: the forward pass is a sequence of matrix products and activations, section 8, the backward pass a sequence of products with the transposed matrices and the slopes of the activations, and at the end each of the billions of weights has its number. That nobody derives these derivatives by hand any more has been taken care of by software since the 2010s: tools such as Theano, TensorFlow and PyTorch record during the forward pass which computation followed from which, and run the recording backwards. This is called automatic differentiation, and it is Linnainmaa’s procedure of 1970 made convenient. The training of a model is thus three lines that run trillions of times: compute the loss, differentiate backwards, step.

What the hidden layers learn. Rumelhart, Hinton and Williams showed in 1986 not only that the procedure works, but what comes of it. A network that learned family relationships developed in its hidden layer neurons that stood for generation and branch of the family, without anyone having told it these concepts. The hidden layer had found a representation, a depiction of the input in numbers in which the task becomes easy, and that has been the promise of the method ever since: one gives the network the raw data and the right answers, and the intermediate layers find the features themselves. Kunihiko Fukushima had built layers in 1980 with the Neocognitron that scan small patches of an image, and LeCun trained such a network in 1989 with backpropagation on handwritten postcodes; its successors read a part of America’s cheques in the 1990s.⁶¹ In 1989 George Cybenko and, independently, Kurt Hornik with colleagues proved the universal approximation theorem: a network with a single, sufficiently wide hidden layer and a sigmoid activation can reproduce any continuous function as exactly as one wishes.⁶² The theorem does not say how wide the layer has to be, and not whether gradient descent finds the weights; it says that it does not fail on account of the design.

The vanishing slope. What it failed on instead was found by Sepp Hochreiter in 1991 in his diploma thesis in Munich: the slope of the sigmoid curve is at most 0.25, and the chain rule multiplies by such a slope at every step back. Ten layers deep, the product is at most 0.25¹⁰, one millionth; the weights of the front layers receive almost nothing of the error any more and do not learn. This problem is called the vanishing gradient, and it kept the networks flat for twenty years.⁶³ Three changes lifted it. The activation function ReLU, which instead of the S-curve simply computes “everything below zero becomes zero, everything else stays”, has the slope 1 or 0 and nothing in between, so that the chain does not shrink; it took hold from 2011 and is in AlexNet of 2012.⁶⁴ The residual connection, which Kaiming He and colleagues introduced in 2015, lets each layer compute only the change and adds it to the unchanged input, x + f(x), so that the error comes forward on the straight path past every layer; with that, 152 layers became trainable, and every transformer is built this way.⁶⁵ And normalisation, which brings the numbers in every layer back to a fixed scale, keeps the slopes in a range in which the procedure works. Geoffrey Hinton received the Nobel Prize in Physics in 2024, together with John Hopfield, and the citation named his work of the 1980s.⁶⁶

Try it yourself. The network from this section in Python, with all intermediate values:

import math
sig = lambda z: 1/(1+math.exp(-z))
x, t, w1, w2, rate = 1.0, 0.25, 0.5, 1.0, 0.5
for step in range(30):
    z = w1*x; h = sig(z); y = w2*h; L = (y-t)**2      # forward
    dy = 2*(y-t); dw2 = dy*h; dh = dy*w2               # backward
    dz = dh*h*(1-h); dw1 = dz*x
    w1 -= rate*dw1; w2 -= rate*dw2                     # step
    print(step, round(y,4), round(L,5))

The first line reads 0 0.6225 0.13873, the second 1 0.4622 0.04502, and after thirty steps the output stands at 0.25. Whoever wants to see the procedure with hundreds of weights, without a library, will find it in Andrej Karpathy’s micrograd, a hundred lines of Python with automatic differentiation, and in his video on it, which carries out the calculation of this section in two and a half hours.⁶⁷

12. The measure of surprise: Information and entropy (1928–1951)

The problem. The squared error from section 6 suits numbers that one measures, not words that one predicts. A language model outputs a distribution over a hundred thousand words, section 5, and then one of them comes. How good was the prediction? The answer comes from communications engineering, where people wanted to know how much a message actually contains, in order to load telegraph lines to capacity. Ralph Hartley of the Bell Laboratories proposed in 1928 measuring the content of a message by the logarithm of the number of possible messages, and his colleague Claude Shannon, the student of relays from section 9, published in 1948 in the Bell System Technical Journal the “Mathematical Theory of Communication”, two parts on eighty pages, which introduced the word bit and founded a science.⁶⁸

Surprise. Shannon’s measure begins with a single event. The less probable something is, the more one learns when it occurs, and the measure of that is the logarithm to base 2 of 1 divided by the probability, section 3. A coin toss, probability ½, has the content log₂ 2 = 1 bit; a face of a die, 1/6, has log₂ 6 = 2.58 bits; an event with probability 1/1000 almost 10 bits; a certain event, probability 1, has log₂ 1 = 0 bits, one learns nothing. A bit is a yes-or-no question, and the content of an event is the number of questions one needs on average to guess it, if one asks cleverly. The logarithm ensures that the content of two independent events adds up, as it must, although their probabilities multiply. For a language model this measure is the surprise: if it gives the word that then comes the probability 0.9, the surprise is log₂(1/0.9) = 0.15 bits, if it gives it 0.1, it is 3.3 bits, and if it gives it 0.001, almost 10 bits. The smaller the probability the model has given the right word, the greater the surprise, without limit, for the logarithm of small numbers falls without limit.

Entropy. For a whole distribution Shannon takes the expected value of the surprise, section 5: every surprise weighted by its probability and added up. That is the entropy, a word he took over from thermodynamics. A fair coin has the entropy ½ · 1 + ½ · 1 = 1 bit per toss. A coin that shows heads 90 per cent of the time has 0.9 · 0.15 + 0.1 · 3.3 = 0.47 bits: one is mostly hardly surprised and rarely very, on average less than with the fair coin. A coin that always shows heads has 0 bits. The entropy is greatest when all outcomes are equally probable, and it measures how unpredictable a source is. 26 letters, all equally probable, would have 4.7 bits per letter. In 1951 Shannon had test subjects guess English texts letter by letter and estimated from their hits that the entropy of English lies around 1 bit per letter: three quarters of every text are predictable from what stands before them.⁶⁹ That was the first language model, with humans as the arithmetic unit, and its result is at the same time a limit: no model can predict better than the entropy of the language, and no procedure can squeeze a text into fewer bits than its entropy amounts to. The programs that shrink files live on the fact that texts have little entropy per character, and on random numbers they fail.

Cross-entropy. Now the case that matters: the world has a distribution p, the model believes in another, q, and the outcomes come according to p, but the surprise is computed according to q, because the model knows only q. The expected value of this surprise is called cross-entropy. Three words come in truth with 0.5, 0.25 and 0.25; the model says 0.25, 0.5 and 0.25. The surprises according to the model are 2, 1 and 2 bits, weighted with the truth: 0.5 · 2 + 0.25 · 1 + 0.25 · 2 = 1.75 bits. Had the model known the truth, it would have been 0.5 · 1 + 0.25 · 2 + 0.25 · 2 = 1.5 bits, the entropy of the truth itself. The cross-entropy is never smaller than the entropy, and it is equal exactly when the model hits the truth; the excess, here 0.25 bits, measures the distance between the two distributions and is called, after Solomon Kullback and Richard Leibler, who studied it in 1951, the KL divergence.⁷⁰ From this follows the recipe: whoever reduces the cross-entropy of a model brings its distribution closer to the truth, and at the minimum the two are equal. In practice nobody knows the distribution of the world; one has only the texts, in each of which one word actually came. Then the cross-entropy for this one word is simply the surprise the model feels, minus the logarithm of the probability it had given the right word, and averaged over all words of a text this is the number that has served as the loss function of every language model since the 1990s. It is, apart from the base of the logarithm, the same number as Fisher’s minus log-likelihood from section 6: the most probable explanation of the data is the least surprised one.

Perplexity. A cross-entropy of 1.75 bits means the model is on average as surprised as if it had to guess each time among 2^1.75 = 3.4 equally probable words. This number, 2 to the power of the cross-entropy, is called perplexity, and it is the measure in which language models have been compared since the 1970s: a model with perplexity 20 guesses as well as if it chose every word among 20 equally probable ones, one with perplexity 5 among 5. The loss figures that stand in the reports on large models are cross-entropies, mostly computed to base e instead of 2, and a loss of 2.0 per word then means a perplexity of e² = 7.4. The curves that show how this loss falls with the size of the models are the subject of the last section.

Shannon’s game. In the same paper of 1948 Shannon did something that was a game in 1948 and an industry in 2022: he generated text by drawing letters according to their probabilities, first singly, then depending on the previous letter, then on the two previous ones, then words depending on the previous word, and printed the results, which looked more English with every stage: “THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARACTER OF THIS POINT IS THEREFORE ANOTHER METHOD”. That was the product rule from section 5, with counted probabilities, and the procedure with which every model answers today. Only the probabilities no longer come from counting but from backpropagation, and how words become numbers at all, with which one can calculate, is the next section.

Try it yourself. A program with zip estimates the entropy of a text: a text file of one megabyte shrinks to a third, a file of random numbers, generated in Python with open('zufall.bin','wb').write(os.urandom(1000000)) after import os, not one bit. The entropy of the letter distribution of a German text is computed with from collections import Counter, import math, n = Counter(text), s = sum(n.values()) and -sum(c/s*math.log2(c/s) for c in n.values()): around 4.1 bits per letter, against 4.7 with a uniform distribution, and with pairs of letters instead of single letters the number per letter falls further.⁷¹

13. Words become numbers: Markov, n-grams and embeddings (1913–2013)

The problem. Everything described so far calculates with numbers, and words are not numbers. That language nevertheless has statistics was shown by the Russian mathematician Andrei Markov on 23 January 1913 before the Academy in Saint Petersburg: he had divided the first 20,000 letters of Pushkin’s “Eugene Onegin” by hand into vowels and consonants and counted what follows what. 43 per cent of the letters were vowels, but after a vowel another vowel came in only 13 per cent of cases, after a consonant in 66 per cent. The letters of a text are not independent, and Markov proved that Bernoulli’s law of large numbers from section 5 holds nonetheless if each letter depends only on the previous one. Such a sequence has been called a Markov chain ever since, and Markov had developed it against a colleague who had claimed that the law presupposed independence and therefore proved free will.⁷² Shannon made his text machine from section 12 out of it in 1948, and in the 1970s counting became an industry.

Counting. An n-gram is a sequence of n consecutive words, and an n-gram model estimates the probability of the next word by counting how often it has followed the n − 1 previous ones in large quantities of text. In the small body of text “the dog barks. the dog sleeps. a cat sleeps.”, “dog” is followed once by “barks” and once by “sleeps”: P(sleeps | dog) = ½. “the” is followed twice by “dog”: P(dog | the) = 1. “cat” is never followed by “barks”: P(barks | cat) = 0, although cats could bark in their own way and the sentence is possible. That is the weakness of counting: with a hundred thousand words there are a quadrillion possible triples, and almost all of them nobody has ever written, so that the model gives them the probability zero and suffers an infinite surprise when they do occur. Speech recognition at IBM around Frederick Jelinek built models from triples from the 1970s on, with tricks that cede a little probability to the zero, and Jelinek’s saying, “Every time I fire a linguist, the performance of the speech recognizer goes up”, describes the attitude with which statistics took over language.⁷³ Richard Bellman had called the underlying problem the curse of dimensionality in 1957: with every further input the number of possible cases multiplies, and no quantity of data fills them.⁷⁴

The neighbourhood. The way out was a sentence of the English linguist John Rupert Firth from 1957: “You shall know a word by the company it keeps.” Zellig Harris had formulated the same as a programme in 1954: words that occur in similar environments have similar meaning, and the environments can be counted.⁷⁵ Instead of noting the number zero for “cat barks”, the model should know that “cat” and “dog” stand in similar sentences, and transfer the knowledge about the dog to the cat. For that a word has to become a vector. The naive vector is a list of a hundred thousand zeros with a single one at the position of the word, called one-hot; it carries no similarity, the dot product of two different words is always zero. Gerard Salton described documents in 1975 as vectors of word counts and compared them with the cosine from section 8, the basis of every search engine. And Yoshua Bengio and his co-workers in Montréal built the first neural language model in 2003: every word gets a vector of a few dozen numbers, which are random at the start; the vectors of the last words go into a multilayer perceptron that outputs the probability of the next word; and backpropagation adjusts not only the weights of the network but also the numbers in the word vectors, so that words standing in the same environments acquire similar vectors over time.⁷⁶ Such a learned vector is called an embedding, because it embeds the word in a space in which nearness means similarity.

word2vec. That these spaces contain more than nearness was shown in 2013 by a team around Tomáš Mikolov at Google. Their procedure word2vec learned from one and a half billion words in a day vectors with 300 numbers, and in these vectors one could calculate: the vector of “king” minus that of “man” plus that of “woman” lay near that of “queen”, Paris minus France plus Italy near Rome. The direction from “man” to “woman” was the same in the space as from “king” to “queen”, the direction from country to capital the same everywhere: relationships between words had become arrows that one can add, although nobody had taught them to the model.⁷⁷ That is the representation from section 11, for words, and its numbers mean nothing individually; what counts are distances and directions.

Tokens. The unit in which today’s models calculate is not the word. A procedure from data compression, which Philip Gage described in 1994 and which Rico Sennrich adopted for translation in 2016, breaks the text into pieces that occur often enough: it begins with single characters and step by step merges the most frequent adjacent pair into a new character, some tens of thousands of times. Frequent words become one piece, rare ones several, “Unwahrscheinlichkeit” for instance “Un”, “wahrscheinlich”, “keit”, and every string of characters in every language can be written this way, without any word being able to remain unknown.⁷⁸ Such a piece is called a token, and GPT-3 knows 50,257 of them. The first layer of a language model is the embedding matrix: a table with one row per token and, in GPT-3, 12,288 columns, whose rows are the learned vectors; looking up a token is the same as multiplying the one-hot vector by the matrix, section 8. This table alone has 617 million numbers, and it is trained like everything else, by backpropagation from the error of the prediction back into the rows. What goes into the model is thus a sequence of vectors, one per token; what comes out is a distribution over the 50,257 tokens, from which the next one is drawn; and the computation in between has to achieve one thing that n-grams and Bengio’s network could not: relate all tokens of a long text to one another.

Try it yourself. An n-gram model in Python is a counter. With a longer text in the variable text, say a book from Project Gutenberg, w = text.split(), from collections import Counter, defaultdict, n = defaultdict(Counter) and for a, b in zip(w, w[1:]): n[a][b] += 1 deliver for every word the counts of its successors; n['the'].most_common(5) shows the five most frequent. With import random and a loop that each time appends random.choices(list(n[w].keys()), n[w].values())[0] to the sentence, Shannon’s text of 1948 arises, diced word by word. How a current model breaks text into tokens is shown by OpenAI’s tokenizer in the browser: “Unwahrscheinlichkeit” is split into several pieces, “the” is one.⁷⁹

14. Attention: The transformer and the language model (1997–2017)

The problem. An n-gram looks two or three words back, Bengio’s network a dozen. A sentence like “The dog, which yesterday buried the bone in the garden, barked” requires that “barked” belong to “dog”, across eleven words. The networks of the 1990s read text word by word and carried along a state vector that changed with every word; Hochreiter and Jürgen Schmidhuber gave it, in 1997 with the LSTM, a memory with gates that survived the vanishing gradient over hundreds of steps.⁸⁰ But a vector of fixed size into which a whole sentence is pressed forgets, and the computation could not be parallelised, because every word had to wait for the previous one, which left the graphics cards from section 8 idle. Dzmitry Bahdanau, Kyunghyun Cho and Bengio introduced an aid for translation in 2014: when generating each word, the network may look back at all words of the source sentence and give each of them a weight that it computes itself.⁸¹ They called it attention, and three years later eight researchers at Google showed that one can leave out everything else. Their paper of June 2017 is called “Attention Is All You Need”, and the model in it is called the transformer.⁸² What it computes consists of nothing that this text has not already introduced.

The weighted mean. Every token arrives as a vector, section 13. Attention gives every token a new vector that is a weighted sum of the vectors of all tokens, with weights that add up to 1, an expected value as in section 5, and the token computes the weights itself, from similarity. For this every token vector is multiplied by three matrices, section 8, and yields three vectors: a query, with which the token asks, a key, by which it can be found, and a value, which it contributes when it is found. The token forms the dot product of its query with the keys of all tokens, section 8, gets a score for each, sends the scores through the softmax from section 3 and obtains weights; then it adds up the values of all tokens, each with its weight. A numerical example with three tokens and two dimensions: let the query of the first token be (2, 0), the keys of the three tokens (1, 0), (0, 1) and (1, 1). The dot products are 2, 0 and 2. Softmax turns them into e², e⁰ and e², that is 7.39, 1 and 7.39, divided by the sum 15.78: the weights 0.468, 0.063 and 0.468. The first token listens 47 per cent to itself, 47 per cent to the third and hardly at all to the second, whose key stands at right angles to its query. If the values are (1, 2), (3, 0) and (0, 1), the new vector is 0.468 · (1, 2) + 0.063 · (3, 0) + 0.468 · (0, 1) = (0.66, 1.40). In the model, query, key and value stand in matrices Q, K and V with one row per token, the dot products of all queries with all keys are the matrix product Q·Kᵀ, the transposing from section 8, and the whole computation reads, in the notation of the paper of 2017: Attention(Q, K, V) = softmax(Q·Kᵀ / √d) · V. The √d, the square root of the number of dimensions, divides the scores down so that with long vectors the softmax does not throw all the weight onto one token; it is the temperature from section 3. Whoever has read this far can read the formula: it is a weighted sum whose weights come from dot products and a softmax, and all the matrices in it are learned.

The layer. The transformer repeats this computation several times side by side, with its own matrices Q, K and V per repetition, which are called heads, because each can attend to something different, one to the subject, one to the previous word, one to the opening bracket, and joins the results into one vector. Then follows, for each token separately, a multilayer perceptron from section 10, two matrices with a ReLU between them, in which most of the model’s weights sit. Around both lie the residual connections from section 11, the vector of a token is not replaced but supplemented by the result, and the normalisation that brings it back to a fixed scale. Attention plus perceptron plus both of these is a transformer layer, and the model stacks them: 12 in GPT-1 of 2018, 96 in GPT-3, in the models of 2026 more than the makers say. Because attention does not know the order of the tokens, each token sees all the others as a set, a second vector is added to each vector at the start that encodes its position, in the paper of 2017 from sine and cosine curves, later learned or as a rotation. All tokens of a text pass through all layers at the same time, as matrices, and that was precisely the purpose: the computation is a stack of matrix products, and graphics cards like nothing better. For generating text only one more rule is needed: each token may attend only to the tokens before it, so that the model does not see into the future during training.

The count. The size of a model can now be worked out. GPT-3 has vectors with d = 12,288 numbers and 96 layers. Per layer, the four matrices of the attention, for Q, K, V and the joining together, are each 12,288 by 12,288, together 4·d² weights, and the perceptron, which widens to four times and back again, has 8·d², together 12·d² = 1.8 billion per layer, times 96 is 174 billion, plus the embedding matrix from section 13 with 0.6 billion: 175 billion parameters, the number from the announcement of 2020.⁴¹ Each of these weights is an unknown in the sense of section 1, and the training has found them all.

The whole training. With this the sentence from the beginning of this text can be resolved. A text from the Internet is broken into tokens, section 13. The tokens become vectors, embedding matrix. The vectors pass through 96 layers of matrix products, softmax, ReLU and additions, sections 3, 8, 11 and 14. At the end the vector of each token is multiplied by a last matrix and yields 50,257 logits, one score per possible next token, and softmax turns them into a distribution, sections 3 and 5. The loss is the cross-entropy, the model’s surprise at the token that actually follows in the text, averaged over all tokens, section 12, at the same time Fisher’s minus log-likelihood, section 6. Backpropagation passes the loss backwards through all layers into each of the 175 billion weights and into the embeddings, sections 4 and 11. Adam takes a step downhill with all the weights, section 7. Then the next mini-batch, and so on over 300 billion tokens, a few months on a few thousand graphics cards. Whoever puts a question to the finished model lets it compute forwards once, draws a token from the distribution, appends it to the question and computes again, sections 5 and 12: Shannon’s game, with 350 billion multiplications per word. That this computation turns into answers that look like understanding, nobody predicted from the mathematics; that the computation is as described here is in the papers, and it has been unchanged at its core since 2017.

After pretraining. A model trained this way continues texts; that it answers questions and follows instructions comes from two further steps that use the same mathematics. First it is trained further on some tens of thousands of examples of question and good answer, the same cross-entropy on other data. Then humans rate pairs of answers, a second model learns from the ratings to give answers a number, and the language model is adjusted with gradient descent so that this number rises; the procedure comes from Paul Christiano and colleagues in 2017 and in 2022 turned GPT-3 into the model behind ChatGPT.⁸³ Human preferences become a number, and the number becomes the loss that Cauchy’s descent reduces, section 7 of the history of AI.

Try it yourself. Georgia Tech’s Transformer Explainer runs GPT-2 in the browser and shows for every sentence entered the attention weights of every head, the logits and the softmax, with a slider for the temperature. The count above can be repeated for GPT-2: d = 768, 12 layers, 12·768²·12 = 85 million, plus embeddings 50,257·768 = 39 million, together 124 million, the number in the model’s name. Grant Sanderson’s video series on 3Blue1Brown demonstrates the computation of this section in pictures, three episodes for attention and the perceptron.⁸⁴

15. What the mathematics does not explain (since 2017)

The problem. Everything a language model computes is described in the previous sections, and none of it is younger than 2017. What the mathematics does not supply is the explanation of why the computation works so well, and the gap has, if anything, grown since 2019. Three findings contradict what statistics had taught since Galton. The first is the overfitting from section 6: a model with more parameters than data points should learn the data by heart and fail on the unknown. Mikhail Belkin and colleagues showed in 2019 that the error on test data at first rises with growing model size as expected, is worst at the boundary at which the model can just about memorise the training data, and then falls again the larger the model becomes, a second descent for which there are attempted explanations but no theory.⁸⁵ The second finding is the mountain range from section 7: that gradient descent in a loss landscape with a hundred billion directions does not get stuck in a bad valley can be proved for simple networks and only observed for the large ones. The third is that the models learn things they were not taught, long after the point at which they knew the training data by heart: a network that Alethea Power and colleagues trained in 2022 on an arithmetic task had stored the examples after a few thousand steps and grasped the rule only after a million steps, with a jump that nobody had predicted and that they called grokking.⁸⁶

The laws of size. What the mathematics supplies instead are curves. Jared Kaplan and colleagues at OpenAI showed in January 2020 that the loss from section 12, the cross-entropy on unknown text, falls with the number of parameters, the quantity of training data and the computing power, each according to a power law, smoothly over seven orders of magnitude, and that this makes it possible to predict how good a model will be that has not yet been built.⁸⁷ GPT-3 was the bet on this curve, and it paid off. Jordan Hoffmann and colleagues at DeepMind corrected the ratio in 2022: for a given computing power the best model is smaller and trained on more data than previously assumed, about twenty tokens per parameter. The scaling laws are the nearest thing to a theory there is, and they are empirical, like Kepler’s laws before Newton: they describe that it is so, not why. Richard Sutton had drawn the balance in 2019 in a short text, “The Bitter Lesson”: in seventy years of AI research the general method with more computing power had won every time over built-in knowledge, and the curves are the numbers to go with it.⁸⁸

Looking inside. The third direction tries to read the computation instead of explaining it. The weights of a model are numbers that mean nothing individually, section 13, but the vectors that flow between the layers can be examined. A team at Anthropic decomposed in 2024 the activations of a medium-sized production model into millions of directions, each standing for a concept, from the Golden Gate Bridge to programming errors, and could change the model’s behaviour by amplifying a direction; the mathematics behind it is again the regression from section 6, this time with the side condition that only a few directions are active at once.⁸⁹ This work, which is called interpretability, reads the model as one reads a brain with electrodes, and it is far from reading a whole one. The Nobel Prizes of 2024, in physics for Hopfield and Hinton, in chemistry for the prediction of protein structures with the same methods, acknowledged that the procedure works, and the citations left open why.⁶⁶

The open bottleneck. The mathematics behind AI is that of the 17th to 20th centuries, coordinates and derivative, probability and least squares, vectors and matrices, information and entropy, assembled into an algorithm that was described in 1847 and completed in 1986. What is new is the size, and the size has effects that this mathematics describes but does not explain. The chain of this text therefore ends not with a solution but with the form the bottleneck has taken: there is a procedure that turns text into a computation that answers, and there is no theory that says what the computation has learned, when it errs and how much more size still helps. Those who use it nonetheless, and in 2026 that is most people, do so as the astronomers of the 17th century did with Napier’s tables: the calculation is right, and it does not replace understanding.

The chain in brief

1. The Babylonians and al-Khwarizmi calculate backwards from the result to the unknown, Viète and Descartes give it letters, Euclid the algorithm; a language model is an equation with billions of unknowns that nobody solves by rearranging.
2. Descartes turns points into pairs of numbers, Euler the function into a machine; the straight line y = a·x + b with slope and intercept becomes the weighted sum with weights and bias, the computation of every neuron, and cannot bend.
3. Napier turns multiplication into addition, Bernoulli and Euler find e; sigmoid and softmax turn arbitrary numbers into probabilities, and the temperature controls how sharply.
4. Newton and Leibniz find the slope at a point, the derivative; minima lie where it is zero, and the chain rule says that the slopes of chained functions multiply.
5. Pascal, Fermat and Bernoulli make chance calculable, Bayes turns the condition around, Kolmogorov lays down the rules; a language model is a distribution over the next word, from which one draws.
6. Legendre and Gauss find the best straight line through measured points by minimising squared errors; Galton gives the procedure the name regression, Fisher the principle of maximum likelihood; training is a regression whose equations can no longer be solved.
7. Cauchy goes downhill: compute the derivative, take a small step against it, repeat; with billions of directions the derivative is called the gradient, with random selection stochastic, with adapted steps Adam.
8. Hamilton, Grassmann and Cayley turn lists and tables of numbers into objects of calculation; a layer is a matrix, a dot product measures similarity, and graphics cards compute matrix products in bulk.
9. Boole turns logic into algebra, Shannon builds it from switches, McCulloch and Pitts from neurons with a threshold; the activation function bends the straight line, Hebb lets weights grow through use.
10. Rosenblatt’s perceptron learns from errors but can only separate what a straight line separates; XOR demands a hidden layer, and nobody knows how to train it.
11. Linnainmaa, Werbos and Rumelhart, Hinton and Williams compute the chain rule backwards through all layers at once: backpropagation; ReLU and residual connections keep the gradient alive.
12. Shannon measures information in bits, surprise as a logarithm, entropy as average surprise; cross-entropy is the loss of every language model, perplexity its measure.
13. Markov, Shannon and Jelinek count what follows what; Bengio and Mikolov turn words into learned vectors in which nearness is meaning and relationships are directions.
14. Bahdanau and Vaswani let every token form a weighted sum of all the others, with weights from dot product and softmax; the transformer stacks that, and its training is cross-entropy, backpropagation and Adam over trillions of tokens.
15. Scaling laws describe how the loss falls with size, double descent and grokking contradict the old statistics, and no theory explains why the computation answers. What remains open is the why.

Timeline

Further reading

• Steven Strogatz: Infinite Powers. How Calculus Reveals the Secrets of the Universe. Houghton Mifflin Harcourt, Boston 2019. The history of the derivative from Archimedes to today, without formulas that one does not understand.
• Ian Stewart: Seventeen Equations That Changed the World. Profile Books, London 2012 (German edition: Welt-Formeln. 17 mathematische Gleichungen, die alles veränderten, Rowohlt, Reinbek 2014). One chapter each on Pythagoras, logarithm, derivative, normal distribution, information.
• James Gleick: The Information. A History, a Theory, a Flood. Pantheon, New York 2011 (German edition: Die Information. Geschichte, Theorie, Flut, Redline, Munich 2011). Shannon, bits and entropy, told as a story.
• Sharon Bertsch McGrayne: The Theory That Would Not Die. Yale University Press, New Haven 2011 (German edition: Die Theorie, die nicht sterben wollte, Springer Spektrum, Berlin 2014). Two and a half centuries of dispute over Bayes’s theorem.
• Stephen M. Stigler: The History of Statistics. The Measurement of Uncertainty before 1900. Harvard University Press, Cambridge, MA 1986. Legendre, Gauss, Galton and Pearson from the sources.
• Michael Nielsen: Neural Networks and Deep Learning. Online 2015, http://neuralnetworksanddeeplearning.com/. The perceptron, backpropagation and the universal approximation theorem, with code and pictures, free to read.
• Ian Goodfellow, Yoshua Bengio, Aaron Courville: Deep Learning. MIT Press, Cambridge, MA 2016, free at https://www.deeplearningbook.org/ (German edition: Deep Learning. Das umfassende Handbuch, mitp, Frechen 2018). The textbook, with chapters of its own on linear algebra and probability.
• Grant Sanderson: Neural networks, video series on 3Blue1Brown, 2017–2024, https://www.3blue1brown.com/topics/neural-networks. Gradient descent, backpropagation and transformers in moving pictures.
• Andrej Karpathy: Neural Networks: Zero to Hero, video series, 2022–2023. From backpropagation in a hundred lines to GPT, every line of code demonstrated.

Sources

The chronicle rests on the original works named in the footnotes and on the historical accounts under “Further reading”. All numerical examples have been recalculated; the figures for GPT-3 come from the publication of 2020, those for the models of 2026 from the makers’ statements.