Glossary Mathematics behind AI

All technical terms of the article The History of the Mathematics behind AI in alphabetical order. The number at the end names the section in which the term is introduced.

A

• activation function – function applied to the weighted sum of a neuron that is not a straight line: the step of 1943, the sigmoid curve, ReLU. It allows networks to bend. 9
• Adam – method of Kingma and Ba (2014) that adapts the step size for every weight separately from running averages of its gradients; the standard for training large models. 7
• algorithm – sequence of calculation steps that can be carried out without judgement, says unambiguously at each step what to do next and stops after finitely many steps; the word comes from al-Khwarizmi’s name, the oldest example from Euclid (c. 300 BC). 1
• attention – every token forms a weighted sum of the values of all tokens, with weights from the dot product of its query with the keys and a softmax; Bahdanau et al. (2014), Vaswani et al. (2017): softmax(Q·Kᵀ/√d)·V. 14
• automatic differentiation – software records every calculation during the forward pass and runs the record backwards to deliver all derivatives (Theano, TensorFlow, PyTorch); Linnainmaa’s method with comfort. 11

B

• backpropagation – method that computes the partial derivatives of the loss with respect to all weights in one backward pass by applying the chain rule from the end; Linnainmaa (1970), Werbos (1974), Rumelhart, Hinton and Williams (1986). 11
• backward pass – the chain rule from the loss back to the first weight, where each layer multiplies the number received from behind by its own slope and passes it on; costs about as much as the forward pass. 11
• Bayes’s theorem – turns the condition around: P(cause | observation) = P(observation | cause) · P(cause) / P(observation); Bayes (1763), Laplace (1774). Easy with counts instead of fractions, often wrong with fractions. 5
• bias – the constant in a weighted sum, the shift that remains even when all inputs are zero; in the threshold neuron the negative threshold. 2
• bit – unit of information (Shannon, 1948): one yes-or-no question, the content of a coin toss. 12
• Boolean algebra – logic as calculation with 0 and 1 (Boole, 1847/1854): “and” is A·B, “not” is 1 − A, “or” is A + B − A·B; Shannon built it from switches in 1937. 9

C

• chain rule – slopes of chained functions multiply: dy/dx = dy/du · du/dx (Leibniz). The basis of backpropagation, where the chain has hundreds of links. 4
• conditional probability – probability of an event when another is known, P(A | B): a six, given even, 1/3. 5
• coordinate system – x-axis and y-axis on which the coordinates are marked; turns geometry into calculation and calculation into pictures. 2
• coordinates – the two numbers that fix a point in the plane, distance to the right and upwards; Descartes (1637) and Fermat (1636). 2
• correlation – number between −1 and 1 for the strength of a linear relation (Pearson, 1896): 1 for a rising line, −1 for a falling one, 0 without slope. 6
• cosine similarity – dot product divided by both lengths, a number between −1 and 1 that depends only on direction; the measure with which models compare words and sentences. 8
• credit assignment – the question which weight inside a network bears how much blame for the error at the end; the answer is the partial derivative, computed by backpropagation. 10
• cross-entropy – average surprise when the outcomes follow the true distribution but are scored by the model; never smaller than the entropy, equal only for a correct model. The loss function of every language model. 12
• curse of dimensionality – Bellman’s word (1957) for the fact that with every further input the number of possible cases multiplies and no amount of data fills it. 13

D

• degree – of an equation: the highest exponent with which the unknown occurs; x² + 10x = 39 has degree two. Up to the fourth degree there are formulas for the solution, from the fifth on there are none (Abel, 1824). 1
• derivative – the slope of a curve at a point, the limit of the secant slopes; says how strongly the output reacts to a small nudge of the input. The derivative of x² is 2x, that of eˣ is eˣ. Newton (1665) and Leibniz (1675). 4
• dimension – number of components of a vector; the embeddings of GPT-3 have 12,288. 8
• distribution – the list of the probabilities of all possible outcomes, summing to 1; a language model outputs a distribution over its whole vocabulary for every next token. 5
• dot product – component times component, everything added, giving one number: (1, 2, 0)·(2, 1, 1) = 4. The weighted sum as vector arithmetic; large for the same direction, zero for perpendicular arrows. 8
• drawn – an outcome is chosen at random according to the probabilities of a distribution, like a ticket from an urn; this is how a language model picks its next token, which is why it answers the same question differently. 5

E

• embedding – learned vector for a token in which nearness means similarity and relations are directions (Bengio 2003, word2vec 2013: king − man + woman ≈ queen). 13
• embedding matrix – table with one row per token whose rows are the embeddings; the first layer of every language model, 617 million numbers in GPT-3. 13
• entropy – expected value of the surprise of a distribution, the measure of its unpredictability (Shannon, 1948): fair coin 1 bit, English about 1 bit per letter. Lower bound for prediction and compression. 12
• epoch – one pass through all training data; large language models see most of their data only once. 7
• equation – statement that two calculations give the same result; the sign = comes from Recorde (1557). Solving means running the calculation backwards until the unknown stands alone (al-Khwarizmi, c. 820). 1
• error – difference between a model’s prediction and the measured value, also residual; in training it is squared and summed. 6
• Euler’s number – e = 2.718281828…, the limit of Bernoulli’s compound interest with ever more frequent payments (1683); letter from Euler (1731). The curve eˣ has at every point the slope of its own value. 3
• expected value – every outcome weighted by its probability and added up, the average in the long run (Huygens, 1657); a weighted sum whose weights add up to 1, like attention. 5
• exponent – the superscript of a power; it may also be zero, negative or a fraction: 2⁰ = 1, 2⁻¹ = ½, 2 to the ½ = √2. 1
• exponential growth – growth in which every equal step multiplies by the same factor, the curve y = 2ˣ or y = eˣ; compound interest, bacteria, computing power. 3

F

• forward pass – computation of a network’s output and loss from the input, layer by layer forwards. 11
• function – rule that assigns exactly one output to every input, a machine; word from Leibniz (1673), notation f(x) from Euler (1734). 2

G

• gradient – the list of all partial derivatives of the loss, one per weight; points in the direction in which the loss rises most steeply. 7
• gradient descent – Cauchy’s method of 1847: compute the gradient, take a small step against it, repeat; the algorithm with which every neural network is trained. 7
• graph – the picture of a function in the coordinate system: for every input x the point with the output y; a straight line for a linear function, the parabola for x². 2

H

• heads – several attention computations side by side with their own matrices, able to attend to different things; GPT-3 has 96 per layer. 14
• Hebbian rule – first learning rule (Hebb, 1949): connections between simultaneously active cells grow stronger, “cells that fire together wire together”. 9
• hidden layer – neurons between the input and output of a network whose values are not visible from outside; with it arbitrarily shaped regions can be fenced in, such as XOR. 10

I

• intercept – the place where a line crosses the y-axis, the value at x = 0; for the taxi the base fare, in y = a·x + b the b. 2

K

• key – vector by which a token is found in attention; the dot product query·key is the score. 14
• KL divergence – cross-entropy minus entropy, the distance between two distributions (Kullback and Leibler, 1951). 12

L

• law of large numbers – with a growing number of trials the share of an outcome approaches its probability (Jacob Bernoulli, 1713); allows probabilities to be estimated by counting. 5
• layer – row of neurons that all see the same input; as a calculation a matrix times the input vector, followed by an activation function. 8
• learning rate – the step size of gradient descent by which the derivative is multiplied; too small costs time, too large jumps over the valley or blows everything up. 7
• likelihood – the probability of the observed data under a setting of the model; Fisher (1921/1922): choose the setting that makes it largest. With normally distributed errors this gives least squares, with language models the cross-entropy. 6
• limit – the number that a sequence of approximations approaches as closely as one likes without having to reach it; Cauchy (1821) founded the derivative on it. 4
• linear function – function of the form y = a·x + b, whose graph is a straight line; with several inputs a weighted sum. Cannot bend. 2
• linearly separable – examples that a straight line (or plane) divides into the right classes; only such can a single neuron learn. XOR is not. 10
• local minima – valleys of the loss landscape that are not the deepest; gradient descent can get stuck in them. That it nevertheless finds usable valleys in large models is experience, not theory. 7
• logarithm – the question for the exponent: the logarithm of 8 to base 2 is 3, because 2³ = 8. Turns products into sums (Napier, 1614) and small numbers into large negative ones. 3
• logits – the raw scores a model computes for every possible next token before the softmax turns them into probabilities. 3
• loss function – a number summarising how badly a model fits the data, such as the sum of squared errors or the cross-entropy; training makes it smaller. In AI simply the loss. 6

M

• Markov chain – sequence in which every element depends only on the previous one; Markov (1913) showed on Pushkin’s text that the law of large numbers holds even without independence. 13
• matrix – table of numbers with rows and columns, an object of calculation since Cayley (1858); matrix times vector is one dot product per row, that is a layer of neurons. 8
• mean – sum of the values divided by their number; the best line passes through the point of the two means. 6
• method of least squares – chooses the line (or other function) for which the sum of the squared errors is smallest: differentiate, set to zero, solve; Legendre (1805), Gauss (1809, by his own account since 1795). 6
• mini-batch – small random selection of training examples from which the gradient for one step is estimated. 7
• model – rule with adjustable numbers that is meant to describe data and predict the unknown, from the line through five points to the language model. 6
• multilayer perceptron – network of an input layer, one or more hidden layers and an output layer; a component of every transformer layer. 10

N

• n-gram – sequence of n consecutive words or characters; an n-gram model estimates the next word by counting how often it followed the previous ones, and fails on sequences never seen. 13
• natural logarithm – the logarithm to base e, written ln: ln e = 1, ln 1 = 0, ln 2 ≈ 0.693, ln 0.01 ≈ −4.6. 3
• Negative numbers – numbers below zero, to the left of zero on the number line; Brahmagupta gave the rules in 628 (debts and fortunes, minus times minus is plus). About half the weights of a language model are negative and mean “subtract this”. 1
• normal distribution – the bell curve that arises when many small random influences add up (de Moivre 1733, Gauss 1809, Laplace 1810); Gauss’s assumption about measurement errors, from which least squares follow. 5

O

• one-hot – representation of a word as a vector with a single one among zeros; carries no similarity, all dot products of different words are zero. 13
• overfitting – a model learns the training data by heart instead of the relation and predicts the unknown badly, like the curve through all five points. 6

P

• parameters – the adjustable numbers of a model, weights and biases; a language model of 2026 has a few hundred billion, and training is the attempt to determine them. 1
• partial derivative – derivative with respect to one of several inputs while the others are held fixed, written with ∂; in training the derivative of the loss with respect to a single weight. 4
• perceptron – Rosenblatt’s learning neuron (1957): after every error the input is added to or subtracted from the weights; finds a separating line after finitely many corrections if one exists (Novikoff, 1962). 10
• perplexity – 2 to the cross-entropy: a model with perplexity 20 guesses as well as if it chose every word among 20 equally likely ones; measure for language models since the 1970s. 12
• power – a number multiplied by itself several times, written with a superscript: x² = x·x, 2¹⁰ = 1,024; notation of Descartes (1637). 1
• probability – number between 0 and 1 for an event, with equally possible cases the favourable ones divided by all: a six 1/6; Pascal and Fermat (1654). 5
• product rule – the probability of a sequence of events is the product of the conditional probabilities, P(A)·P(B | A)·P(C | A and B); this is how a language model scores a text word by word. 5

Q

• query – vector with which a token asks in attention which other tokens matter to it; computed from the token vector with a learned matrix. 14

R

• regression – fitting a function to data by least squares; named after Galton’s “regression towards mediocrity” (1886). The multiple regression of 1900 is the artificial neuron with solved instead of groped weights. 6
• ReLU – activation function “everything below zero becomes zero, everything else stays”, slope 1 or 0; the standard since 2011 because it does not shrink the gradient. 11
• representation – the numbers in which a hidden layer represents an input; they arise by themselves in training, such as generation and family branch (1986) or word vectors. 11
• residual connection – each layer computes only a change that is added to the unchanged input, x + f(x) (He et al., 2015); lets the error pass every layer on its way forward and makes hundreds of layers trainable. 11
• root – the reverse of squaring: √9 = 3. Most roots never end (√2 = 1.414213…) and are approximated as precisely as one likes, as already on Babylonian tablets. 1

S

• scaling laws – empirical power laws (Kaplan et al., 2020; Hoffmann et al., 2022) according to which the loss of a language model falls smoothly with parameters, data and computing power; they describe, they do not explain. 15
• sigmoid function – σ(z) = 1/(1 + e⁻ᶻ), an S-shaped curve that squeezes every number between 0 and 1: σ(0) = 0.5, σ(2) ≈ 0.88; Verhulst’s logistic curve (1838), since 1986 an activation function with slope σ·(1 − σ). 3
• slope – change of the output divided by the change of the input that caused it; the same everywhere on a line, in y = a·x + b the a. 2
• softmax – turns several scores into probabilities that sum to 1: e to the score, divided by the sum; 2, 1, 0 become 0.665, 0.245, 0.090. Name from Bridle (1989), formula from physics; the last calculation of every language model. 3
• square – a number times itself, x² = x·x; the operation is called squaring, its inverse is the root. 1
• stochastic gradient descent – gradient descent with noisy estimates from mini-batches instead of the loss over all data; according to Robbins and Monro (1951) it still reaches the minimum. 7
• surprise – information content of an event, log₂(1/p) bits: probability ½ gives 1 bit, 1/1,000 almost 10 bits, 1 gives 0 bits. The smaller the probability a model gave the right word, the larger. 12

T

• temperature – number by which the scores are divided before the softmax; small makes the distribution sharper, large flatter. The setting that makes chatbots more predictable or more inventive. 3
• tensor – table of numbers with more than two directions, a stack of matrices; namesake of TensorFlow. 8
• test data – held-back part of the data on which one checks whether the model also hits the unknown; protection against overfitting. 6
• threshold neuron – computing element of McCulloch and Pitts (1943): weighted sum of the inputs, output 1 if it reaches a threshold, otherwise 0; with suitable weights it computes “and”, “or” and “not”. 9
• token – piece of text a model computes with, usually a word or part of a word, obtained by merging frequent character pairs (BPE); GPT-3 knows 50,257. 13
• transformer – model of stacked layers with attention and a multilayer perceptron, without recurrence, that processes all tokens at once as matrices (Vaswani et al., 2017); the architecture of all large language models since 2018. 14
• transformer layer – attention with several heads followed by a multilayer perceptron, both with residual connection and normalisation; GPT-3 stacks 96 of them. 14
• transpose – swap rows and columns of a matrix, written Kᵀ; in the transformer formula Q·Kᵀ means that all queries form dot products with all keys. 8

U

• universal approximation theorem – a network with one sufficiently wide hidden layer can reproduce any continuous function as precisely as one likes (Cybenko, Hornik, 1989); does not say how wide, nor whether training finds the weights. 11

V

• value – vector a token contributes in attention when it is found; the values are added with the softmax weights. 14
• vanishing gradient – in deep networks with sigmoid curves the product of the slopes (at most 0.25 per layer) shrinks towards zero and the front layers do not learn (Hochreiter, 1991); cured by ReLU, residual connections and normalisation. 11
• variable – letter standing for a number that is not yet known or may change; Viète (1591) and Descartes (1637, x, y, z for unknowns). 1
• vector – list of numbers, readable as a point or arrow in space: (3, 4) has length 5; word from Hamilton (1846), calculation in any number of dimensions in Grassmann (1844). Embeddings, queries, keys and values are vectors. 8

W

• weighted sum – every input multiplied by its own number, everything added, a constant on top; the most frequent calculation in AI, the neuron of 1943, the regression of 1900, the dot product. 2
• weights – the multipliers in a weighted sum; they say how much each input counts, zero switches it off, negative subtracts it. Most parameters of a model are weights. 2