Glossary

the share of predictions that are right. Misleading when one class is rare.

the "squish" a neuron applies to its score (sigmoid, tanh, ReLU). Without it a deep network is just a linear model.

a relay of weak learners; each one concentrates on the examples the previous ones got wrong, and better learners get a louder vote.

how many times an eigenvalue appears as a root of the characteristic equation.

the area under the ROC curve. 0.5 is guessing, 1.0 is perfect ranking.

the chain rule run backwards through a network so one pass gives the gradient of every weight.

training many copies of a model on random samples (with replacement) and voting. Calms jumpy models (cuts variance).

a smallest set of independent vectors that can build everything in a space.

flips a probability around: $P(A|B)=P(B|A)P(A)/P(B)$.

a bell-like curve on $[0,1]$, used as a belief about a coin's heads-probability. Beta($\alpha,\beta$) acts like $\alpha$ pretend heads and $\beta$ pretend tails.

error from a model being too simple to capture the pattern. Underfitting.

the constant added to a neuron's weighted sum.

a score for choosing how many mixture components: fit minus a penalty for extra parameters. Lower is better.

training models one after another, each fixing the previous ones' mistakes. Cuts bias.

a sample of size $n$ drawn from $n$ items with replacement.

the price of a mistake. In scikit-learn it is the INVERSE of regularisation: small C means strong regularisation.

averages of many independent trials are roughly bell-shaped, whatever the original shape.

subtracting the mean so the data cloud is centred at the origin. Essential before PCA.

$\det(A-\lambda I)=0$. Its roots are the eigenvalues.

$P(|X-\mu|\ge k\sigma)\le1/k^2$. A guarantee that few values are far from the average.

predicting a category.

sorting unlabelled data into groups of similar items.

everything you can build as $A\mathbf x$; the span of the columns.

a table of true vs predicted classes. In scikit-learn: [[TN, FP], [FN, TP]].

a prior chosen so the posterior is the same kind of curve (Beta prior with coin-flip data gives a Beta posterior).

a function you can draw without lifting the pencil.

bowl-shaped; the straight chord between two points lies on or above the graph. Every local minimum is the global minimum.

the smallest convex set containing some points (stretch a rubber band round them).

a shape where the straight line between any two of its points stays inside it.

a table of variances (diagonal) and how pairs of features move together. Symmetric and positive semi-definite.

the loss for probabilities: $-\ln$ of the probability given to the right answer.

rotate which slice of the training data is held out for validation, and average the scores.

in very many dimensions all points look equally far apart, so "nearest" loses meaning.

test-set information sneaking into training, giving falsely good scores. Always fit on training data only.

clustering by density: finds any-shaped groups and labels loners as noise ($-1$).

the line (or surface) where the classifier switches its answer.

a flowchart of yes/no questions built by always choosing the split that makes the groups purest.

the slope of a curve at one exact spot.

a number saying how a matrix scales area; the product of the eigenvalues. Zero means it squashes space flat.

a matrix with enough independent eigenvectors to write $A=PDP^{-1}$.

describing data with fewer numbers while keeping what matters.

the slope in a chosen direction: $\nabla f\cdot\mathbf u$ with $\mathbf u$ a unit vector.

multiply matching entries and add. Zero means perpendicular.

the same optimisation written with Lagrange multipliers; for the SVM it uses only dot products between points.

the stretch factor of an eigenvector.

a direction a matrix only stretches or flips, never turns.

alternate between guessing hidden labels (E-step, responsibilities) and refitting parameters using those guesses (M-step).

several models combined into one stronger model.

how mixed a group is: 0 when pure, 1 bit for a 50/50 two-class mix.

one full pass over the training data.

any scikit-learn object that learns from data with fit.

distributions that share one recipe card: $h(x)\exp(\eta T(x)-A(\eta))$.

the harmonic mean of precision and recall. High only if both are high.

one measurable fact about a data point.

rewriting features to comparable ranges (standardise or min-max).

data from several hidden bell-curve groups, with soft memberships.

models how data is produced for each class, $P(\mathbf x|y)P(y)$. Naive Bayes is one.

$1-\sum p_k^2$, a mixedness measure for trees.

the vector of partial derivatives; points uphill.

each new tree is trained on the leftover errors (residuals) of the team so far.

repeatedly step against the gradient to find a minimum.

the table of kernel values $k(\mathbf x_i,\mathbf x_j)$. Must be symmetric PSD.

tries every combination of hyperparameters with cross-validation. Fits = combinations x folds.

the matrix of second derivatives; measures curvature.

$\max(0,1-yf(\mathbf x))$: zero for points safely beyond the margin.

a setting you choose before training (tree depth, learning rate, C).

filling in missing values.

knowing one event tells you nothing about the other.

the k-means objective: total squared distance from points to their centres.

parent entropy minus the size-weighted entropy of the children.

alternate "assign each point to its nearest centre" and "move each centre to the mean of its points".

choose starting centres one by one, with probability proportional to squared distance from the centres already chosen.

predict from the $k$ most similar stored examples.

$k(\mathbf x,\mathbf y)=\phi(\mathbf x)^T\phi(\mathbf y)$: the dot product after lifting, computed without lifting.

replacing dot products by a kernel so linear methods draw curved boundaries.

stationarity, primal feasibility, dual feasibility ($\mu\ge0$), complementary slackness ($\mu g=0$).

a number that tells how hard a constraint is pushing at the optimum.

least squares with an $\ell_1$ penalty; sets some weights exactly to zero.

a hidden quantity the model needs but we never observe (which cluster a point came from).

the step size $\eta$ in gradient descent.

the probability of the observed data, viewed as a function of the parameter.

replacing a curve by its tangent line (plane) near a point: $f(a)+f'(a)(x-a)$.

no vector in the set can be built from the others.

fitting a straight line (plane) by least squares.

a linear score squeezed through a sigmoid to give a probability.

a number saying how wrong a model is.

the peak of the posterior: the most probable parameter after seeing data and the prior.

the width of the empty road between the classes: $2/\|\mathbf w\|$.

for $X\ge0$, $P(X\ge a)\le E[X]/a$.

a rectangle of numbers; $A\mathbf x$ mixes the columns of $A$.

pick the parameter that makes the observed data most probable.

the average of the squared errors.

for the exponential distribution, having waited already does not change the chance of waiting longer.

one label of many / several yes-no labels / several targets at once.

classify with Bayes' rule, pretending features are independent given the class.

layers of neurons, each a weighted sum plus a non-linear squish.

the length of a vector, $\sqrt{\sum v_i^2}$.

$X^TX\mathbf w=X^T\mathbf y$: the least-squares solution.

all $\mathbf x$ with $A\mathbf x=\mathbf 0$. Dimension $n-r$.

one 0/1 column per category.

multiclass tricks needing $K(K-1)/2$ and $K$ binary classifiers.

perpendicular unit columns; keeps lengths; $Q^{-1}=Q^T$.

the examples a bootstrap sample missed (about 37%); a free validation set.

great on training data, poor on new data. The model memorised noise.

find the directions of greatest variance (eigenvectors of the covariance matrix) and keep the top few.

a linear classifier updated only when it makes a mistake: $\mathbf w\leftarrow\mathbf w+y\mathbf x$.

a chain of transformers followed by an estimator, all fitted together.

$\mathbf x^TA\mathbf x>0$ for all non-zero $\mathbf x$; all eigenvalues positive; a bowl.

$\mathbf x^TA\mathbf x\ge0$; all eigenvalues $\ge0$. Covariance matrices and Gram matrices are PSD.

your belief after seeing data: likelihood x prior.

of everything called positive, how much really was: $TP/(TP+FP)$.

your belief before seeing data.

the shadow of a vector on a line or subspace; the closest point there.

bagged trees where each split may only use a random subset of features.

the number of independent columns (equals the number of independent rows).

of all truly positive cases, how many were found: $TP/(TP+FN)$.

predicting a number.

a penalty that keeps the model simple (ridge, LASSO).

the activation $\max(0,z)$.

the leftover: true value minus fitted value.

the probability that a point came from a given component.

least squares with an $\ell_2$ penalty: $(X^TX+\lambda I)^{-1}X^T\mathbf y$.

true-positive rate against false-positive rate as the threshold moves.

$1/(1+e^{-z})$, an S-curve squeezing any number into (0,1).

any matrix equals rotate, stretch, rotate: $A=U\Sigma V^T$.

an SVM that lets a few points break the rules, at price $C$.

turns a list of scores into probabilities that add to 1.

a real symmetric matrix has real eigenvalues and perpendicular eigenvectors: $A=Q\Lambda Q^T$.

train a meta-model on the predictions of other models.

a split that keeps the class proportions the same in train and test.

learning from examples that come with the right answers.

a training point on or inside the margin; only these decide the SVM boundary.

$A=A^T$.

the score cut-off for calling something positive. Raising it raises precision and lowers recall.

the sum of the diagonal entries; equals the sum of the eigenvalues.

the model is too simple; poor on training and test data alike.

finding structure in data without labels.

how much the learned model changes when the training sample changes. Overfitting.

how spread out a random variable is around its mean.

a list of numbers; a point or an arrow.

a model only slightly better than guessing, like a one-question tree (a stump).