Home / MLF / Week 11

Week 11 · The exponential family

Official syllabus line: Exponential Family of distributions

About 60 minEnd Term material (see exam radar)BSCS2004 · Harish Guruprasad Ramaswamy, Arun Rajkumar, Prashanth LA

Before you start: Week 8 (convex functions) and week 10 (distributions, expectation, variance). Logarithms and exponentials.

Watch first: the animated story of this weekAbout 1-2 minutes of real 3D animation with a friendly voice. Pause any time.

Read the words of the video

Just like baking, many probability distributions share a secret "recipe card." You change the ingredients, but the basic layout stays the same. This means learning how to handle one distribution's recipe helps you understand many others, saving you lots of time and effort!
This shared recipe card is called the 'exponential family'. It lets us write many distributions like coin flips or bell curves in one special form. The magic is, if you learn how to find the mean or fit one of these models using this recipe, you've learned it for all of them!
Take the Poisson distribution, for counting things like buses. We can write its formula in our special recipe card form. Here, our natural parameter, 'eta', becomes 'log lambda', where 'lambda' is the average count. The 'T of x' part just becomes 'x', our actual count.
While the name 'exponential family' might not appear on exams, its core ideas are super important for other topics like MLE and regression! Remember the recipe card, how 'A prime of eta' gives the mean, and how to find the best parameter by matching model mean to sample mean. Keep practicing!

The one-minute story

Think of it like this

Bake cakes, bread and biscuits and you will notice they all start from the same kind of recipe card: flour, a sweetener, something to make it rise, a little oven time. The card has a fixed layout and you only change the amounts.

Many of the probability distributions you already know (coin flips, waiting times, counts of buses, the bell curve) secretly share one recipe card too. It is called the exponential family. The beauty: if you learn how to do something once on the recipe card (find the mean, find the best parameter from data, fit a model), you have learned it for all of them at once.

That is the reason logistic regression, linear regression and many other models look so alike: they are all the same recipe with different ingredients.

Tip

How much to invest in this week. Looking at every MLF paper on the practice site, the exponential family is not asked by name in any of them, but the ideas here (natural parameter, log-partition function, "model mean = sample mean") are the foundation of the MLE questions that are asked, and of logistic regression in MLT and MLP. Learn the recipe, practise the Bernoulli and Poisson conversions, and move on.

By the end of this week you can:

write a distribution in the exponential-family form and name its parts;
show that Bernoulli, Poisson, exponential and Gaussian (known variance) belong to the family;
get the mean and variance from the log-partition function;
find the maximum-likelihood estimate by matching the model mean to the sample mean.

Words you need

Exponential family - distributions that can be written as $p(x\mid\eta)=h(x)\exp\big(\eta\,T(x)-A(\eta)\big)$ .
Natural parameter $\eta$ - the "dial" of the family, in its most convenient form.
Sufficient statistic $T(x)$ - the one summary of each data point that matters. For many data points you only need $\sum T(x_i)$ .
Log-partition function $A(\eta)$ - the "normaliser" that makes the total probability equal 1.
Base measure $h(x)$ - a background factor that does not depend on the parameter.

Idea 1: The recipe card

\boxed{p(x\mid\eta)=h(x)\,\exp\!\big(\eta\,T(x)-A(\eta)\big)}

(One parameter for simplicity; with several, $\eta T(x)$ becomes a dot product $\boldsymbol\eta^T\mathbf T(x)$ .)

$T(x)$ : what you record from a data point.
$\eta$ : the parameter that multiplies it inside the exponent.
$A(\eta)$ : whatever you must subtract so the probabilities add up to 1: $A(\eta)=\ln\sum_xh(x)e^{\eta T(x)}$ .
$h(x)$ : leftover factor that does not involve $\eta$ .

To show a distribution is in the family: take its formula, write it as $\exp(\ln\ldots)$ , and sort the terms into "involves both $x$ and the parameter" ( $\eta T(x)$ ), "involves only the parameter" ( $-A$ ), and "involves only $x$ " ( $h$ ).

Solved example: 1. Bernoulli (a coin with heads-probability

p

)

p(x)=p^x(1-p)^{1-x}=\exp\!\Big(x\ln\tfrac p{1-p}+\ln(1-p)\Big),\qquad x\in\{0,1\}

Sorting: $T(x)=x$ , $\eta=\ln\dfrac p{1-p}$ (the log-odds, or logit), $h(x)=1$ , and since $p=\dfrac{e^\eta}{1+e^\eta}$ we get $\ln(1-p)=-\ln(1+e^\eta)$ , so $A(\eta)=\ln(1+e^\eta)$ .

Now invert $\eta=\ln\frac p{1-p}$ : $p=\dfrac1{1+e^{-\eta}}$ . That is the sigmoid from week 9! Logistic regression is just "set $\eta=\mathbf w^T\mathbf x$ for a Bernoulli".

Solved example: 2. Poisson (counts of events in a fixed time, mean

\lambda

)

p(x)=\frac{\lambda^xe^{-\lambda}}{x!}=\frac1{x!}\exp\big(x\ln\lambda-\lambda\big),\qquad x=0,1,2,\dots

$T(x)=x$ , $\eta=\ln\lambda$ , $h(x)=\frac1{x!}$ , $A(\eta)=\lambda=e^\eta$ .

Solved example: 3. Exponential distribution and Gaussian with known variance

Exponential ( $\lambda e^{-\lambda x}$ for $x\ge0$ ): write it as $\exp(-\lambda x+\ln\lambda)$ . So $T(x)=x$ , $\eta=-\lambda$ (negative!), $h=1$ , $A(\eta)=-\ln(-\eta)$ .

Gaussian with variance 1 ( $\mu$ unknown): $\frac1{\sqrt{2\pi}}e^{-(x-\mu)^2/2}=\underbrace{\tfrac1{\sqrt{2\pi}}e^{-x^2/2}}_{h(x)}\exp\!\big(\mu x-\tfrac{\mu^2}2\big)$ . So $T(x)=x$ , $\eta=\mu$ , $A(\eta)=\frac{\eta^2}2$ .

Distributions not in the family: the uniform on $[0,\theta]$ (its range depends on the parameter), the Cauchy, and the Student- $t$ .

Idea 2: The log-partition function knows the mean and variance

Differentiate $A$ :

\boxed{A'(\eta)=E[T(X)],\qquad A''(\eta)=\operatorname{Var}[T(X)]}

One function holds the whole story. Check with what we found:

Distribution	$A(\eta)$	$A'(\eta)=E[X]$	$A''(\eta)=\operatorname{Var}[X]$
Bernoulli	$\ln(1+e^\eta)$	$\frac{e^\eta}{1+e^\eta}=p$	$p(1-p)$
Poisson	$e^\eta$	$e^\eta=\lambda$	$e^\eta=\lambda$
Exponential	$-\ln(-\eta)$	$-\frac1\eta=\frac1\lambda$	$\frac1{\eta^2}=\frac1{\lambda^2}$
Gaussian (var 1)	$\eta^2/2$	$\eta=\mu$	$1$

Every row reproduces a mean and variance you already knew from week 10. And because a variance is never negative, $A''\ge0$ , which means $A$ is a convex function (week 8). That convexity is exactly what makes fitting these models a nice optimisation problem.

Idea 3: Fitting the family by maximum likelihood

Suppose you observe $n$ independent data points $x_1,\dots,x_n$ . The log-likelihood of $\eta$ is

\ell(\eta)=\eta\sum_iT(x_i)-nA(\eta)+\text{(terms not involving }\eta)

Differentiate and set to zero: $\sum_iT(x_i)-nA'(\eta)=0$ , i.e.

\boxed{A'(\hat\eta)=\frac1n\sum_{i=1}^nT(x_i)}\qquad\text{"model mean = sample mean"}

Since $A''\ge0$ , the log-likelihood is concave in $\eta$ , so this equation gives the global maximum. This is called moment matching.

Solved example: 4. MLE in one line each

Bernoulli. $A'(\eta)=p$ , so $\hat p=\frac1n\sum x_i=\bar x$ (the fraction of heads).

Poisson. $A'(\eta)=\lambda$ , so $\hat\lambda=\bar x$ . For the counts $(2,\,0,\,3,\,1,\,4)$ : $\bar x=\frac{10}5=\mathbf2$ .

Exponential. $A'(\eta)=\frac1\lambda=\bar x$ , so $\hat\lambda=\frac1{\bar x}$ . For waiting times $(2,4,6)$ : $\bar x=4$ , so $\hat\lambda=\mathbf{0.25}$ .

Gaussian, variance 1. $\hat\mu=\bar x$ .

Notice the sufficient statistic at work: to estimate $p$ , $\lambda$ or $\mu$ you only needed the total $\sum x_i$ . The individual values were not needed.

Idea 4: Why this matters for the other courses

The idea

In a generalised linear model you choose a member of the family for the thing you are predicting, and set its natural parameter to a linear function of the inputs: $\eta=\mathbf w^T\mathbf x$ .

You predict	Family	Model
a real number with bell-shaped noise	Gaussian	linear regression (MLT week 5)
yes / no	Bernoulli	logistic regression (MLT week 9)
a count (0, 1, 2, ...)	Poisson	Poisson regression
one of $K$ categories	categorical	softmax (neural nets, MLT week 12)

Your turn

Your turn: 1. Identify the parts

Write the Poisson( $\lambda$ ) distribution in exponential-family form and name $T$ , $\eta$ , $h$ and $A$ .

Need a hint?

Take $\exp(\ln\ldots)$ of the formula $\frac{\lambda^xe^{-\lambda}}{x!}$ .

Show the full solution

$\frac1{x!}\exp(x\ln\lambda-\lambda)$ . So $T(x)=x$ , $\eta=\ln\lambda$ , $h(x)=\frac1{x!}$ , $A(\eta)=e^\eta$ .

Your turn: 2. Mean from the log-partition

For the Bernoulli, $A(\eta)=\ln(1+e^\eta)$ . Compute $A'(\eta)$ and check it is the mean.

Need a hint?

Differentiate $\ln(1+e^\eta)$ with the chain rule.

Show the full solution

$A'(\eta)=\frac{e^\eta}{1+e^\eta}$ . With $\eta=\ln\frac p{1-p}$ , $e^\eta=\frac p{1-p}$ , so $A'=\frac{p/(1-p)}{1+p/(1-p)}=p$ . ✓

Your turn: 3. Maximum likelihood by moment matching

Defects per batch of a product are modelled as Poisson. Five batches show $3,\,5,\,4,\,6,\,2$ defects. What is the MLE of $\lambda$ ?

Need a hint?

Poisson: model mean $=\lambda$ .

Show the full solution

$\hat\lambda=\bar x=\frac{20}5=\mathbf4$ .

Your turn: 4. Exponential MLE

Five service times (minutes) are $1,\,2,\,2,\,3,\,2$ . Estimate the rate $\lambda$ of an exponential model, and the probability that a service takes over 3 minutes.

Need a hint?

$\hat\lambda=1/\bar x$ and $P(X>t)=e^{-\lambda t}$ .

Show the full solution

$\bar x=2$ , so $\hat\lambda=0.5$ . $P(X>3)=e^{-1.5}\approx0.223$ .

Your turn: 5. Is it in the family?

Is the uniform distribution on $[0,\theta]$ in the exponential family?

Need a hint?

Look at where $\theta$ appears.

Show the full solution

No. Its density is $\frac1\theta$ only on $0\le x\le\theta$ , so the support (where the density is non-zero) depends on $\theta$ . In the family the support cannot depend on the parameter.

Exam radar

Be honest about priorities. Across all MLF papers on the practice site, none of the concept summaries mention the exponential family, natural parameter or sufficient statistic. What is examined near this week: MLE (in 16 of the 42 End Term summaries), Poisson and Bernoulli facts (16 and 11), and the exponential distribution story problems (waiting times). So treat this week as the foundation behind those, and for logistic regression in MLT and MLP.

If your term's paper does ask about it, the likely question is "write this distribution in exponential-family form" or "find $A'(\eta)$ " (Examples 1 to 3).

Watch out - real exam trap

For the exponential distribution, $\eta=-\lambda$ is negative, and $A(\eta)=-\ln(-\eta)$ needs $\eta<0$ .
$T(x)$ is not always $x$ (for the Gaussian with unknown variance it is $(x,x^2)$ ).
"Moment matching" means model mean $A'(\eta)$ = sample mean of $T(x)$ , not of $x$ in general.
A distribution whose range depends on the parameter is not in the family.

Quick quiz

For the Poisson distribution written in exponential-family form, the natural parameter eta is...

lambda
ln lambda
1 / lambda
e to the lambda

Poisson(x) is (1/x!) exp(x ln(lambda) - lambda), so eta = ln(lambda) and A(eta) = e^eta.

For an exponential-family distribution, A prime of eta equals...

the variance
the expected value of T(X)
the likelihood
zero

The first derivative of the log-partition function is the mean of the sufficient statistic; the second derivative is its variance.

The MLE of the Poisson rate from counts 1, 3, 2 is...

The MLE is the sample mean: (1+3+2)/3 = 2.

Select all TRUE. (select all that apply)

The Bernoulli natural parameter is the log-odds.
The log-partition function A is convex.
The uniform distribution on [0, theta] is in the exponential family.
The sufficient statistic is always a constant.

The uniform's support depends on theta, so it is not in the family. A'' is a variance and hence non-negative, so A is convex.

Recap

Remember this

Recipe card: $p(x\mid\eta)=h(x)\exp(\eta T(x)-A(\eta))$ .
Bernoulli: $\eta=\ln\frac p{1-p}$ (logit; inverse is the sigmoid). Poisson: $\eta=\ln\lambda$ . Exponential: $\eta=-\lambda$ . Gaussian (var 1): $\eta=\mu$ .
$A'(\eta)=E[T]$ , $A''(\eta)=\operatorname{Var}[T]\ge0$ so $A$ is convex.
MLE: $A'(\hat\eta)=\frac1n\sum T(x_i)$ , "model mean = sample mean". Log-likelihood is concave.
Setting $\eta=\mathbf w^T\mathbf x$ gives linear regression, logistic regression, Poisson regression, softmax.

Formulas to keep

$p(x\mid\eta)=h(x)\exp(\eta T(x)-A(\eta))$
$E[T]=A'(\eta)$ , $\operatorname{Var}[T]=A''(\eta)$
MLE: $A'(\hat\eta)=\frac1n\sum_iT(x_i)$
Bernoulli $\hat p=\bar x$ ; Poisson $\hat\lambda=\bar x$ ; Exponential $\hat\lambda=1/\bar x$
Logit: $\eta=\ln\frac p{1-p}$ ; inverse: $p=\frac1{1+e^{-\eta}}$

Connected weeks (learn them together)

MLT week 9 - Perceptron and logistic regression: Logistic regression is a member of the exponential family of models.

Practise with real exam questions

These come straight from past papers (with the real figures and the official answer key). Open 2 real questions on this week All End Term papers