Home / MLF / Week 2

Week 2 · Slopes, change and the steepest way up

Official syllabus line: Calculus

About 70 minQuiz 1 material (see exam radar)BSCS2004 · Harish Guruprasad Ramaswamy, Arun Rajkumar, Prashanth LA

Before you start: Week 1 (vectors and dot products). School-level algebra: expanding brackets, powers and roots.

Watch first: the animated story of this weekAbout 1-2 minutes of real 3D animation with a friendly voice. Pause any time.

Read the words of the video

Imagine riding a bike up a hill. Your friend asks how steep it is right now. You can't just say "I climbed 100 meters in 2 kilometers" because that's an average. They want the steepness at this exact spot. If you zoom in really close, the road looks like a straight line.
This "slope at one spot" is called the derivative. It's super important in machine learning, guiding us to reduce mistakes, just like finding the fastest way downhill. We'll learn to calculate it, use it to make curvy functions appear straight near a point, and extend this idea to many inputs.
To estimate square root 4.2 without a calculator, we use the known square root of 4, which is 2. The derivative of square root x at x equals 4 is one-fourth. Our estimate becomes 2 plus one-fourth times 0.2, giving us 2.05. This simple trick provides a very close answer.
For exams, always normalise direction vectors and evaluate partial derivatives at the specific point. Remember, differentiable means continuous, but not the reverse. The gradient points uphill fastest! Master the derivative as the slope, linear approximation, and the gradient's direction. Keep practicing, you've got this!

The one-minute story

Think of it like this

You are riding a bike up a hill. Your friend asks: "How steep is it right here?"

You cannot answer with "I climbed 100 metres in 2 kilometres", because that is an average over the whole ride. Your friend wants the steepness at this exact spot. If you zoom in on the road so closely that it looks like a straight line, the slope of that tiny straight piece is the answer.

That slope-at-one-spot is called the derivative. It is the single most useful idea in machine learning, because learning means "nudge the settings so the mistakes get smaller", and the derivative tells us which way is downhill.

This week has three jobs: understand the derivative; use it to replace a curvy function with a straight one close to a point (linear approximation); and then do it again when there are many inputs (partial derivatives, the gradient, and the directional derivative).

By the end of this week you can:

find slopes with the standard derivative rules and spot where a function is continuous or differentiable;
write the linear approximation (tangent line or tangent plane) of a function at a point;
compute partial derivatives, the gradient and a directional derivative;
find the best value of something by setting the derivative to zero (the fencing problem).

Words you need

Function - a machine: put a number in, get a number out. $f(x)=x^2$ turns 3 into 9.
Slope - how much the output rises when the input goes up by 1 (for a straight line).
Derivative $f'(x)$ - the slope of the curve at exactly one spot.
Continuous - you can draw the graph without lifting your pencil.
Differentiable - the graph is smooth enough at that spot to have one clear slope (no sharp corner).
Partial derivative - the slope when you move only one input and freeze the others.
Gradient $\nabla f$ - the vector that collects all the partial derivatives.

Idea 1: The derivative is the slope at a point

For a straight line $y=mx+c$ the slope is $m$ everywhere. For a curve the slope changes from place to place. Pick a point $x=a$ and a nearby point $x=a+h$ . The slope of the little line joining them is $\frac{f(a+h)-f(a)}{h}$ . Shrink $h$ towards 0 and that settles to a number, the derivative:

f'(a)=\lim_{h\to0}\frac{f(a+h)-f(a)}{h}

Think of it like this

Limit just means "what number are we creeping up on?" If you walk halfway to a wall, then half the remaining way, and so on, your distance to the wall is creeping up on 0 even though you never get there. We say the limit is 0.

The rules you actually need

Function	Derivative
constant $c$	$0$
$x^n$	$n\,x^{n-1}$
$e^x$	$e^x$
$\ln x$	$1/x$
$\sin x$ , $\cos x$	$\cos x$ , $-\sin x$
$c\,f(x)$	$c\,f'(x)$
$f+g$	$f'+g'$
$f\cdot g$ (product)	$f'g+fg'$
$f(g(x))$ (chain)	$f'(g(x))\cdot g'(x)$

The chain rule deserves a picture. If $g$ is "double it" and $f$ is "square it", then $f(g(x))=(2x)^2$ . Changing $x$ a little changes $g$ by 2 times as much, and that change then gets multiplied by $f'$ . Rates multiply along the chain: $\frac{d}{dx}(2x)^2=2(2x)\cdot2=8x$ .

Continuous and differentiable: two different promises

The idea

Continuous at $a$ : the left limit, the right limit and the value $f(a)$ are all the same number. No jump, no hole, no blow-up.
Differentiable at $a$ : the slope from the left equals the slope from the right.

Differentiable $\Rightarrow$ continuous, but not the other way round. The graph of $|x|$ is continuous everywhere, yet at $x=0$ it has a sharp V-corner: slope $-1$ on the left, $+1$ on the right, so no single slope. Not differentiable at 0.

Watch out - real exam trap

For a piecewise function, check two things at the joining point: do the values match (continuity), then do the slopes match (differentiability)? Papers love "how many points is this function discontinuous / non-differentiable?" Count each joint separately.

Idea 2: Linear approximation - replace the curve by its tangent

Zoom in far enough on any smooth curve and it looks like a straight line: the tangent line. So near $x=a$ we can pretend:

f(x)\;\approx\;L(x)=f(a)+f'(a)\,(x-a)

Think of it like this

You know a fact exactly at one spot (the value $f(a)$ ) and you know how fast things change there (the slope $f'(a)$ ). Then a good guess for a spot a little way off is: start value + slope $\times$ distance moved. That is how you estimate that a car doing 60 km/h will be about 6 km further after 6 minutes, without solving the whole trip.

Solved example: 1. Estimating a square root without a calculator

Question. Estimate $\sqrt{4.2}$ using the linear approximation of $f(x)=\sqrt x$ at $a=4$ .

Solution.

$f(4)=2$ .
$f'(x)=\frac1{2\sqrt x}$ , so $f'(4)=\frac14$ .
$L(x)=2+\frac14(x-4)$ .
$L(4.2)=2+\frac14(0.2)=2.05$ .

The true value is $2.0494\ldots$ , so the guess is off by less than 0.001. Not bad for arithmetic you can do in your head.

Idea 3: Many inputs - partial derivatives and the gradient

Real models have many inputs: a house price depends on area and bedrooms and age. So take $f(x,y)$ , a surface over a flat floor.

A partial derivative asks: "if I walk only in the $x$ direction and keep $y$ frozen, what is the slope?" To compute it, treat the other variables as ordinary numbers.

f_x=\frac{\partial f}{\partial x},\qquad f_y=\frac{\partial f}{\partial y}

Solved example: 2. Partial derivatives

Question. $f(x,y)=x^2y+3y^2$ . Find $f_x$ and $f_y$ at $(1,2)$ .

Solution. Freeze $y$ and differentiate in $x$ : $f_x=2xy$ . Freeze $x$ and differentiate in $y$ : $f_y=x^2+6y$ . At $(1,2)$ : $f_x=2\cdot1\cdot2=4$ , $f_y=1+12=13$ .

Put the partials into one vector and you get the gradient:

\nabla f=\left(\frac{\partial f}{\partial x},\;\frac{\partial f}{\partial y}\right)

Think of it like this

Stand on a hillside in thick fog. You cannot see the top, but you can feel the ground under your feet. The gradient is an arrow, drawn on the ground, that points in the direction the slope climbs fastest, and its length tells you how steep that is. Walk against the arrow and you go downhill fastest. That one sentence is the whole of gradient descent in week 7.

Linear approximation in two variables (the tangent plane)

The tangent line becomes a tangent plane:

L(x,y)=f(a,b)+f_x(a,b)\,(x-a)+f_y(a,b)\,(y-b)

In vector form that is $L(\mathbf x)=f(\mathbf a)+\nabla f(\mathbf a)\cdot(\mathbf x-\mathbf a)$ , the same shape as the one-variable version with a dot product instead of a multiplication.

Solved example: 3. The best linear approximation (a favourite exam question)

Question. Find the best linear approximation of $f(x,y)=\sqrt{x+y}$ around $(4,5)$ .

Solution.

Value: $f(4,5)=\sqrt9=3$ .
Partial derivatives: $f_x=\dfrac{1}{2\sqrt{x+y}}$ and, because $y$ enters the same way, $f_y=\dfrac1{2\sqrt{x+y}}$ . At $(4,5)$ both equal $\dfrac1{2\cdot3}=\dfrac16$ .
Plug in: $L(x,y)=3+\dfrac16(x-4)+\dfrac16(y-5)$ .

Always do the three steps in this order: value, partials, plug in. Most wrong answers come from forgetting to evaluate the partials at the point, or from mixing up $(a,b)$ and $(x,y)$ .

Idea 4: The directional derivative

The partials tell you the slope when you walk exactly east or exactly north. What if you walk north-east? Pick a unit vector $\mathbf u$ for the direction (length 1, see week 1). The slope in that direction is

D_{\mathbf u}f=\nabla f\cdot\mathbf u

Since $\nabla f\cdot\mathbf u=\|\nabla f\|\cos\theta$ :

it is biggest ( $=\|\nabla f\|$ ) when $\mathbf u$ points along $\nabla f$ ( $\theta=0$ ): steepest climb;
it is most negative when $\mathbf u$ points against $\nabla f$ : steepest descent;
it is zero when $\mathbf u\perp\nabla f$ : walking along a level contour, neither up nor down.

Solved example: 4. Directional derivative

Question. $f(x,y)=x^2y$ . Find the rate of change at $(1,2)$ in the direction of $\mathbf v=(3,4)$ .

Solution.

$\nabla f=(2xy,\;x^2)$ , so at $(1,2)$ : $\nabla f=(4,1)$ .
Normalise the direction: $\|\mathbf v\|=5$ , so $\mathbf u=(0.6,\,0.8)$ .
$D_{\mathbf u}f=4(0.6)+1(0.8)=2.4+0.8=\mathbf{3.2}$ .

If you had skipped step 2 and used $(3,4)$ directly you would get $12+4=16$ , five times too big. The exam always offers that wrong number as an option.

Idea 5: Where is the best? Set the slope to zero

At the top of a hill or the bottom of a valley the ground is flat, so the slope is zero. To find best values, solve $f'(x)=0$ .

Solved example: 5. The fencing problem

Question. A farmer has 120 m of fence and wants to enclose a rectangular field along a straight river. The river side needs no fence. What is the largest area?

Solution.

Let the two sides that touch the river be $x$ each. The side parallel to the river is $120-2x$ .
Area $A(x)=x(120-2x)=120x-2x^2$ .
$A'(x)=120-4x=0\Rightarrow x=30$ .
Check it is a maximum: $A''(x)=-4<0$ (the curve bends downwards).
$A(30)=30\times60=\mathbf{1800\ m^2}$ .

Same recipe every time: write the quantity in one variable, differentiate, set to zero, check the bend.

Your turn

Your turn: 1. Derivatives

Differentiate: (a) $x^3-5x+2$ , (b) $e^{3x}$ , (c) $x\ln x$ .

Need a hint?

Use the table: power rule for (a), chain rule for (b), product rule for (c).

Show the full solution

(a) $3x^2-5$ . (b) Chain rule: outer $e^{u}$ , inner $u=3x$ : $e^{3x}\cdot3=3e^{3x}$ . (c) Product rule: $1\cdot\ln x+x\cdot\frac1x=\ln x+1$ .

Your turn: 2. Linear approximation

Approximate $\ln(1.1)$ using the linear approximation of $f(x)=\ln x$ at $a=1$ .

Need a hint?

$f(1)=0$ and $f'(x)=1/x$ .

Show the full solution

$f'(1)=1$ , so $L(x)=0+1\cdot(x-1)=x-1$ . Then $L(1.1)=0.1$ . (The true value is $0.0953$ .)

Your turn: 3. Tangent plane

Find the linear approximation of $f(x,y)=x^2+xy$ at $(1,1)$ and use it to estimate $f(1.1,\,0.9)$ .

Need a hint?

$f(1,1)=2$ . $f_x=2x+y=3$ , $f_y=x=1$ at $(1,1)$ .

Show the full solution

$L(x,y)=2+3(x-1)+1(y-1)$ . At $(1.1,0.9)$ : $2+3(0.1)+1(-0.1)=2+0.3-0.1=\mathbf{2.2}$ . The true value is $1.21+0.99=2.2$ exactly here.

Your turn: 4. Which way is steepest?

$f(x,y)=x^2+3y^2$ . At $(2,1)$ , in which direction does $f$ increase fastest, and how fast?

Need a hint?

$\nabla f=(2x,\,6y)$ .

Show the full solution

$\nabla f(2,1)=(4,6)$ . Fastest increase is along $(4,6)$ , i.e. the unit vector $\frac1{\sqrt{52}}(4,6)$ , at rate $\|\nabla f\|=\sqrt{16+36}=\sqrt{52}\approx7.21$ .

Your turn: 5. Maximise

A gardener has 200 m of fencing and builds a rectangular garden against a straight wall (no fence on the wall side). Find the maximum area.

Need a hint?

Same shape as the farmer problem, with 200 instead of 120.

Show the full solution

$A=x(200-2x)$ , $A'=200-4x=0\Rightarrow x=50$ , $A=50\times100=\mathbf{5000\ m^2}$ .

Exam radar

Of the 53 MLF Quiz 1 papers, the summaries mention linear approximation in 28, directional derivatives in 14, partial derivatives in 11, continuity and differentiability in 10 and limits in 7. That makes this the heaviest single topic in Quiz 1.

In the older End Term papers, "the linear approximation $L=ax+by+c$ at $(2,3)$ , find $a$ " appears 21 times, and the fencing problem (with 120 m or 200 m) about 8 times.

Typical marks: 2 to 3 per question, mostly MCQ and one numeric (SA).

Watch out - real exam trap

Forgetting to normalise the direction vector in a directional derivative.
Not evaluating partials at the point before plugging into $L$ .
Saying "continuous, therefore differentiable". It is the other way round.
For $\sqrt{x+y}$ , remember the chain rule: the partial in $x$ is $\frac1{2\sqrt{x+y}}$ , not $\frac1{2\sqrt x}$ .
The gradient points uphill. Descent goes the opposite way.

Quick quiz

A function is differentiable at a point. Which statement must be true?

It is continuous at that point.
Its graph has a sharp corner there.
Its second derivative exists there.
It is zero there.

Differentiable implies continuous. A sharp corner is exactly what stops a function being differentiable. The other two options do not follow.

For f(x,y) = x^2 y at (1,2), the gradient is...

(2, 1)
(4, 1)
(1, 4)
(2, 4)

f_x = 2xy = 4 and f_y = x^2 = 1 at (1,2).

The direction in which a function increases fastest at a point is...

the direction of the second derivative
perpendicular to the gradient
the direction of the gradient
the direction opposite to the gradient

The gradient points uphill fastest. The opposite direction is steepest descent. Perpendicular to it is a level direction with rate zero.

Which of these are true? (select all) (select all that apply)

The directional derivative uses a unit vector.
The maximum directional derivative at a point equals the length of the gradient.
The linear approximation of f at a is f(a) + f'(a) x.
|x| is differentiable at 0.

The tangent line is f(a) + f'(a)(x - a), not f(a) + f'(a)x. |x| has a corner at 0, so it is not differentiable there.

Recap

Remember this

The derivative is the slope at one spot. Differentiable $\Rightarrow$ continuous, not the reverse.
Linear approximation: $L(x)=f(a)+f'(a)(x-a)$ ; two variables: $L=f(a,b)+f_x(x-a)+f_y(y-b)$ . Order of work: value, partials, plug in.
Gradient $\nabla f$ = vector of partials, points uphill. $D_{\mathbf u}f=\nabla f\cdot\mathbf u$ with $\|\mathbf u\|=1$ . Max rate $=\|\nabla f\|$ .
Optimise by setting the derivative to zero and checking the bend.

Formulas to keep

Derivative: $f'(a)=\lim_{h\to0}\frac{f(a+h)-f(a)}{h}$
Power: $(x^n)'=nx^{n-1}$ ; chain: $(f(g))'=f'(g)g'$ ; product: $(fg)'=f'g+fg'$
Linear approximation: $L(x)=f(a)+f'(a)(x-a)$
Two variables: $L(x,y)=f(a,b)+f_x(a,b)(x-a)+f_y(a,b)(y-b)$
Gradient: $\nabla f=(f_x,f_y)$ ; directional derivative $D_{\mathbf u}f=\nabla f\cdot\mathbf u$ , $\|\mathbf u\|=1$
Steepest rate: $\|\nabla f\|$ along $\nabla f$

Connected weeks (learn them together)

MLT week 3 - K-means: grouping without labels: K-means rests on one calculus fact: the average is the number that makes squared distances smallest.
MLT week 4 - MLE, Bayes, Gaussian mixtures and EM: Maximum likelihood means 'set the derivative to zero', which is MLF week 2 in action.

Practise with real exam questions

These come straight from past papers (with the real figures and the official answer key). Open 38 real questions on this week All Quiz 1 papers