Optimization

Make the derivative, state, update, and recorded history explicit.

1 / 6

Trace the recurrence

For f=(x-3)^2, x=0, lr=0.25:

x values: 0 → 1.5 → 2.25 → 2.625.
Optimizer history contains initial plus post-update parameters.
Three steps give four entries.

2 / 6

Step size can defeat convexity

For f=x^2: x_next=(1-2*lr)*x.

lr=0.1 shrinks magnitude.
lr=1 oscillates.
lr>1 diverges from nonzero x.

3 / 6

Momentum preserves state

v_next=momentumv+gradient; x_next=x-lrv_next.

Initial velocity is zero.
At x=2, lr=0.1, momentum=0.5: x becomes 1.6 then 1.08.
No extra (1-momentum) factor in this exercise.

4 / 6

Mean squared error retains its factor two

X=[[0],[2]], y=[1,5], zero parameters:

Loss 13; dw=-10; db=-6.
lr=0.1 gives w=1, b=0.6.
Regression records pre-update loss [13]; new loss is 2.96.

5 / 6

Verify before adding complexity

Preserve independent parameter-history snapshots.
Duplicate the batch: mean gradient stays fixed.
Compare analytic gradients with finite differences.
Tiny-data fitting and held-out evaluation answer different questions.

6 / 6

←/→ or click edges to navigate · ? help · N notes · F fullscreen