All simulators
Gradient Descent vs Newton
Race the two algorithms on an ill-conditioned bowl and on a curved valley; tune the step size until it breaks.
Day 25: Unconstrained Optimisation & ConvexitySteps to reach the minimum
46
within 0.001
f at last point
0.000000
2/λmax
0.200
η below: converges
Click to choose a starting point. Gradient descent follows the downhill direction with a fixed step; Newton jumps to the minimum of the local quadratic approximation. On the ill-conditioned quadratic, gradient descent zig-zags or diverges once η > 2/λmax.