# Newton Method

Using second-derivative information to jump toward the optimum, converging quadratically near the solution instead of linearly.

**Rests on:** Automatic Differentiation, Gradient Descent

*Mathematical Foundations · Optimization*


## Why learn this

You have likely used a method that moves toward an answer by taking small steps in the direction where the error decreases. This process is safe but slow. The Newton Method is a more powerful alternative that reaches the best answer much faster, often in just a few steps instead of thousands. This matters because it turns calculations that would take hours into tasks that finish in seconds, provided you start close to the solution. It is the core engine behind how computers tune the settings in complex models, allowing them to 'learn' from data efficiently without wading through every possibility one by one.


## The intuition

Imagine you are trying to find the lowest point in a valley while blindfolded. Gradient descent works by feeling the slope of the ground under your feet and taking a step downhill. If the valley is steep, you take a big step; if it flattens out near the bottom, you must take tiny steps to avoid walking past the lowest point. This process is safe but slow.

The Newton Method is different because it does not just feel the slope; it estimates the overall shape of the valley itself—how curved the sides are. By sensing this curvature, it can predict exactly where the bottom is and jump almost directly there.

Consider a valley shaped like a perfect, smooth bowl (a mathematically simple 'convex' problem). If you drop into any part of this bowl, the Newton Method calculates the precise depth and angle of the curve and jumps immediately to the very center. It lands on the optimum in one step. If the valley is irregular with bumps and dips (a 'nonconvex' problem), it cannot jump all the way there at once, but each jump brings it significantly closer than walking would. It uses the local shape to make an educated guess about the global minimum.


## An analogy

Think of trying to hit a bullseye on a dartboard with your eyes closed. Gradient descent is like throwing darts at random and then stepping toward the nearest previous throw, inching closer slowly. The Newton Method is like having a thermal camera that shows you not just where you are, but exactly how the target's heat gradient flows to the center. You look at the flow pattern, calculate the exact vector pointing to the hottest spot (the bullseye), and step directly there. If your thermal reading is perfect, you hit the bullseye on the first try. However, if the heat source moves or your camera glitches, you might aim at a fake hot spot that is not the true center. This highlights the trade-off: high speed with higher risk of misjudgment from bad shape estimates.


## Definition

The Newton Method finds the lowest point of a function by measuring the slope to find the direction and measuring how much that slope changes (the curvature) to determine the exact size of the step, which allows it to reach the answer very quickly if you start close enough.


## How it connects

This sits beside Gradient Descent, which uses only the slope to move forward. The Newton Method adds the dimension of curvature to navigate more directly. It relates to Convex Optimization, a type of problem where the landscape has only one single bottom (shaped like a perfect bowl), making the Newton Method highly effective because the estimated shape never contradicts the true bottom. In contrast, for Nonconvex Optimization, where the landscape has multiple peaks and valleys, the method is less reliable without modification. Practitioners often use Quasi-Newton methods, which are algorithms that estimate the curvature using only slope information to get similar speed benefits without the heavy computational cost of calculating the full shape matrix directly.


## The mathematics

Newton’s Method finds the bottom of a curve by using two pieces of information at your current position: how steep the hill is, and how curved the hill is.

Imagine you are standing on a smooth, bowl-shaped hill. You want to find the lowest point. You cannot see the bottom, but you can feel two things under your feet:
1. The slope: This tells you which way is down. If the ground slopes up to the right, the bottom is to the left.
2. The curvature: This tells you how sharply the ground bends. A sharp bend means the bottom is close by. A gentle bend means the bottom is far away.

Newton’s Method combines these two feelings into one step. It assumes that near your current position, the hill looks exactly like a perfect parabola (a simple U-shape). If you know the slope and the curvature at your current spot, you can calculate exactly where the bottom of that local U-shape is.

The formula for this step is:

New Guess = Old Guess minus (Slope divided by Curvature)

Let us break down why we subtract. 
- If the slope is positive, the ground is going up to the right. The bottom must be to the left. Subtracting a positive number moves you left.
- If the slope is negative, the ground is going down to the right. The bottom is still to the right (because you are on the left side of the bowl). Subtracting a negative number is the same as adding, so you move right.
- The curvature acts as a scaler. If the curvature is large (sharp bend), the division makes the step size small. This prevents you from overshooting the bottom. If the curvature is small (flat bend), the division makes the step size large, helping you cover ground quickly.

This works beautifully for simple curves with one input variable. For functions with many inputs (like a landscape with hills and valleys in every direction), the "slope" becomes a list of slopes called the gradient, and the "curvature" becomes a grid of curvatures called the Hessian matrix. The logic remains the same: use the local shape to predict the bottom.

The method is famous because it converges very fast. Once you are close to the bottom, each step roughly doubles the number of correct digits in your answer. However, it requires calculating the curvature at every step, which can be computationally expensive for complex problems.


## A worked example

We want to find the value of x that minimizes the function f(x) = (x - 3)^2 + 5.

This function describes a parabola. The lowest point is at x = 3, where the value is 5. We will use Newton’s Method to find this minimum starting from a guess of x = 0.

First, we need the formulas for slope and curvature for this specific function.
The function is f(x) = (x - 3)^2 + 5.
To find the slope (first derivative), we apply the power rule. The derivative of (x-3)^2 is 2 times (x-3). The derivative of the constant 5 is 0. So, Slope = 2 * (x - 3).
To find the curvature (second derivative), we take the derivative of the slope. The derivative of 2*(x-3) is just 2. So, Curvature = 2.

Step 1: Evaluate at the current guess x_old = 0.
Calculate the slope: Slope = 2 * (0 - 3) = 2 * (-3) = -6.
Calculate the curvature: Curvature = 2.
The slope is negative, meaning the ground slopes down to the right. The bottom is to the right of our current position.

Step 2: Calculate the new guess.
Formula: x_new = x_old - (Slope / Curvature).
Substitute the values: x_new = 0 - (-6 / 2).
Perform the division inside the parentheses first: -6 divided by 2 is -3.
Now perform the subtraction: x_new = 0 - (-3).
Subtracting a negative is adding: x_new = 0 + 3 = 3.
Our new guess is 3.

Step 3: Check if we are at the minimum.
We evaluate the slope at our new guess x = 3.
Slope = 2 * (3 - 3) = 2 * 0 = 0.
The slope is zero. This means the ground is flat. We are at the bottom of the bowl.

Step 4: Final update.
Formula: x_new = x_old - (Slope / Curvature).
Substitute values: x_new = 3 - (0 / 2).
0 divided by 2 is 0.
x_new = 3 - 0 = 3.
The guess does not change. We have converged to the exact minimum.

Result: The Newton Method found the exact minimum x = 3 in one step. This happened because our starting function was a perfect parabola, and Newton’s Method assumes the local shape is a parabola. Since the assumption matched reality perfectly, the prediction was exact.


## From scratch, in Python

```python
def newton_method(f_prime, f_double_prime, x_start, tolerance=0.001):
    """Find the x that minimizes a function, given its slope and curvature."""
    x = x_start
    for i in range(100):
        slope = f_prime(x)
        curvature = f_double_prime(x)
        if curvature == 0:
            break  # Can't divide by zero
        step = slope / curvature
        x_new = x - step
        print(f"Iteration {i+1}: x = {x_new:.4f}, f'(x) = {slope:.4f}, f''(x) = {curvature:.4f}")
        if abs(x_new - x) < tolerance:
            break
        x = x_new
    return x

# Example usage: minimize f(x) = (x-3)^2 + 5
def slope(x):
    return 2 * (x - 3)  # derivative: 2(x-3)

def curvature(x):
    return 2  # second derivative is constant 2

minimum = newton_method(slope, curvature, 0)
print(f"Minimum found at x = {minimum}")
```


## The same thing, with a framework

```python
import numpy as np

# Define the function and its derivatives
def f(x):
    return (x - 3)**2 + 5

def slope(x):
    return 2 * (x - 3)

def curvature(x):
    return 2

x = 0.0
for i in range(10):
    x_new = x - slope(x) / curvature(x)
    print(f"Iteration {i+1}: x = {x_new:.6f}")
    if abs(x_new - x) < 1e-8:
        break
    x = x_new

print(f"Final x = {x:.6f}")
```


## What the code does

The code defines a function `newton_method` that takes the slope function, curvature function, starting point, and a tolerance. In each iteration, it computes the slope and curvature at the current guess, computes the step as slope divided by curvature, and updates the guess. It stops when the change is smaller than the tolerance. The example uses f(x) = (x-3)^2+5, where the slope is 2(x-3) and the curvature is 2. The loop starts at x=0, prints each iteration, and converges to x=3. The framework version (NumPy) does the same arithmetic but without the wrapper, showing the raw update rule. Note that in real applications, you wouldn't code this by hand — you'd use a library like SciPy's `scipy.optimize.newton`, but seeing the arithmetic makes the concept clear.


## Common misconceptions

**Newton's method always finds the global minimum** — Since Newton's method is so fast, surely it always lands on the true minimum, not some local ditch.

The method only guarantees convergence to a stationary point (where the slope is zero), but that could be a local minimum, a local maximum, or even a saddle point. Whether it finds the global minimum depends on the function's shape and your starting point. In nonconvex functions, Newton's method can happily jump into a shallow dip and stop. In fact: Newton's method is fast, but it is still a local optimizer. It finds the nearest stationary point, which may not be the global minimum. To find a global minimum, you need multiple starts or a global search strategy.

**Newton's method requires second derivative to be positive always** — The formula looks like division by curvature, so if curvature is negative, it must blow up.

A negative curvature means the function is concave at that point, so Newton's update will go in the opposite direction — effectively maximizing rather than minimizing. It doesn't blow up; it simply head toward a maximum. The formula is valid for any nonzero curvature, but the direction changes. In fact: The Newton step works for any sign of curvature. Positive curvature means you're heading to a minimum, negative curvature means a maximum. Division by zero is the only real problem, which is rare.

**Newton's method is the same as gradient descent with a different step size** — It's just gradient descent but you pick a smarter step size each time.

While both update the current point, Newton's method uses second-order information to determine both the direction and the step size. Gradient descent always moves in the direction of steepest descent (negative gradient), while Newton's method adjusts the direction based on curvature — it can even move diagonally on a trade-off between variables. In fact: Newton's method uses curvature to transform the gradient direction into a more direct route to the optimum, not just a different step size. The difference is qualitative, not just quantitative.


## In short

Newton's method is a fast optimization algorithm that uses both slope and curvature to jump toward a minimum or maximum. It converges quadratically near the optimum, meaning each step roughly doubles the number of accurate digits. But it requires computing the Hessian (curvature), which is expensive for large models. That's why deep learning mostly uses quasi-Newton methods like Adam. Remember: gradient descent walks, Newton's method leaps, and the leap is only safe when the terrain is well understood.


## Where this is used

Where does the Newton Method actually appear in modern AI? The honest answer: rarely by itself, but its ideas are everywhere.

**Large Language Models (LLMs):** Training GPT-class models uses millions of parameters, and computing the full Hessian (the curvature table) is impossible — it would be a matrix with trillions of entries. So LLMs use gradient descent variants like Adam, which are quasi-Newton methods: they approximate the curvature using only the gradient history, giving some of Newton's speed without the cost. When you hear about "second-order optimizers" in research papers, they're trying to bring Newton's ideas into LLM training in a tractable way.

**Small Models on Devices (Edge AI):** When a model is small enough, say running on a phone, the Hessian might be computable. Some on-device training methods use Newton-style updates for fine-tuning with very few samples. The benefit: with only a handful of examples, Newton's method can adapt the weights much faster than gradient descent, which is crucial when you can't do many iterations on a battery.

**Retrieval-Augmented Generation (RAG):** RAG systems embed text and retrieve similar documents. The embedding vectors are optimized using loss functions, and sometimes the final tuning of the embedding model uses Newton-like techniques. More generally, the retrieval step itself is a nearest-neighbour search, not optimization, but the underlying embedding training often uses Adam — again a Newton-inspired method.

**Agentic Systems:** Agents that plan and call tools often use RL or search. The planning step may involve optimizing a policy, and again, second-order methods can make each learning step more efficient, especially in low-sample settings. You'll see Newton's method in continuous control problems (e.g., simulating a robot) where the dynamics are smooth.

**Everyday Applied ML:** In traditional statistics and linear regression, Newton's method (or its relative, iteratively reweighted least squares) is used for logistic regression and generalized linear models. Libraries like scikit-learn use `liblinear` which employs a Newton method under the hood for some problems. When you fit a logistic regression on a dataset with millions of rows, Newton's method can converge in a handful of steps where gradient descent would take thousands.

**What a developer actually writes:** You'd call `sklearn.linear_model.LogisticRegression(solver='newton-cg')` to use Newton's method. Underneath, the library computes the Hessian (or an approximation) and does the update. The developer doesn't write the math — they choose a solver flag. But understanding Newton's method tells you why some solvers are faster, and why they might fail on huge datasets (because the Hessian is too big).

**What breaks when done badly:** If the Hessian is ill-conditioned (curvature is very different in different directions), Newton's method can overshoot wildly. If the function is nonconvex, it can converge to a local minimum or a saddle point. That's why deep learning doesn't use it — the loss landscape is a nightmare, and the Hessian would be misleading. That's also why quasi-Newton methods like Adam are preferred: they're more robust.

So, while Newton's method itself is rare in cutting-edge AI training, its DNA is everywhere — in optimizers, in classic regression, and in any setting where you can afford to compute curvature. Understanding it gives you insight into why modern optimizers do what they do.


---

Read this on the web: https://basavapurushottam.com/learn-ai/newton-method
