Suppose we have a simultaneous system of 2 black-box non-linear equations in 2 variables $\{a,b\}$:
$$ F_1(a,b) = 0$$ $$ F_2(a,b) = 0$$
Newton's iteration method for solving this system of equations goes as follows:
- Starting with any initial guess, eg. $a_0=0$, $b_0=0$, we calculate
$$ \begin{bmatrix} \frac{dF_1}{da} & \frac{dF_1}{db} \\ \frac{dF_2}{da} & \frac{dF_2}{db} \end{bmatrix} \begin{bmatrix} \sigma_1 \\ \sigma_2 \end{bmatrix} = - \begin{bmatrix} F_1(a_0, b_0) \\ F_2(a_0, b_0) \end{bmatrix}$$
- This $\sigma$ correction amounts can be found by inverting the left-hand-side Jacobian matrix, and adding the amounts to the original guess $a_0, b_0$:
$$ \begin{bmatrix} a_1 \\ b_1 \end{bmatrix} = \begin{bmatrix} a_0 \\ b_0 \end{bmatrix} + \begin{bmatrix} \sigma_1 \\ \sigma_2 \end{bmatrix} $$
and the iteration can be continued to find $a_2, b_2$ etc, converging incrementally upon the solution to the original system of equations.
However, I have heard it mentioned that this method fails when $F_1$ and $F_2$ are non-linear.
I not sure the exact reason why this is true for non-linear equations. In the above sequence of steps, where precisely do the non-linear $F_1$, $F_2$ break the convergence to the correct solution?
You do not need Newton's method to solve linear systems. Meaning that the method is constructed for non-linear systems. It would not be a famous, named method if it was a failure for its purpose.
For non-linear systems it is highly unlikely, but not entirely impossible, that you find a solution with the first step. So more commonly you have to iterate the Newton step. In fortunate cases you get eventually convergence, the function values and step sizes reducing to zero. In some neighborhood of the solution this reduction will happen from the first step on, in other cases the sequence will jump around seemingly randomly. In unfortunate cases, which will happen more often the higher the dimension of the system, there will be no convergence.
It depends on context what "the method fails" precisely means.