I have a function of the type (for simplicity I use a similar and more straightforward function):
$ f(x) = \Vert Ax - b \Vert, \quad A\in R^{m\times n}, x \in R^n, \text{ and } b \in R^m$.
I would like to find the zero of this function, i.e., finding x such that $f(x) = 0$. For this aim, I would like to use the Newton method, whose iterative formula is:
$x^{t+1} = x^t - \frac{f(x)}{f'(x)}$
where $f'$ is the derivative of $f$, i.e., $f'(x) = A^TAx - A^Tb$. As is seen, f(x) is unidimensional, while $f'(x)$ is a vector.
Now my question is, how am I supposed to use the Newton method for such a function?
You can not use Newton Method to solve $f(x)=0$, a.k.a. $Ax=b$. If $m >> n$ then generically there is no such solution. More likely you want to use Newton's Method to find the minimum of this function, a.k.a. the least squares solution. In that case you can use Newton Method on the gradient of $f$, $\nabla f: \mathbb{R}^n \to \mathbb{R}^n$, in that case Newton's step is: $$x_{k+1} = x_{k}- (\nabla^2 f(x_{k}))^{-1}\nabla f(x_{k}) $$ where $\nabla^2 f$ is the Hessian matrix. You can also solve by a direct method solving the normal equations $A^T A\hat{x} = A^T b$.