Say I am solving an equation numerically -- the derivatives in the equation I find by a finite difference scheme with an accuracy of the grid spacing $h$.
Does this imply that the final solution I find to the equation has an error at least on the order of the the square of the grid spacing?
On
This is not an answer but it is too long for a comment.
In fact, in the old time (a time I knew), some people used to compute the derivative only once at the starting point of Newton iterations and never updated it. Which means that the iterative scheme was just $$x_{n+1}=x_n-\frac{f(x_n)}{f'(x_0)}$$ For sure this slows down the convergence but it works.
Consider $$f(x)=(x+2)\log(x+2)-3x$$ and let us use $x_0=1$. The iterates will be $$x_1= 1.32820$$ $$x_2=1.34743$$ $$x_3=1.34971$$ $$x_4= 1.34998$$ $$x_5=1.35002$$ which is the solution for six significant figures.
For the normal scheme $$x_{n+1}=x_n-\frac{f(x_n)}{f'(x_n)}$$ the iterates will be $$x_1= 1.32820$$ $$x_2=1.34993$$ $$x_3=1.35002$$
So, basically, the derivative can be very wrong as long as the sign is correct (ssuming that the method converges). In Lutzl's comment, the key word is contractive.
No, the derivatives need not be exact. If you have somewhat correct derivatives that, among others, can be obtained from divided differences, then the resulting approximate method like the simplified Newton method is contractive, you get (at least) geometric convergence to the solution and the error in the numerical limit is the error of the step computation, i.e., function evaluation scaled by division by the derivative.
If you can make the derivative (dynamically) as "exact" as the function value is "large", i.e., the computed derivative is $J_k=f'(x_k)+O(f(x_k))$, then the Newton method will still converge quadratically, and a measure for the distance to the solution is the length of the last Newton step. Which is in the numerical limit the same measure as above.