How to verify the gradient of a symbolic function using numerical gradient?

Question

I have a function $f$, which takes as inputs a three arrays and returns an array. I have written a symbolic function $g$ to calculate the gradient of this function and I want to verify that it computes it correctly. For this I would like to use numerical gradient.

Would I be right in doing this way?

• Calculate $f(x,a,b)$

• To calculate $x+\epsilon$ will it be right to add $\epsilon$ to every value of the matrix x. Then calculate $f(x+\epsilon,a,b), \ f(x, a+\epsilon, b), \ f(x,a,b+\epsilon)$

• Find: \begin{align} & z_1 = (f(x+\epsilon,a,b)- f(x,a,b))/\epsilon, \\ & z_2 = (f(x,a+\epsilon,b)- f(x,a,b))/\epsilon, \\ & z_3 = (f(x,a,b+\epsilon)- f(x,a,b))/\epsilon. \end{align}

If each value of the matrices $z_1$, $z_2$, $z_3$ approximates (within some tolerance) to the gradient matrix computed computed by $g$, it is probably correct?

Answer 1

The test you described is incomplete in the sense that you are only considering the directional derivatives of $f$ in the specific directions: adding the same number to all elements of $x$, etc. The full test would be

• Calculate $f(x,a,b)$
• Add $\epsilon$ to one entry of matrix $x$, and calculate the new value of $f$
• Take the difference of two values and divide it by $\epsilon$. Compare the result with the output of $g$
• Repeat for other entries of $x$
• Repeat for $a$ and $b$.