Suppose, $f\colon\mathbb{R}^n\to\mathbb{R}$, defined by $f(x)=\lVert x\rVert^2$, the Euclidean norm.
What is $f'(x)$?
I think, if I write $x=(x_1,x_2,\ldots,x_n)$, then
$$ f(x)=\langle x,x\rangle=x_1^2+x_2^2+\ldots+x_n^2 $$ and thus $$ f'(x)=(2x_1,2x_2,\ldots,2x_n)=2x $$
Am I correct?
If $x$ is fixed in $\mathbb R^n$, we have, for $h\in\mathbb R^n$:
\begin{align} f(x+h)-f(x) &= (x+h\mid x+h)-(x\mid x)=\\ &= (x\mid x)+(x\mid h)+(h\mid x)+(h\mid h)-(x\mid x)=\\ &= 2(x\mid h)+(h\mid h) \end{align}
so the differential (or derivative) $\;df(x)\;$ (or $f'(x)$) is the function $\;\mathbb R^n\to\mathbb R\;$ given by
$$df(x)(h) = 2(x\mid h) $$
since that function is linear, and, for $h\neq 0$
$$ \frac{(h\mid h)}{\|h\|} \leq \frac{\|h\|\,\|h\|}{\|h\|}=\|h\| \to 0 \;\;\text{ as }\;\;h\to 0. $$
Note that
$$ df(x)(h)=2(x\mid h)=(2x\mid h)=(\nabla f(x)\mid h) $$
so the gradient of $f$ in $x$ is given by
$$ \nabla f(x) = 2x=(2x_1, 2x_2,\ldots,2x_n).$$