Define the function $f\colon \mathbb{R}^n \to \mathbb{R}$ by $f(x) = |x|^2 - 2\langle a , x \rangle$ where $a \in \mathbb{R}^n$ is non-zero. We have:
$$f(x + h) - f(x) = |x+h|^2 - 2\langle a, x + h\rangle - |x|^2 + 2\langle a, x\rangle \\ = |h|^2 - 2\langle x, h\rangle - 2\langle a, h\rangle$$
And thus we have $Df(x)(h) = -2\langle x, h\rangle -2\langle a, h\rangle$. The only critical point is $x = -a$, and given that $D^2f(x)(h, h) = -2|h|^2 < 0$, this critical point is a maxima.
But now let's look at the particular case where $n = 2$ and $a = (1, 0)$. We have $f(x, y) = x^2 + y^2 - 2x$. It's derivative is given by:
$$f'(x, y) = \begin{pmatrix}2x - 2 & 2y \end{pmatrix}$$
This has a critical point at $x = 1$ and $y = 0$. Also, we have:
$$f''(x, y) = \begin{pmatrix}2 & 0 \\ 0 & 2 \end{pmatrix}$$
This matrix is positive definite, so the critical point is actually a minima. What is wrong with the general case?
You made a sign mistake. Indeed, $x \mapsto |x|^2$ is a quadratic form, hence its derivative is $h \mapsto 2\langle x,h \rangle$.
The second term is linear, and coincides with its derivative. Hence $$ Df(x) \colon h \mapsto 2\langle x,h \rangle - 2\langle a,h \rangle. $$