Why are dual spaces called "dual spaces", even when I don't see if they're really "complements" of the original space.
In optimization the primal-dual distinction seems to rely on logical complement. That the dual of the primal is the "complement" of the primal.
Couldn't dual spaces be called e.g. "functional space of" or e.g. "bounded functional space of" space? Assuming that the primal is non-function space or e.g. "only linear functionals".
The (topological) dual $X^*$ of a normed vector space $X$ is the vector space of continuous linear functionals equipped with the norm $$\|f\| = \sup_{\|x\| \le 1} |f(x)|.$$ This leads to the inequality, $|f(x)| \le \|f\| \|x\|$.
Ideally, to call $^*$ a duality relation, we'd want it at least to be an involution, so that $X^{**} = X$ (for some definition of $=$).
We can think of each $x \in X$ as a functional $\hat{x} \in X^{**}$, defined by: $$\hat{x} : X^* \to \Bbb{R} : f \mapsto f(x).$$ It's not hard to see that, for a fixed $x \in X$, this map is linear. Moreover, it's continuous; if we have $\|f\| \le 1$, then $$|\hat{x}(f)| = |f(x)| \le \|f\|\|x\| \le \|x\|,$$ so this proves $\|\hat{x}\| \le \|x\|$. Using the Hahn-Banach Extension Theorem, we can find some $f$ with $\|f\| = 1$ and $f(x) = \|x\|$, which implies that $\|\hat{x}\| = \|x\|$.
So, what does this tell us? It means that this $\hat{}$ operation, mapping $X$ to $X^{**}$, maps any element of $X$ to an element of $X^{**}$. Moreover, this map doesn't change the norm, meaning that it's an isometry, and thus a continuous, injective, and invertible on its range. If it's surjective, this would mean that $X$ and $X^{**}$ are isometrically isomorphic, and so $X = X^{**}$ for a pretty good definition of $=$.
BUT, unfortunately, in general, this $\hat{}$ map is not surjective. The spaces for which it is surjective are known as reflexive spaces (and they are a large and important class of spaces). There are, however, spaces that are not reflexive.
Of course, every finite-dimensional space is reflexive, because $\hat{x}$ maps injectively between two spaces of the same dimension, and hence is surjective. Every Hilbert space is reflexive, as is every $L_p$ space with $1 < p < \infty$. For such spaces, $X$ and $X^{**}$ are isometrically isomorphic; they are structurally the same.
As an example of a non-reflexive space, take $X = c_0$, the space of $0$-convergent real sequences uner the supremum ($\infty$-)norm. Then $X^*$ is isometrically isomorphic to $l^1$, the space of summable sequences under the $1$-norm. The second dual $X^{**}$ is isometrically isomorphic to $l^\infty$, the space of bounded sequences under the supremum/$\infty$-norm again. This space is quite a bit larger than $c_0$!
So, it doesn't perfectly earn its title of "dual", but it does so in enough important situations that we call it a "dual" anyway.