From Wikipedia:https://en.wikipedia.org/wiki/Convex_conjugate
Let ${\displaystyle X}$ be a real topological vector space, and let ${\displaystyle X^{*}}$ be the dual space to ${\displaystyle X}$. Denote the dual pairing by ${\displaystyle \langle \cdot ,\cdot \rangle :X^{*}\times X\to \mathbb {R} .}$ For a functional ${\displaystyle f:X\to \mathbb {R} \cup \{+\infty \}}$ taking values on the extended real number line, the convex conjugate ${\displaystyle f^{\star }:X^{*}\to \mathbb {R} \cup \{+\infty \}}$ is defined in terms of the supremum by ${\displaystyle f^{\star }\left(x^{*}\right):=\sup \left\{\left.\left\langle x^{*},x\right\rangle -f\left(x\right)\right|x\in X\right\}} $
I am still bothered by the idea of a "dual space", because from the point of view of convex analysis, it seems that the idea of a dual space is tied in with the idea of a convex conjugate, whereas the dual space seems to be a much more general concept (i.e. the space of linear functional). For instance, we know that $\mathbb{R}^n$ is a Hilbert space, hence $(\mathbb{R}^n, \|\cdot\|_2)$ is the dual of itself. However, convex analysis is giving me some un-intuitive results.
For instance, in example 5.1 of http://www.doc.ic.ac.uk/~ahanda/lfreport.pdf
We are given $f(y) = \|y\|$, presumably the $2$-norm and the function is defined over all of $\mathbb{R}^n$, we find the conjugate to be $f^*(z) = 0, \|z\|\leq 1$.
Now if you had told me that the dual space to $X = (\mathbb{R}^n, \|\cdot\|_2)$ is the set $X^* = \{z\in \mathbb{R}^n | \|z\|_2\leq 1\}$, I think I will have a difficulty understanding you, but isn't this what the example above is showing?
Can someone please help me understand the concept of a dual space?
In your first example,
$$ f^*(z) = \cases{0 & if $\|z\| \le 1$\cr +\infty & otherwise}$$
That doesn't mean $\{z: \|z\| \le 1\} = X^*$. It is just the subset of $X^*$ on which this particular conjugate functional is finite.