I see in many sources that the biconjugate of a convex proper function is always less than the original function:
$$f(x) \ge f^{**}(x)$$
But, when I try to prove this to myself,I get twisted up like this:
For a convex function we know
$$f(x) + f^*(z) \ge \, <z,x> \forall x,z \in dom(f)$$
and we know that we achieve equality when $z$ is in the subdifferential of the function $f$ at $x$, and thus we have something like this:
$$f(x) = <z,x> -f^*(z)$$
$$\iff$$
$$z\in\partial f(x)$$
Now, when I try to work this out myself I do the following:
$$f(x) = <z,x> - f^*(z) \le \sup_{x}\{<z-x> - f^*(z)\}=f^{**}(x)$$, but this tells me that $f(x) \le f^{**}(x)$. But this is the opposite of what I started with. Which one is correct? thank you.
You've got this slightly backwards. Recall that the definition of $f^*(z)$ is $$ f^*(z) = \sup_x ( \langle x,z \rangle - f(x) ), $$ hence $$ f^*(z) \geq \langle x,z \rangle - f(x), $$ which holds for each $z$, and for each $x$ in the domain of $f$ (it's true for every $z$ if we take $f^*(z)=\infty$ when there is no finite supremum). Phrased this way, it looks like we choose $z$ first, and then inequality holds by the fact that the definition uses $\sup_x$. We can rearrange this inequality to $$ f(x) \geq \langle x,z \rangle - f^*(z), $$ which can be interpreted in a different way: pick any $x$ in the domain, and then for any $z$ the right-hand side is smaller. This also means that $$ f(x) \geq \sup_z ( \langle x,z \rangle - f^*(z) ), $$ since $f(x)$ is an upper bound for $\langle x,z \rangle - f^*(z)$, while the supremum over $z$ is the least of its upper bounds. But the right-hand side of this equation is the definition of $(f^*)^*(x) = f^{**}(x)$, so $ f(x) \geq f^{**}(x) $.
Essentially, your problem is that equality holds only for $z$ in the subdifferential, while you need to take the supremum over the whole thing, where the inequality $ f(x) \geq \langle x,z \rangle - f^*(z) $ holds. So the subdifferential is actually the top extreme case, and the supremum over $z$ cannot be larger. (Indeed, if the subdifferential is empty, as when $f$ is nonconvex, it can be smaller, so strict inequality can occur.)