If $f$ is closed and convex then $f = f^{**}$
Let $f$ be closed and convex. Then $f^* = \sup_x(y^Tx - f(x))$.
Since $$\{h(x) = ax + b | h(x) \le f(x) \text{ for all $x$ }\} = \{h(x) = y^Tx + c | y \in \text{dom}(f^*), c \le - f^*(y)\}$$ we have $$f(x) = \sup\{g(x) | g \text{ affine}, g(z) \le f(z) \text{ for all } z\}$$
And $$f= f^{**} $$
How is $f = f^{**}$ here? Why is $f^{**} = \sup\{g(x) | g \text{ affine}, g(z) \le f(z) \text{ for all } z\}$ If $f^*(y) \equiv \sup_x(y^Tx - f(x))$ then shouldn't $(f^{*}(y))^*(u) = f^{**}(u) = \sup_y (u^Ty - f^*(y))$?
What exactly am I missing here?
For the affine minorant part, notice that we may write
$$ f^{**}(x) = \sup_{y} x^\top y - f^*(y) = \sup_{y, \beta} \left\{ x^\top y - \beta \ \middle|\ \beta \geq f^*(y) \right\} \\ = \sup_{y, \beta} \left\{ x^\top y - \beta \ \middle|\ \beta \geq \sup_{z} z^\top y - f(z) \right\} = \sup_{y, \beta} \left\{ x^\top y - \beta \ \middle|\ z^\top y - \beta \leq f(z), \; \forall z \right\}. $$
The first equality between suprema is easy to verify, and the supremum in the final expression is taken over all functions $g_y(z) := z^\top y - \beta$ such that $g_y(z) \leq f(z), \; \forall z$. However, any function $g_y(z)$ that satisfies this is an affine minorant of $f$, so you are effectively taking the supremum over all affine minorants.