According to my computations:
The function which minimizes $$\int_\Omega \|\operatorname{curl} f\|^2\,dx$$ should satisfy $$\operatorname{curl}(\operatorname{curl}f) = 0$$ everywhere on $\Omega$, provided $\operatorname{curl} f = 0$ on $\partial \Omega$.
I followed the same kind of computation that the one demonstrating the the argmin of $\int_\Omega \|\nabla f\|^2~dx$ should satisfy $\Delta f = 0$. However, I am not sure whether my computations are right... May anyone check that please ?
1) We first start with a functional $$G(f) = \int_\Omega \|\operatorname{curl} f\|^2\,dx.$$
2) We compute $$V(f,h) = \lim_{\epsilon\rightarrow 0} \frac{G(f+\epsilon h)-G(f)}{\epsilon} = 2\int_\Omega \operatorname{curl} f\cdot\operatorname{curl} h\,dx.$$
3) We use the identity : $$\operatorname{div}(A\times B) = -A\cdot\operatorname{curl} B + B\cdot\operatorname{curl} A,$$ with $A=\operatorname{curl} f$ and $B=h$.
4) We obtain $$\int_\Omega \operatorname{curl} f\cdot \operatorname{curl} h\:dx = -\int_\Omega \operatorname{div}(\operatorname{curl} f\times h)\,dx + \int_\Omega h\cdot \operatorname{curl}(\operatorname{curl}f)\,dx.$$
5) We use the divergence theorem to obtain : $$\int_\Omega \operatorname{curl} f\cdot \operatorname{curl} h\,dx = -\int_{\partial\Omega} \operatorname{curl} f\times h\,ds + \int_\Omega h\cdot\operatorname{curl}(\operatorname{curl}f)\,dx$$
6) We assumed $\operatorname{curl} f = 0$ on $\partial\Omega$, so the first term is zero.
7) $V(f,h)$ should equal zero for all $h$ for the function to be minimized, so $$\int_\Omega h\cdot\operatorname{curl}(\operatorname{curl}f) dx = 0\quad\forall h,$$ which implies $\operatorname{curl}(\operatorname{curl}f) = 0$ locally.
I guess this reasonning may be wrong in several places..... or is it right?? Thanks!
(This is supposed to be a shorter comment but I find myself powerless).
I haven't checked you reasoning, but I wanted to offer a simple (standard) proof. Some authors call this Dirichlet's principle.
Define $E(f) := \int_{\Omega} \| \nabla f \|^2 dx$. It's clear that $E(f) \geq 0$.
Assume $f, g$ are equal on $\partial \Omega$ Let's see that, if $f$ is harmonic then $E(f) \leq E(g)$.
For that, let $u := f - g$. Now calculate $E(g) = E(f - u)$ using Green's first identity (here you'll use that $u = 0$ on $\partial \Omega$ and that $\Delta f = 0$).
You'll get immediately that $E(g) = E(f) + E(u)$ , and using that $E(u) \geq 0$ that $E(f) \leq E(g)$.