I am stuck with the following exercise of my textbook:
Cotlar's Identity: For $f\in \mathcal{S}(\mathbb{R})$, we have $(Hf)^2 = f^2 + 2H(f\cdot Hf)$, where $H$ is the Hilbert transform.
The exercise is accompanied by the following hint:
Since $\widehat{Hf}(\xi) = -i\:\text{sgn}(\xi)\hat{f}(\xi)$. Directly compute $$\widehat{Hf}*\widehat{Hf} =\hat{f}*\hat{f} - 2i \: \text{sgn}(\xi)(\hat{f}*\widehat{Hf})$$
But I don't even understand why it's enough to prove the above expression involving convolutions.
The reason it's enough to prove the statement about convolution is that Fourier transform sends products to convolutions. So, the transform of $f^2$ is $\hat f*\hat f$, the transform of $f\, Hf$ is $\hat f*\widehat{Hf}$, and so on. If two expressions have the same Fourier transform, they are equal.
To prove the claimed identity, let's write it in symmetric form $$ \widehat{Hf}*\widehat{Hf} =\hat{f}*\hat{f} - i \, \operatorname{sgn}(\xi)(\hat{f}*\widehat{Hf}) - i \, \operatorname{sgn}(\xi)(\widehat{Hf}*\hat f) \tag1 $$ By the definition of convolution, $$ (\hat{f}*\hat{f})(\xi) = \int_{\mathbb{R}} \hat f(\xi-t)\hat f(t)\,dt $$ Similarly, $$ (\widehat{Hf}*\widehat{Hf})(\xi) = \int_{\mathbb{R}} \widehat{Hf}(\xi-t)\widehat{Hf}(t)\,dt = - \int_{\mathbb{R}} \operatorname{sgn}(\xi-t) \operatorname{sgn}(t) \,\hat f(\xi-t)\hat f(t)\,dt $$ and also $$ i\operatorname{sgn} (\xi) (\hat f*\widehat{Hf})(\xi) = \int_{\mathbb{R}} \operatorname{sgn} (\xi) \operatorname{sgn}(t) \,\hat f(\xi-t)\hat f(t)\,dt $$ $$ i\operatorname{sgn} (\xi) (\widehat{Hf}*\hat f)(\xi) = \int_{\mathbb{R}} \operatorname{sgn} (\xi) \operatorname{sgn}(\xi-t) \,\hat f(\xi-t)\hat f(t)\,dt $$ Comparing these four integrals, we see that (1) boils down to the algebraic identity $$ -\operatorname{sgn}(\xi-t) \operatorname{sgn}(t) = 1 - \operatorname{sgn} (\xi) \operatorname{sgn}(t) - \operatorname{sgn} (\xi) \operatorname{sgn}(\xi-t) \tag2 $$ To verify this, rearrange it as $$ (\operatorname{sgn} (\xi)-\operatorname{sgn}(t)) \operatorname{sgn}(\xi-t) = 1 - \operatorname{sgn} (\xi) \operatorname{sgn}(t) \tag3 $$ If $\xi$ and $t$ have the same sign, both sides of (3) are equal to $0$. If they have opposite signs, both sides of (3) are equal to $2$.
Aside: a much shorter proof comes from complex analysis, according to which $f$ and $Hf$ are the real and imaginary parts of the boundary values of a holomorphic function $u+iv$ in the upper half-plane. Since $(u+iv)^2 = (u^2-v^2) + 2iuv$ is also holomorphic, the Hilbert transform of $f^2-(Hf)^2$ is $2f Hf$. Hence, $H(2fHf) = (Hf)^2 - f^2$.