Suppose we have a Gaussian white noise $f(t)$. Then the covariance $\langle f(t)f(t')\rangle = c\times\delta(t - t')$ for some $c > 0$.
However, since $f(t)$ is a Gaussian random variable (with variance $c$, say). Then $f(t)^2$ should be a chi-squared variable with one degree of freedom, and should have mean $c$.
As the noise at different times are independent of each other, we should rather have
$\langle f(t)f(t')\rangle = \begin{cases} c&\text{if }\, t = t^\prime\\ 0&\text{if }\, t \neq t^\prime\\ \end{cases} =c\times\delta_{(t-t^\prime),0} $ (which is a Kronecker delta, not a Dirac delta).
If we think physically, suppose we have many copies noise generators and we measure $f(t)^2$ at the same time in all of them and take the average. Clearly, it will be finite and approximately equal to $c$.
Why do we have the Dirac delta function in the correlation instead of the Kronecker delta function? Am I missing something?
Please note that this is not a duplicate of this because I am asking about why the covariance (or correlation) of the noise is not finite. Sure, the Fourier transform of the spectral density implies it is a Dirac delta function, but I don't understand how it happens physically.
$<f(t)^2>$ is infinite! Its frequency spectrum is flat. The auto-correlation and the frequency spectrum are Fourier transforms of each other.