Reference: "Pattern Recognition and Machine Learning" by Christopher Bishop
When choose squared loss in the Loss functions for regression. The expected loss could be written as $$ E[L] = \int_{-\infty}^\infty \int_{-\infty}^{\infty} [ y(x) - t]^2 p(x,t) \,dxdt$$
In order to choose $y(x)$ which minimum the loss, use calculus of variations and get $$\frac{dE[L]}{dy(x)} = 2 \int_{-\infty}^\infty [y(x) - t] p(x,t) \, dt = 0$$
Solving for y(x), $$ y(x) = \frac{\int_{-\infty}^\infty t p(x,t) \, dt }{p(x)} = E_t[t|x]$$
My questions
I don't quite understand how they calculate the following 2 formulas as it lack details. My thoughts as below (might wrong!)
Particular how they solve 1st derivate for double integral and how they solve $y(x)$ after take 1st derivative?
\begin{align} \frac{dE[L]}{dy(x)} &= \frac{d}{dy(x)} \int_{t = -\infty}^{t = \infty} \int_{x = -\infty}^{x=\infty} [y(x) - t]^2 p(x,t) \, dxdt\\ &= \int_{t = -\infty}^{t = \infty} \left[\frac{d}{dy(x)} \int_{x = -\infty}^{x=\infty} [y(x) - t]^2 p(x,t) \, dx\right]dt \end{align}
Let
\begin{align} H(x) &= \frac{d}{dy(x)} \int_{x = -\infty}^{x=\infty} [y(x) - t]^2 p(x,t) \,dx\\ & = \frac{d}{dy(x)} [y(x) - t]^2 * p(x,t) + [y(x) - t]^2 * \frac{d}{dy(x)} [p(x,t)]\\ & = 2[y(x) - t] * p(x,t) + 0\\ & = 2[y(x) - t] * p(x,t) \end{align}
So
$$ \frac{dE[L]}{dy(x)} = 2 \int_{-\infty}^\infty [y(x) - t] p(x,t) \, dt = 0 $$
Is that means that when take a 1st derivative to a double integral, we only need to eliminate one integral (i.e., only the inner integral participate in calculation? And the outer integral keep unchanged?)
I know that a derivate and an integral is inverse relationship. So that when we take the derivate to an integral, we just write down what inside the integral. Here, they take derivate respect to $y(x)$, and the inner integral is respect to x. Why the x inside the $p(x,t)$ does not change to $y(x)$ after the calculation? i.e., How can we decide in this case which parameter should change respect to the derivation?
I also get some idea to prove y(x), but I did not get the answer either...
Set that 1st derivative to 0 $$2 \int_{-\infty}^\infty [y(x) - t] p(x,t)dt = 0$$ $$ \int_{-\infty}^\infty [y(x) - t] p(x,t)dt = 0$$ $$ \int_{-\infty}^\infty y(x)p(x,t) dt - \int_{-\infty}^\infty t p(x,t)dt = 0$$ $$ y(x)\int_{-\infty}^\infty p(x,t) dt = \int_{-\infty}^\infty t p(x,t)dt$$ $$ y(x) = \frac{\int_{-\infty}^\infty t p(x,t)dt} {\int_{-\infty}^\infty p(x,t) dt }$$
$$... ... $$