Let $X\geq 0$ be a real random variable and $h:\mathbb{R} \rightarrow \mathbb{R}$ a monotonously growing, continuously differentiable function with $h(0)=0$.
Show:
$E[h(X)] = \int_0^{\infty} h'(t)P[X>t]dt$
With the given hint I got
$E[h(X)] = \int_0^{\infty} \int_{0}^x h'(t)dt dF(x)$
with Fubinis theorem I can change the order of integration to
$\int_0^{\infty} \left( \int_{t}^{\infty} dF(x)\right)h'(t)dt$
which gives me
$\int_0^{\infty} P[X>t]h'(t)dt$
How would I go about finding the first two moments in integral form like above?
I got this far:
first moment
$E[X] = \int_0^{\infty} x F(x) dx = \int_0^{\infty}\int_0^{x}dx F(x) dt = \int_0^{\infty}\int_t^{\infty} F(x) dx dt = \int_0^{\infty}P[X>t]dt$
second moment
$E[X^2] = \int_0^{\infty} x^2 F(x) dx = \int_0^{\infty}\int_0^{x}2x F(x) dx dt = 2\int_0^{\infty}\int_t^{\infty} x F(x) dx dt$
But now what? $\int_t^{\infty}x dx$ doesn't converge.
The only explicit question seems to ask to explain where the following computation is faulty:
Already the first equal sign is wrong. If $X$ has a density $f$ then $$ E(X^2)=\int_0^\infty x^2f(x)\mathrm dx, $$ but for every random variable $X$, if $F$ is the CDF of $X$ then $F(x)\to1$ when $x\to\infty$ hence $$ \int_0^\infty x^2F(x)\mathrm dx=+\infty. $$ Another mistake lurks in the computation of the first moment, since you write:
Here again, if $F$ is the CDF of $X$ the first equal sign is wrong, for the same reasons. And the last equal sign is wrong as well, since, for every $t$, $$ \int_t^{\infty} F(x) \mathrm dx $$ is equal to $+\infty$, not to $P(X\gt t)$.
What you are asked to do is simply to plug in $h(x)=x$ and $h(x)=x^2$ in the formula you first proved, namely, $$ E(h(X)) = \int_0^{\infty} h'(t)P(X>t)\mathrm dt, $$ leading to $$ E(X) = \int_0^{\infty}P(X>t)\mathrm dt, \qquad E(X^2) = \int_0^{\infty} 2tP(X>t)\mathrm dt. $$