Express $\mathbb{E}(X|Y>a)$ using conditional density and $\mathbb{E}(X|Y=y)$.

Question

Let $X$ and $Y$ be (Edit thx to the comment: integrable) two random variables with joint density $f_{X,Y}(x,y)$. Let $a\in\mathbb{R}$ be such that $\mathbb{P}(Y>a)>0$. Then, We know that $$\mathbb{E}(X|Y>a)=\dfrac{\int_{Y>a}Xd\mathbb{P}}{\mathbb{P}(Y>a)}=\dfrac{\int_{\Omega}X\mathbf{1}_{Y>a}d\mathbb{P}}{\mathbb{P}(Y>a)}=\dfrac{\int_{\mathbb{R}}\int_{\mathbb{R}}\mathbf{1}_{(a,\infty)}(y)xf_{X,Y}(x,y)dxdy}{\mathbb{P}(Y>a)}=\dfrac{\int_{a}^{\infty}\int_{\mathbb{R}}xf_{X,Y}(x,y)dxdy}{\int_{a}^{\infty}f_{Y}(y)dy}.$$ I know that this is the most convenient way to compute the conditional expectation given an event. On the other hand, we can also compute $\mathbb{E}(X|Y=y)$ using the conditional density (suppose that it exists $\neq 0$). And we use it in this way: we know that $\mathbb{E}(X|Y=y)=\mathbb{E}(X|Y)(\tilde{\omega})$ for $\tilde{\omega}\in\{\omega:Y(\omega)=y\}$, and thus $$\mathbb{E}(X|Y=y)=\int_{\mathbb{R}}xf_{X|Y}(x,\tilde{\omega})dy=\int_{\mathbb{R}}x\dfrac{f_{X,Y}(x,Y(\tilde{\omega}))}{f_{Y}(Y(\tilde{\omega})}dx=\dfrac{1}{f_{Y}(y)}\int_{\mathbb{R}}xf_{X,Y}(x,y)dx.$$

---

My question is, what should I do next to go back to $\mathbb{E}(X|Y>a)$? For example, perhaps $$\mathbb{E}(X|Y>a)=\int_{\{\omega:Y(\omega)>a\}}\mathbb{E}(X|Y)(\omega)d\mathbb{P}?$$ or $$\mathbb{E}(X|Y>a)=\int_{a}^{\infty}\mathbb{E}(X|Y=y)dy=\int_{a}^{\infty}\dfrac{1}{f_{Y}(y)}\int_{\mathbb{R}}xf_{X,Y}(x,y)dxdy?$$ For the first guess, I cannot proceed further since it sort of require me to know the joint density between $Y$ and $\mathbb{E}(X|Y)$? For the second, I don't see a promising way to express it in terms of the fraction for $\mathbb{E}(X|Y>a)$.

Any idea or reasonable guess? I know that some post has mentioned this before, e.g. How to write $E(X|A)=\frac{E(X\mathbf{1_A})}{\mathbb{P}(A)}$ in form of conditional density and Expectation of $X$ conditional on $aX+(1-a)Y>c$.. However, the solution was to redefine the conditional density when conditioning on an event. I want to directly use the density $f_{X|Y}(x,\tilde{\omega})$ and $\mathbb{E}(X|Y=y)$ to see if there anyway to go back, instead of redefining the density.

Edit: the answer from Henry needs $X$ and $Y$ to be absolutely continuous. There is a way to show the same equality by only assuming that $Y$ is absolutely continuous (and more rigorous in the definition of $\mathbb{E}(X|Y=y)$.) Please see my own answer of this post. I really appreciate all the discussion and help!

Answer 1

From comment:

$\mathbb{E}[X\mid Y=y]$ just looks like some function of $y$.

Why can you not use $$\dfrac{\int_a^\infty \mathbb E[X \mid Y=y] \, f_Y(y)\, dy}{\int_a^\infty \mathbb f_Y(y)\, dy}\,?$$

You need the marginal density for $Y$ but, since you say you know $\mathbb{E}[X\mid Y=y]$, you do not need any joint or conditional densities.

Answer 2

Instead, we can only require $Y$ to be absolutely continuous. Since $\mathbb{E}(X|Y)$ is $\sigma(Y)$-measurable, there exists a Borel function $g$ such that $\mathbb{E}(X|Y)=g(Y)$. Moreover, by the definition of conditional expectation, we have $\mathbb{E}(X\mathbf{1}_{A}|Y)=\mathbb{E}(X\mathbf{1}_{A})$ for all $A\in\sigma(Y)$. Note that $A:=\{Y>a\}=Y^{-1}((a,\infty))\in\sigma(Y)$, and thus we have \begin{align*} \mathbb{E}(X|Y>a)=\dfrac{\int_{Y>a}Xd\mathbb{P}}{\mathbb{P}(Y>a)}=\dfrac{\int_{Y^{-1}((a,\infty))}\mathbb{E}(X|Y)d\mathbb{P}}{\mathbb{P}(Y>a)}&=\dfrac{\int_{Y^{-1}((a,\infty))}g(Y)d\mathbb{P}}{\mathbb{P}(Y>a)}\\ &=\dfrac{\int_{a}^{\infty}g(y)\mathcal{P}_{Y}(dx)}{\mathbb{P}(Y>a)}\\ &=\dfrac{\int_{a}^{\infty}g(y)f_{Y}(y)dy}{\mathbb{P}(Y>a)}\\ &=\dfrac{\int_{a}^{\infty}\mathbb{E}(X|Y=y)f_{Y}dy}{\int_{a}^{\infty}f_{Y}(y)dy}. \end{align*}

---

Note that $\mathbb{E}(X|Y=y)$ can be defined to be any Borel function $g=g(y)$ such that $$\int_{X^{-1}(B)}Xd\mathbb{P}=\int_{B}g(y)\mathcal{P}_{Y}(dy),\forall B\in\mathcal{B}(\overline{\mathbb{R}})$$

The LHS is a signed measure that is absolutely continuous with respect to $\mathcal{P}_{Y}$, and Radon-Nikodym gives us a version of conditional expectation.

Hence, the proof above fits more in this rigorous setting