$X$ is a random variable with pdf proportional to $x$ for $x\in[0, 1]$ and $0$ otherwise.
We pick an $x\sim X$ and toss a coin with $2$ possible outcomes $Y= 0$ or $1$. The probability of getting $Y = 1$ is $x$.
Calculate:
(1) $E[Y]$
(2) $E[Y|X]$
(3) $E[X|Y]$
I know that the first thing to do is to find the actual pdf of $X$, i.e $f_x(X)$ which I do by integration and require that it equals $1$. From this I find $c=2$.
Thus:
$f_{X}(X)=2x$ for $x\in [0, 1]$ and $0$ otherwise.
Now, what I really need is: $f_{Y}(Y)$ to be able to compute the various expected values. But, how do I get it? I'm stuck.
In this answer I choose for a more general setting.
Let $X$ be random variable with support $\left[0,1\right]$ and a CDF $F_{X}$ that is differentiable on $\left(0,1\right)$.
Let $f_{X}$ be the derivative of $F_{X}$ functioning as PDF and let $\mathbb{E}X>0$.
Then $\mathbb{E}\left[Y\mid X=x\right]=x$ for $x\in\left[0,1\right]$ so that $\mathbb{E}\left[Y\mid X\right]=X$ and consequently $\mathbb{E}Y=\mathbb{E}\left[\mathbb{E}\left[Y\mid X\right]\right]=\mathbb{E}X$. This concerns the (more general) answers on $(1)$ and $(2)$.
Observe here that consequently $P\left(Y=1\right)=\mathbb{E}Y=\mathbb{E}X$ which is used in the sequel.
Now let $F_{X\mid Y=1}$ denote the CDF of $X$ under condition $Y=1$.
Then for $x\in\left(0,1\right)$ and small $\delta\neq0$ we find:
$\begin{aligned}\frac{1}{\delta}\left[F_{X\mid Y=1}\left(x+\delta\right)-F_{X\mid Y=1}\left(x\right)\right] & =\frac{1}{\delta}P\left(x<X\leq x+\delta\mid Y=1\right)\\ & =\frac{1}{\delta}P\left(x<X\leq x+\delta\wedge Y=1\right)/P\left(Y=1\right)\\ & =\frac{1}{\delta\mathbb{E}X}P\left(x<X\leq x+\delta\wedge Y=1\right)\\ & =\frac{1}{\delta\mathbb{E}X}P\left(Y=1\mid x<X\leq x+\delta\right)P\left(x<X\leq x+\delta\right)\\ & =\frac{1}{\mathbb{E}X}P\left(Y=1\mid x<X\leq x+\delta\right)\frac{1}{\delta}\left(F_{X}\left(x+\delta\right)-F\left(x\right)\right) \end{aligned} $
The limit for $\delta\to0$ on RHS exists and takes value $\frac{xf_{X}\left(x\right)}{\mathbb{E}X}$.
This allows us to use the function $f_{X\mid Y=1}\left(x\right)=\frac{xf_{X}\left(x\right)}{\mathbb{E}X}$ as PDF of $F_{X\mid Y=1}$ and now we can find:
$\mathbb{E}\left[X\mid Y=1\right]=\int_{0}^{1}xf_{X\mid Y=1}\left(x\right)\;dx=\int_{0}^{1}\frac{x^{2}f_{X}\left(x\right)}{\mathbb{E}X}\;dx=\frac{\mathbb{E}X^{2}}{\mathbb{E}X}$
On a similar way we find $f_{X\mid Y=0}\left(x\right)=\frac{\left(1-x\right)f_{X}\left(x\right)}{1-\mathbb{E}X}$ and:
$\mathbb{E}\left[X\mid Y=0\right]=\int_{0}^{1}xf_{X\mid Y=0}\left(x\right)\;dx=\int_{0}^{1}\frac{x\left(1-x\right)f_{X}\left(x\right)}{1-\mathbb{E}X}\;dx=\frac{\mathbb{E}X-\mathbb{E}X^{2}}{1-\mathbb{E}X}$.
Now we can state:
$$\mathbb{E}\left[X\mid Y\right]=\frac{\mathbb{E}X^{2}}{\mathbb{E}X}Y+\frac{\mathbb{E}X-\mathbb{E}X^{2}}{1-\mathbb{E}X}\left(1-Y\right)\tag3$$
If the PDF is proportional then we are dealing with $f_X(x)=2x$ for $x\in[0,1]$ and find easily that $\mathbb EX=\frac23$ and $\mathbb EX^2=\frac12$.
Substituting these values in $(3)$ we find that in your case:$$\mathbb{E}\left[X\mid Y\right]=\frac34Y+\frac12\left(1-Y\right)=\frac12+\frac14Y$$