I want to calculate the conditional expectation $E(X|Y)$, assuming $X(x)=x^2$ and $\{Y=y\}=[\frac{y-1}{4},\frac{y}{4})$ for $y=1,2,3,4$ with $P(Y=y)=\frac{1}{4}$ on the probability space $\left([0,1), \mathcal{B}^1_{|[0,1)}, \lambda^1_{|[0,1)}\right)$.
I am trying to solve it as follows:
By the definition of conditional expectation, \begin{align*} E(X|Y=y) &= \sum_{i=1}^4 x_i \frac{P(X=x_i,Y=y)}{P(Y=y)} \\ &= 4 \sum_{i=1}^4 x_i P(X=x_i,Y=y) \\ &= 4 \int_{\{Y=y\}} X dP \\ &= 4 \int_{[\frac{y-1}{4},\frac{y}{4})} x^2 dP \end{align*}
I have to stop here. Because I don't understand how to integrate $x^2$ over $[\frac{y-1}{4},\frac{y}{4})$ with $dP$. Is the $P$ defined by $P(X=x^2,Y=y)$ and what is this? Can you show me a sketch of $X$, $Y$, and $E(X|Y)$ on $[0,1)$?
Since the probability space is the unit interval with Lebesgue measure, the integral $dP$ is just normal calculus integration: $$\int_{[\frac{y-1}{4},\frac{y}{4})} x^2 dP=\int_{(y-1)/4}^{y/4}x^2dx.$$
Looking at it again, your derivation is flawed. The equation $E[X|Y=y] = \sum_{i} x_i P(X=x_i,Y=y)/P(Y=y)$ is only valid when $X$ is a discrete random variable. This does not apply here, since $X$ is continuous.
What you instead want is $$ E[X|Y=y] = \frac{E[X1_{\{Y=y\}}]}{P(Y=y)}=4\int X1_{\{Y=y\}}\,dP=4\int_0^1 x^21_{[(y-1)/4,y/4)}\,dx=4\int_{(y-1)/4}^{y/4}x^2\,dx $$ The first equality is the definition of the conditional expectation of a (general) random variable on an event. The third inequality follows from the substitutions $X=x^2,\{Y=y\}=[(y-1)/4,y/4)$, and $dP=dx$.