I was working on the solution of this exercise but the solution's book give details that I'm not pretty sure if they're ok.
This problem talks about the conditional expectation of a random variable given another random variable ($\mathbb{E}(\xi|\eta)$). So, in order to calculate the conditional expectation we need first to find the $\sigma$-algebra $\sigma(\eta)$.
Well, it's clear that $\eta(x)$ is a symmetric function about $\frac{1}{2}$. This random variable is: $\eta:[0,1]\rightarrow[0,1]$, so if I want to find the $\sigma(\eta)$, I need to choose a borel set, $B \in [0,1]$, and find $\{\eta \in B\}=\{x\in [0,1]:\eta(x)\in[0,1]\}$. I can see that for any $B$ there will be two borel sets, say, $A_{1}$ and $A_{2}$ which will be $\{\eta \in B\}$: $$\{\eta \in B\}=A_{1}\cup A_{2}$$ and what I've noticed is that $A_{2}=1-A_{1}$. So here is my first question. This expression $1-A$ where $A$ is a interval set, means that, e.g., $A=[0.3, 0.4]$, then $1-A=[0.6, 0.7]$? I used that representation because Brzezniak use it under the idea of symmetry. If it that's true, also, $A_{1} \subseteq [0,\frac{1}{2}]$ and so it does $A_{2}$.
Brzezniak propose a Borel set $A \subset [0,1]$ which is symmetric about $\frac{1}{2}$, i.e.: $$A=1-A$$ Here, I have another question, how is possible $A=1-A$ if e.g. $A=[0.1, 0.3]$, then $1-A=[0.7,0.9]$? Again, what does it mean $1-A$? Then, how is it possible that $A=1-A$? the only way to have that is when $A=[0, \frac{1}{2}]$.
For me $A$ and $1-A$ are differents but they're: $\{\eta\in B\}=\{A\} \cup \{1-A\}$ and with these sets I can fill the $\sigma$-algebra $\mathcal{B}([0,1])$. So, $\mathbb{E}(\xi|\eta)$ is $\sigma(\eta)$-measurable and also it's true, as Brzezniak says, that $\mathbb{E}(\xi|\eta)(x) = \mathbb{E}(\xi|\eta)(1-x)$. What he does next is transform the integral of $\int_{A}2x^{2}dx$ to make the integrand symmetric about $\frac{1}{2}$. I assume that he does like that because he's using the fact that $\int_{A}\mathbb{E}(\xi|\eta)dP=\int_{A}\xi dP$, so he is calculating $\int_{A}\xi dP = \int_{A}2x^{2}dx$, Am I right?
I can't upload a picture of the book's solution but is in the page 35. Then, he just do a split of $\int_{A}2x^{2}dx$ in order to obtain a result: $$\int_{A}\Big(x^2+(1-x)^2 \Big)dx$$
and he finally says that $\mathbb{E}(\xi|\eta)=x^2+(1-x)^2$ and for that, I assume, that he use the fact that: $$\int_{A}\mathbb{E}(\xi|\eta)(x)dx= \int_{A}\Big(x^2+(1-x)^2 \Big)dx$$ It's true that $x^2+(1-x)^2$ is symetric, but how can I be sure if that function is $\mathbb{E(\xi|\eta)}$ if it isn't the final result of $\int_{A}\xi dx$ but it's symmetric.
Please, someone explain to me.
I forgot to answer my own question. I want to make clear all the doubts that I had in that moment. The question itself is very interesting because it is a applied exercise from a theoretical definition like the Conditional Expectation for random variables.
The first question about the definition of symmetry $A=1-A$ has the next explanation. If we consider any subset $A_{1}$ in $\mathbb{R}$ and if we know that it has the property of symmetry, then we know that $A_{1}=1-A_{2}$, where we are saying that the values in the subset $A_{1}$ once they are evaluated in the function $f(x)$, we will obtain the same values if we evaluate $f(x)$ in the subset $1-A_{2}$. The subset $1-A$ means take values transforming the order of $A$, e.g. if $A=[0.1,0.3]$ then we will get $1-A=[0.7,0.9]$ and one aditional restriction is that $A \subset [0,1/2]$. So, the $\sigma(\eta)=A_{1} \cup A_{2}$ where $A_{1}=1-A_{2}$ and $A_{1} \subset [0,1/2]$. If we choose any subset of the Borel Set $\mathcal{B}([0,1])$ we will get an inverse image of $\eta$ of the form like stated before. And we know that the inverse image of a random variable is a $\sigma$-algebra, so it is enough to define the form of the space of inverse images of $\eta$.
By definition of conditional expectation we know that: $$\int_{D}\mathbb{E}(\xi|\eta)dP=\int_{D}\xi dP, \quad \forall \, D \in \sigma(\eta)$$ We will calculate $\int_{D}\xi dP$ and once we have the result we will proof that $\mathbb{E}(\xi|\eta)$ is the integrand in the result of $\int_{D}\xi dP$. $$\int_{D}\xi dP = \int_{D}2x^{2}dx$$ here we know that $D \in \sigma(\eta)$ so we can say that $D=A_{1} \cup A_{2}$ where $A_{1}=1-A_{2}$ and $A_{1} \subset [0,1/2]$, then: $$\int_{D}2x^{2}dx=\int_{A_{1} \cup A_{2}}2x^{2}dx$$ $$\int_{D}2x^{2}dx=2\int_{A_{1}}x^{2}dx+2\int_{A_{2}}x^{2}dx$$ where the sets $A_{1}$ and $A_{2}$ are disjoint and: $$\int_{D}2x^{2}dx=2\int_{A_{1}}x^{2}dx+2\int_{1-A_{1}}x^{2}dx$$ here we will use the variable change (remember that $A_{1}$ is a subset of $\mathbb{R}$), so: $$\int_{D}2x^{2}dx=2\int_{A_{1}}x^{2}dx+2\int_{A_{1}}(1-x)^{2}dx$$ $$\int_{D}2x^{2}dx=2\int_{A_{1}}x^{2}+(1-x)^{2}dx$$ At first sight, it is not easy to notice that we have found another symmetric function that we will call $h(x)=x^{2}+(1-x)^{2}$. By symmetry we mean that $h(x)=h(1-x)$, so we have that: $$2\int_{A_{1}}x^{2}+(1-x)^{2}dx=\int_{A_{1}}x^{2}+(1-x)^{2}dx+\int_{A_{1}}x^{2}+(1-x)^{2}dx$$ and we will use again the fact that $A_{1}=1-A_{2}$, then: $$=\int_{A_{1}}x^{2}+(1-x)^{2}dx+\int_{1-A_{2}}x^{2}+(1-x)^{2}dx$$ we apply again the variable change, then: $$=\int_{A_{1}}x^{2}+(1-x)^{2}dx+\int_{A_{2}}(1-x)^{2}+(x)^{2}dx$$ and finally we will obtain: $$\int_{D}\xi dP=\int_{A_{1} \cup A_{2}}x^{2}+(1-x)^{2}dx$$ Now, using the definition of conditional expectation, we have that: $$\int_{D}\mathbb{E}(\xi|\eta) dP=\int_{A_{1} \cup A_{2}}x^{2}+(1-x)^{2}dx$$ So, in order to set that $\mathbb{E}(\xi|\eta)=x^{2}+(1-x)^{2}$ this must be true for all $D \in \mathcal{B}([0,1])$ and $h(x)=x^{2}+(1-x)^{2}$ must be $\sigma(\eta)$-measurable. The first part was proved when we used the fact that for any $D$ it has the form of $A_{1} \cup A_{2}$ with the addtional restrictions. The second part, we can use the fact that for any $B \in \mathcal{B}([0,1])$ in the range of $h$ we will find inverse images of the form $A_{1} \cup A_{2}$ because $h(x)$ is symmetric and all those inverse images are subsets of $\sigma(\eta)$. QED