Consider an urn containing a large number of coins, each of the coin has some probability of $p$ of turning up head when flipped. However, the value of $p$ varies from coin to coin. Suppose that the $p$-value of a randomly chosen coin can be regarded as being the value of a random variable that is distributed over $[0,1]$ with a probability density function $$f_{P}(p)=\frac{3}{2}p(2-p)$$
What is the conditional expectation of the $p$-value given that the randomly chosen coin turns up head when flipped?
First, by the law of total probabilty, the probabilty of turning up head is $$Pr(H)=\int_{0}^{1}pf_{P}(p)dp=\frac{5}{8}$$
Now $$E[p|H]=\int_{0}^{1}pf_{P|H}(p|H)dp$$ And $$f_{P|H}(p|H)=\frac{f_{P,H}(p, H)}{Pr(H)}$$ But I have problem finding $f_{P,H}(p, H)$, it looks like an joint pdf of a random variable and an event which confuses me.
We have, by Bayes' rule,
$$f_P(p \mid H) = \frac{\Pr[H \mid P = p]f_P(p)}{\Pr[H]}. \tag{1}$$
Note that $\Pr[H \mid P = p] = p$; that is to say, the probability of a single flip of the coin being heads, given that the probability of heads is $p$, is simply $p$. Hence the numerator is simply $$\Pr[H \mid P = p]f_P(p) = p \cdot \frac{3}{2}p(2-p) = \frac{3}{2}p^2(2-p). \tag{2}$$
What is the denominator? This is the unconditional probability of getting heads on a single flip of a randomly selected coin. To calculate this, we apply the law of total probability:
$$\Pr[H] = \int_{p=0}^1 \Pr[H \mid P = p]f_P(p) \, dp = \int_{p=0}^1 \frac{3}{2}p^2 (2-p) \, dp = \frac{5}{8}, \tag{3}$$
as you have computed. Notice this value does not depend on $p$.
Consequently,
$$f_P(p \mid H) = \frac{12}{5} p^2(2-p). \tag{4}$$
The posterior expectation, given heads is observed, is then
$$\operatorname{E}[P \mid H] = \int_{p=0}^1 p f_P(p \mid H) \, dp = \int_{p=0}^1 \frac{12}{5} p^3 (2-p) \, dp = \frac{18}{25}. \tag{5}$$
As a further exercise, what is the posterior variance $\operatorname{Var}[P \mid H]$?
Now suppose we assume the same prior distribution for $P$, but this time, we select one coin at random and flip it twice. What is the posterior expectation if: