I am trying to solve the following:
Suppose X is the number of times a coin is tossed until a heads is observed. Let Y denoted the probability of observing heads and assume $f_Y(y)=ky^2$, ie the chance of getting heads is a random variable that changes from toss to toss. The condition distributional $f_{X\mid Y}\sim F_s(y)$. Find $P(X=x)$, $E[x]$, and $P(Y\mid X=n)$.
So far I have $P(X=x)=\int_0^\infty P(X=x\mid Y=y)\cdot P_Y(y)\,dy=\int_0^\infty y(1-y)^{k-1} \cdot ky^2 \, dy=k\int_0^\infty y^3(1-y)^{k-1}.$ This can be manipulated into a Beta and the remaining terms are $\frac{6}{(k+3)(k+2)(k+1)}$ so $P(X=x)=\frac{6}{(k+3)(k+2)(k+1)}$.
For $E[X]$ I have, $E[X]=E_y[E_{x}[X\mid Y]]$. $E_x[X\mid Y]]= E[F_s(y)]=1/y$. So $E[X]=E_y[\frac{1}{y}]=\int_0^\infty\frac{1}{y}\cdot ky^2\,dy$... but this diverges, which I am uncertain if it makes sense.
Lastly the posterior probability $P(Y\mid X)=\frac{P(X\mid Y)\cdot P(y)}{P(x))}=\frac{y(1-y)^{k-1}\cdot ky^{2}}{\frac{6}{(k+3)(k+2)(k+1)}}$. If you multiply and divide by $\frac{(k-1)!}{(k-1)!}$, you get $P(Y\mid X)= \frac{(k+3)!}{3!(k-1)!} y^{3} (1-y)^{k-1}$, this looks awfully close to a binomial distribution, but not quite... Is this to be expected? Should I get a binomial posterior?
First, if the probability of heads on a single toss is $Y$, then $Y \in [0,1]$, hence $$1 = \int_{y=0}^1 f_Y(y) \, dy = \int_{x=0}^1 ky^2 \, dy = \frac{k}{3},$$ so $k = 3$ and the density of $Y$ is $$f_Y(y) = 3y^2, \quad 0 \le y \le 1.$$ This characterizes the prior distribution of the probability of heads, and is actually a special case of a Beta distribution $$f_Y(y) = \frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)} y^{a-1} (1-y)^{b-1}, \quad 0 \le y \le 1,$$ with $a = 3$ and $b = 1$. Now the conditional distribution of the total number of tosses needed to observe the first instance of heads is geometric, given that the probability of heads is constant throughout the series of tosses: $$\Pr[X = x \mid Y = y] = (1-y)^{x-1} y, \quad x = 1, 2, \ldots.$$ Therefore, the marginal (unconditional) distribution of $X$ is $$\begin{align*} \Pr[X = x] &= \int_{y=0}^1 \Pr[X = x \mid Y = y] f_Y(y) \, dy \\ &= 3 \int_{y=0}^1 (1-y)^{x-1} y^3 \, dy \\ &= 3 \cdot \frac{\Gamma(x)\Gamma(4)}{\Gamma(x+4)} \int_{y=0}^1 \frac{\Gamma(x+4)}{\Gamma(x)\Gamma(4)} (1-y)^{x-1} y^{4-1} \, dy \\ &= \frac{18}{(x+3)(x+2)(x+1)x}. \end{align*}$$ Note that you have confused the role of $k$ and $x$ in your calculation, and your result cannot be true because it is not a function of $x$, but $k$. The simplest remedy, of course, is to actually compute the value of $k$ as I have done in the beginning. For the general case of hyperparameters $a, b$, it is easy to see that the unconditional distribution is $$\Pr[X = x] = \frac{\Gamma(a+b)\Gamma(a+1)\Gamma(x+b-1)}{\Gamma(a)\Gamma(b)\Gamma(x+a+b)}, \quad x = 1, 2, \ldots.$$ This is a form of beta-geometric distribution which in turn is a special case of the beta-negative binomial distribution (just as there are versions of the geometric distribution with support on $\{0, 1, 2, \ldots\}$ versus $\{1, 2, \ldots\}$, the resulting beta-geometric also has different versions, which we might regard as location-shifted).
The expectation is $$\operatorname{E}[X] = \sum_{x=1}^\infty x \Pr[X = x]$$ which you can calculate from the above.
The posterior distribution of $Y$ given the single observation $X = n$ is given by Bayes' rule: $$f_{Y \mid X}(y) = \frac{\Pr[X = n \mid Y = y]f_Y(y)}{\Pr[X = n]}.$$ We have already calculated the joint distribution in the numerator, as well as the normalizing constant (the marginal/unconditional distribution) in the denominator. It is immediately obvious that the posterior of $Y$ given $X$ is again Beta--for that is the trick we used to compute the marginal--and that its posterior hyperparameters are $$\alpha^* = \alpha+1, \quad \beta^* = \beta+n-1.$$ Explicitly, $$f_{Y \mid X}(y) = \frac{\Gamma(\alpha+\beta+n)}{\Gamma(\alpha+1)\Gamma(\beta+n-1)} y^{\alpha} (1-y)^{\beta+n-2}, \quad 0 \le y \le 1.$$ We need not do any additional computation because we can recognize that for a Geometric likelihood, the Beta distribution is the conjugate prior: the posterior is again Beta.