Let $X$ be the number of coin tosses until heads is obtained. Suppose that the probability of heads is unknown in the sense that we consider it to be a random variable $Y \in U(0, 1)$.
$(a)$ Find the distribution of $X$
this is what i have come to:
$X \mid Y = y \enspace \in Geo(y)$
The unconditional distribution can be calculated as: $\mathbb{P}(X = k) = \int_{0}^{1} \mathbb{P}(X = k, Y = y ) dy = \int_{0}^{1} \mathbb{P}(X = k \mid Y = y ) f_Y(y) dy = \int_{0}^{1} (1-y)^{k}y^1 \cdot 1 dy = \frac{\Gamma(2)\Gamma(k+1)}{\Gamma( k + 3) } \int_{0}^{1} \frac{\Gamma( k + 3) }{\Gamma(2)\Gamma(k+1)}(1-y)^{k+1-1}y^{2 -1 } \cdot 1 dy = \frac{\Gamma(2)\Gamma(k+1)}{\Gamma( k + 3) } = 1/(k+2)(k+1)$
where i have used the beta density ,which is given by: $f(x) = \frac{\Gamma(r+s)}{\Gamma(r)\Gamma(s)} x^{r-1}(1-x)^{s-1}$ , $\beta(r,s), r,s > 0$
However the answer should be $1/k(k+1)$, so where did i go wrong?
The criteria "tosses until a head" means we are using the geometric distribution rather than the shifted geometric distribution (which would count "tails before a head"). (Misfortunately the shifted geometric distribution is called just "the geometric distribution" in some texts. This is a leading cause of confusion. It is always best to be clear on just which you are using, and whyfor.)
$$\begin{align} (X\mid Y) \sim {\cal Geo}(Y) &\iff \Pr(X=k\mid Y=y) = (1-y)^{k-1} y, \forall k\in \{1,2,\ldots \} \\[1ex] Y \sim {\cal U}[0,1] & \iff f_Y(y) = 1 \\[2ex] \Pr(X=k) & = \int_0^1 \Pr(X=k \mid Y=y) f_Y(y)\operatorname{d}y \\ &= \int_0^1 \left((1-y)^{(k-1)}y\right) \cdot 1\operatorname{d}y \\ & = \left[-\frac{(1-y)^k}{k} y\right]_{y=0}^{y=1} + \int_0^1 \frac{(1-y)^k}{k}\operatorname{d}y \\ & = 0 + \left[-\frac{(1-y)^{k+1}}{k(k+1)} \right]_{y=0}^{y=1} \\[1ex] \therefore \Pr(X=k) & = \frac{1}{k(k+1)} \end{align}$$
For completion: if we had been counting $X_s$ the number of tails before a head occurs — thus using a shifted geometric distribution — then:
$$X_s\mid Y \sim {\cal Geo}_s(y) \iff \Pr(X_s=k\mid Y=y) = (1-y)^k y, k\in\{{\bf 0},1,2,\ldots\} \\ \therefore \Pr(X_s=k) = \frac{1}{(k+1)(k+2)}$$