Find the probability that the equation $x^2-2ax+b=0$ has complex roots, if $a,b$ are random variables following the Uniform $(0,h)$ distribution individually and independently.
So we effectively need to determine $P(b>a^2)$ which, in my case turns out to be $1-\dfrac{h}{3}$ if $h\leq 1$ and $\dfrac{2}{3\sqrt{h}}$ if $h>1$.
However, the answers as mentioned at the back of the text, are $\dfrac{h}{3}$ and $\dfrac{1}{3\sqrt{h}}$ respectively. I would have got those answers if I had calculated $P(b<a^2)$ which is clearly not correct as we are looking for complex roots.
My working sketch:
We want $\int_{a}P(b>a^2)\dfrac{1}{h}da$. So to make the probability positive, we must have $0<a<\min\{h,\sqrt{h}\}$
So we break into two cases, one where $h\leq1$ and the other where $h>1$ and perform the integration in each case, subject to $0<a<h$ in first case and $0<a<\sqrt{h}$ in second case.
Please point out any error you can find.
Your answer is correct as explained and written. Completing the square gives $$0 = x^2 - 2ax + b = x^2 - 2ax + a^2 + (b-a^2) = (x-a)^2 + (b-a^2),$$ and since no real square is negative, then the equation has no solution if $b-a^2 > 0$; i.e., $b > a^2$. The desired probability is therefore $$\Pr[b > a^2] = 1 - \Pr[b < a^2] = 1 - \int_{u=0}^h \min(u^2,h) \cdot \frac{1}{h^2} \, du.$$ The easiest way to see the book cannot be correct is to choose $h = 1$: The integral immediately becomes $1/3$, hence the probability is $2/3$. That is not consistent with the answer given by the text.