Let $U_1$, $U_2$ and $U_3$ be independent random variables uniform on $[0, 1]$. Find the probability that the roots of the quadratic $ U_1 x^2 + U_2 x + U_3 $ are real.
The roots of a quadratic are real if $b^2 - 4ac \geq 0$ so the goal is to find $P(U_2^2 - 4U_1 U_3 \geq 0) = P(U_2^2 \geq 4U_1 U_3 )$
My attempt: $$P(U_2^2 \geq 4U_1 U_3 ) = P(U_2 \geq 2\sqrt{U_1 U_3} ).$$
I'm omitting the possibility of $U_2 \leq - 2\sqrt{U_1 U_3}$ since $ U_2 \geq 0 $. The joint density of the three independent uniform random variables is $f(u_1, u_2, u_2) = 1$ which needs to integrated over:
$$ \int_{0}^{1} \int_{0}^{1} \int_{2\sqrt{u_1 u_3}}^{1} du_2du_1du_3 = \int_{0}^{1} \int_{0}^{1} (1 - 2\sqrt{u_1u_2})du_1du_3 = \int_{0}^{1} [u_1 - \frac{4}{3}\sqrt{u_3}u_1^{3/2}]^{1}_0du_3 \quad = \int_{0}^{1}(1 - \frac{4}{3}\sqrt{u_3}) du_3 = [u_3 - \frac{8}{9}u_3^{3/2}]^1_0 = 1 - \frac{8}{9} = \frac{1}{9}$$
The answer given in the textbook is $ \frac{5}{36} + \frac{log(2)}{6} $.
Your error is in the first step: It is possible to have $2\sqrt{u_1u_3} > 1$ in which case the last integral should give $0$ rather than the negative quantity $(1-2\sqrt{u_1 u_3)}$.
To do this right you have to break up the integral $du_1$ into two pieces (the second pice gives zero since $u_2$ cannot be big enough if $u_1 > 1/(4u_3)$) and the breakup also requires breaking the integral $du_3$ into two pieces because $1/(4u_3)$ can also be greater than $1$.
$$ \int_{u_3=1/4}^1 du_3\int_{u_1 = 0}^{1/(4u_3)}du_1\int_{u2=\sqrt{2u_1u_3}}^1du_2 + \int_{u_3=0}^{1/4} du_3\int_{u_1 = 0}^1du_1\int_{u2=\sqrt{2u_1u_3}}^1du_2 \\= \int_{u_3=1/4}^1 du_3\int_{u_1 = 0}^{1/(4u_3)}du_1(1-\sqrt{2u_1u_3}) + \int_{u_3=0}^{1/4} du_3\int_{u_1 = 0}^1du_1(1-\sqrt{2u_1u_3}) $$
and you can take it from there.
Needless to say, some other order of integration might come out easier.