let $A, B $ be two independent random variables uniformly distributed on $[0,1]$
so $\mathbb{P*} = \mathbb{P}(A^2-B > 0) = \int_{\mathbb{R_{+}}} f_{A^2-B = U}(u)du$
I tried proceeding but got stuck en route :
let $g(U,V) = (A^2-B, A)$ then $(A,B) = (V, V^2-U)$
so $$f_{U,V}(u,v)= |J(a,b)|^{-1}f_{A,B}(v,v^2-u) = |2v|^{-1}\chi_{(v^2-1,v^2)}(u)\cdot \chi_{(0,1)}(v)$$
Hence
$$f_U(u) = \int_{\mathbb{R}}|2v|^{-1}\chi_{(v^2-1,v^2)}(u)\cdot \chi_{(0,1)}(v)dv$$
how do you compute this integral ? also, is there asimpler approach ?
Here is how I would find $P*$. It can also be thought of as the $P(B<A^2)$. The joint pdf for A and B is 1 for 0≤a≤1 and 0≤b≤1, 0 otherwise since they are iid uniform (0,1). So to find the probability you want, integrate the joint probability over the region where 0≤b≤a^2≤1. Easiest to integrate first wrt b from 0 to a^2, then with respect to a from 0 to 1.