A parametrization of the part of the surface $z=yx$ bounded by $x^2+y^2=1$ is \begin{align} x &= u \cos v \\ y &= u \sin v \\ z &= \frac12 u^2 \sin 2v, \end{align} or $$r(u,v)=u \cos v \, {\bf i} + u \sin v \, {\bf j} + \frac12 u^2 \sin 2v \, {\bf k}, \quad 0<v<2\pi, 0<u<1.$$
The norm of the cross product of $r_u$ and $r_v$ is $$\sqrt {u^4+u^2} = u\sqrt {u^2+1},$$ so $$\int^{2\pi}_0\int_0^1u\sqrt {u^2+1}dudv=\int^{2\pi}u\sqrt {u^2+1}=\frac13((4\pi^2+1)^\frac23-1)$$
However, the answer showed in my text book is $\frac{2\pi}{3}(2\sqrt 2-1)$, so I don't know whether I did it wrong or there is a mistake in my textbook.
The first integral equality is not correct---you seem to be integrating with respect to $dv$ but over the interval for $u$.
Since the integrand does not depend on $v$, we can write $$\int_0^{2 \pi} \int_0^1 u \sqrt{u^2 + 1} du\,dv = \int_0^{2 \pi} dv \int_0^1 u \sqrt{u^2 + 1} du = 2 \pi \int_0^1 u \sqrt{u^2 + 1} du$$