F$(x,y) = (2x^2+2xy, 2y^2+2xy)$ where the curve is $f(\theta) = \theta, \theta \in [0,\pi/2]$.
So i attempted to convert the curve into cartesian parametric coordinates so that
$$\gamma = (\theta cos(\theta), \theta sin(\theta))$$
Hence F$(x,y) = (2\theta^2 cos^2(\theta) + 2\theta^2 cos(\theta)sin(\theta), 2\theta^2 sin^2(\theta) + 2\theta^2 cos(\theta)sin(\theta))$
and $\gamma' = (-\theta sin(\theta), \theta cos(\theta)) $
However when i attempt compute the line integral i get $$\int_0^{\pi/2} F dr = 0$$
and the answer in my book says the answer is $ \approx 3.218$
Imo, the parametrization must be
$$\gamma(t)=(t,t)\;,\;\;t\in\left[0,\,\frac\pi2\right]\;\;,\;\;\;\gamma'(t)=(1,1)\implies$$
$$\int_\gamma (2x^2+2xy,\,2y^2+2xy)\cdot d\vec r =\int_0^\frac\pi2 (2t^2+2t^2,\,2t^2+2t^2)\bullet(1,1)dt=$$
$$\int_0^{\frac\pi2}8t^2\,dt=\frac83\left(\frac\pi2\right)^3$$
But then we get something very different of what you say is the answer , so we must assume the curve is in polar coordinates and thus...do basically the same you did! Though I think there is a mistake in the differentiating the curve:
$$\gamma(\theta):=\left(\theta\,\cos\theta,\,\theta\,\sin\theta\right)\;,\;\;0\le\theta\le\frac\pi2\;,\;\;\gamma'(\theta)=(\cos\theta-\theta\,\sin\theta,\,\sin\theta+\theta\,\cos\theta)\implies$$
$$\int_\gamma (2x^2+2xy,\,2y^2+2xy)\cdot d\vec r=\int_0^{\frac\pi2}F(\gamma(\theta))\bullet\gamma'(\theta)\,d\theta$$
The last expression is really nasty though not actually very hard, but still it will take a loooong while to write it down and solve it...at leat to me.