If the mass per unit area of a surface is given by $\rho=xy $ find the mass $\int\int_S xy\ dS$ if $S$ is the part of the cylinder $y^2+z^2=16$ which is in the first octant and contained within the cylinder $x^2+y^2=9$?
First I set up parametrization as $x=x, y=4\cos\theta, z=4\sin\theta$. So $|r_x\times r_\theta|=\sqrt{(4\cos\theta)^2+(4\sin\theta)^2}=4$
Then $dS=4dxd\theta$. So $\int\int_S xy\ dS=\int^\frac{\pi}{2}_0\int^?_0 16x\cos\theta\ dxd\theta$. $($it's in first quadrant$)$
I am uncertain about the '?' part. I guessed $?=\sqrt{9-(4\cos\theta)^2}$ but it gives answer different from correct answer, which is $11.36$.
Your idea for the upper limit for $x$ is right, but not for the lower limit for $\theta$
It must be $\arctan{(\sqrt{7}/3)}$
$$\int\int_S xy\ dS=\int^{\frac{\pi}{2} }_{\arctan{\sqrt{7}/3}}\int^{\sqrt{9-(4\cos\theta)^2}}_0 16x\cos\theta\ dxd\theta$$
Draw the intersection of the cylinders with the plane $y$-$z$. The cylinder with axis in $z$ has two parallel lines. The cylinder with axis on $x$ has a circle of radius $4$, the upper-right intersection of these curves is in $y=3$, $z=\sqrt{7}$. Your angle $\theta$ is the one taking as polar axis $y$ and counterclock wise. The tangent for the intersection is $\sqrt{7}/3$. The angle must run from $\arctan{(\sqrt{7}/3)}$until $\pi/2$, the "end" of the first octant.