I need to transform this integral $\int_0^6\int_0^y x \, dx \, dy$ to polar and then find its value.
I'm stuck finding the r-limits of integration.
I need to transform this integral $\int_0^6\int_0^y x \, dx \, dy$ to polar and then find its value.
I'm stuck finding the r-limits of integration.
On
There is a well-known transformation between polar coordinates and Cartesian coordinates as $x=r\cos(\theta), y=r\sin(\theta)$. So your range for the converted integral is $\theta\mid_{\pi/4}^{\pi/2}$ and $r\mid_{0}^{6/\sin(\theta)}$.

On
This is an example in which it's much simpler to do the thing directly than by converting to polar coordinates, but an exercise is an exercise. First, remember that $$ dx\,dy=r\,dr\,d\theta, $$ and $$ x = r\cos\theta. $$ Most of the work now is in finding the bounds. You have $y$ going from $0$ to $6$, and then for each fixed value of $y$, $x$ goes from $0$ to $y$. So it's a triangle with vertices $(0,0)$, $(0,6)$, and $(6,6)$. Draw the picture. You see the angle $\theta$ going from half a right angle up to a right angle, so it's $$ \int_{\pi/4}^{\pi/2} \cdots\cdots \,d\theta. $$ Then, for any fixed $\theta$, you need $r$ going from $0$ out to the distance to the line $y=6$, so you've got $$ \int_{\pi/4}^{\pi/2} \int_0^{6\csc\theta} r\cos\theta \,r\,dr\,d\theta. $$
(Later, you'll have $\displaystyle\int \Big(\text{a function of }\sin\theta\Big) \Big( \cos\theta\,d\theta\Big)$, so that should suggest a particular substitution.)
The well-known change of coordinates is: $x(r,\theta) = r\cos\theta$ and $y(r,\theta) = r \sin \theta$. The first things we need to try and transform are the differential one-forms $dx$ and $dy$, and then the differential two-form $dx \wedge dy$. Recall that if $f = f(r,\theta)$ then $df = f_r \ dr + f_{\theta} \ d\theta$. It follows that:
$$\begin{array}{ccc} dx & = & \cos\theta \ dr - r\sin\theta \ d\theta \, , \\ dy & = & \sin\theta \ dr + r\cos\theta \ d\theta \, . \end{array}$$
Applying the definition of the exterior product: $dx \wedge dy = r \ dr \wedge d\theta$. Putting this together:
$$\int\int x \ dx \wedge dy \equiv \int \int r^2\cos\theta \ dr \wedge d\theta \, . $$ Next we need to deal with the limits. In your question the limits are $0 < x < y$ and $0 < y < 6$. Thus: $0 < r\cos\theta < r\sin\theta$ and $0 < r\sin\theta < 6$. The seond inequality tells us that:
$$0 < r < \frac{6}{\sin\theta} \, .$$
Putting this into the first inequality gives:
$$ 0 < 6\frac{\cos\theta}{\sin\theta} < 6 \implies 0 < \cot \theta < 1 \implies \frac{\pi}{4} < \theta < \frac{\pi}{2} \, . $$
Finally, we can collect all of this together and we get:
$$\int_0^6 \int_0^y x \ dx \wedge dy = \int_{\pi/4}^{\pi/2} \int_{0}^{6/\sin\theta} r^2\cos\theta \ dr \wedge d\theta \, . $$
The first integration is simple, we can integrate $r^2$ with respect to $r$:
$$\int_{\pi/4}^{\pi/2} \int_{0}^{6/\sin\theta} r^2\cos\theta \ dr \wedge d\theta = \int_{\pi/4}^{\pi/2} \left[\frac{1}{3}r^3\cos\theta \right]_{0}^{6/\sin\theta} d\theta = \int_{\pi/4}^{\pi/2} 72\frac{\cos\theta}{\sin^2\theta} \ d\theta \, . $$
This last expression integrates. Notice that the integrand is $\cot\theta \csc^2\theta$ and we can make the substitution $u = cot\theta$ and $du = -\csc^2\theta$ to give $-\int u \ du$. Thus:
$$ \int_{\pi/4}^{\pi/2} 72\frac{\cos\theta}{\sin^2\theta} \ d\theta = 72\left[ \frac{-36}{\sin^2\theta} \right]_{\pi/4}^{\pi/2} = 36 \, . $$