We are asked to write down the integral $$I=\int_0^1 \int_x^{x\sqrt3}f(\sqrt{x^2+y^2})dydx$$ in a polar form first with respect to $\theta$ and then with respect to $r$.
I have recognised that $0<r<2$ and $r=\frac{1}{\cos\theta}$ but I am finding it difficult to find bounds for $\theta$ as a function of $r$.
On
The boundaries are the the three lines of the triangle, which transform in polar to
$$\begin{cases}y = x \\ y = \sqrt{3}x \\ x = 1\end{cases} \implies \begin{cases}\theta = \frac{\pi}{4} \\ \theta = \frac{\pi}{3} \\ r = \sec\theta\end{cases}$$
giving us the integral
$$\int_{\frac{\pi}{4}}^{\frac{\pi}{3}}\int_0^{\sec\theta}f(r)\:r\:dr\:d\theta$$
To swap the integral, we can either look at the Cartesian graph of the lines
where $\theta$ integrals move along concentric circles, or in an $r\theta$ plane
where $\theta$ integrals move in horizontal lines parallel to the $\theta$ axis. In both cases, we can see that region is split in two by the red line, which represents the circle $r^2=2$. This gives us the two integrals
$$\int_0^{\sqrt{2}}\int_{\frac{\pi}{4}}^{\frac{\pi}{3}}f(r)\:r\:d\theta\:dr + \int_{\sqrt{2}}^2\int_{\sec^{-1}(r)}^{\frac{\pi}{3}}f(r)\:r\:d\theta\:dr$$
$$ = \int_0^{\sqrt{2}}\frac{\pi r}{12}f(r)\:dr + \int_{\sqrt{2}}^2\left(\frac{\pi}{3}-\cos^{-1}\left(\frac{1}{r}\right)\right)rf(r)\:dr$$
To express the integral $I$ in polar form, we can make the substitution $x = r\cos\theta$ and $y = r\sin\theta$. This gives us:
$$I = \int_0^1 \int_x^{x\sqrt{3}} f(\sqrt{x^2 + y^2}) \frac{r}{\cos\theta}\sin\theta drd\theta$$
To find the bounds for $\theta$, we need to consider the region of integration in the $xy$-plane. This region is a triangle with vertices at $(0,0)$, $(1,0)$, and $(1,\sqrt{3})$. We can see that the $x$-coordinate varies from 0 to 1 and the $y$-coordinate varies from 0 to $x\sqrt{3}$, as indicated in the integral.
In polar coordinates, the line $y = 0$ corresponds to the angle $\theta = 0$ and the line $y = x\sqrt{3}$ corresponds to the angle $\theta = \frac{\pi}{3}$. Therefore, the bounds for $\theta$ are $\theta = 0$ to $\theta = \frac{\pi}{3}$.
Substituting these bounds into the integral, we get:
$$I = \int_0^{\frac{\pi}{3}} \int_0^{\frac{1}{\cos\theta}} f(r) r\sin\theta drd\theta$$
This is the integral $I$ expressed in polar form with respect to $\theta$. To express the integral in polar form with respect to $r$, we can simply switch the order of integration:
$$I = \int_0^{\frac{1}{\sqrt{3}}} \int_0^{\frac{\pi}{3}} f(r) r\sin\theta d\theta dr + \int_{\frac{1}{\sqrt{3}}}^2 \int_0^{\arcsin\left(\frac{\sqrt{3}}{r}\right)} f(r) r\sin\theta d\theta dr$$
This is the integral $I$ expressed in polar form with respect to $r$.