From a uniform disk of radius $R$ a circular disk of radius $\frac{R}{2}$ is being cut out.

The center of the "cut out" disk is at $R/2$ from the venter of the original disk. We have to find the center of mass of leftover body.
I thought that we should set up a coordinate system with the center of original disk as the origin. The formula for center of mass is
$$
\vec{R}_{CM} = \frac{1}{M_{tot}} \int \vec{r} ~dm$$
I thought of creating another identical region (identical to what is being cut out) on the left of $O$. Like this
By symmetry, any position vector $\vec{r}$ outside the encircled region (on the left) will have it's counter-part and hence it will cancel up. So, we need to worry only about the integral inside the circular region, even there by symmetry we know that $\vec{R}_{CM}$ will lie on the axis joining their centers (let's call the line joining all three centers as $x$-axis and the line perpendicular to this line as $y$-axis). If we use the polar coordinate then we have
$$
R_{CM} = \frac{1}{M_{tot}} 2\int \int r \cos \theta \sigma dA\\
\text{(I have written $2r\cos \theta$ because that's the thing we would get when we add any two}\\
\text{vectors in that encircled region, $\sigma$ is the mass per unit area, and $dA$ is the area element)}\\
R_{CM} = \frac{1}{M_{tot}}2 \sigma \int_{r=0}^{R} \int_{\theta=0} r \cos \theta ~dA
$$
But the problem is that I don't know the upper limit of $\theta$, I worked hard but it seemed a little different in this case.
We can use our ordinary cartesian system, $$ R_{CM} = \frac{1}{M_{tot}} 2\sigma \int \int x dx dy$$ the limit of $x$ will be (I think) $0$ to $R$ and we can get $y$ as $$ (x-R/2)^2 + y^2 = R^2/4 \\ y= \sqrt{ x^2 - Rx}$$ So, we have $$ R_{CM} = \frac{1}{M_{tot}} 2\sigma \int_{x=0}^{R} \int_{y=0}^{\sqrt{x^2-Rx}} x dy dx\\ R_{CM} = \frac{1}{M_{tot}} 2\sigma \int_{x=0}^{R} x\sqrt{x^2-Rx} ~dx $$ I don't know how to carry out that last integral.
The answer to this question is "$R/6$ to the left of O" but where am I mistaking? Can someone help me out?
Given the symmetry you noted, it seems not worth it to do the integral in polar coordinates when you know the center of mass will only have an $x$. Let's rotate your picture upside down. Notice that computing the integral will give us
$$\iint\limits_{\text{shaded region}}^{} x\:dA = \iint\limits_{\text{small circle on right side}}^{} x\:dA$$
by symmetry because $x$ is an odd function. Next, you want integrate w.r.t. $x$ first because the square roots will cancel
$$\bar{x} =\frac{4}{3\pi R^2}\int_{-\frac{R}{2}}^\frac{R}{2} \int_{\frac{R}{2}-\sqrt{\frac{R^2}{4}-y^2}}^{\frac{R}{2}+\sqrt{\frac{R^2}{4}-y^2}} x\:dx \:dy = \frac{4}{3\pi R}\int_{-\frac{R}{2}}^\frac{R}{2} \sqrt{\frac{R^2}{4}-y^2}\:dy = \frac{R}{6}$$