The goal is to prove that $\vec{r}_1$ is perpendicular to $\vec{r}_2$.
To begin, I started with the definition that if two vectors are perpendicular to each other, then their scalar product will be zero. For the case of this question, $$ \vec{r}_1 \cdot \vec{r}_2 = 0 $$ Expanding this would give $$ \vec{r}_1 \cdot \vec{r}_2 = r_{1x}r_{2x} + r_{1y}r_{2y} $$ From the diagram, we know that $R + r_x = r_{1x}$, and $R - r_x = r_{2x}$, so \begin{align} \vec{r}_1 \cdot \vec{r}_2 &= (R + r_x)(R - r_x) + r_{1y}r_{2y}\\ &= R^2 - r_x^2 + r_{1y}r_{2y} \end{align} Since $r_y = r_{1y} = r_{2y}$, $$ \vec{r}_1 \cdot \vec{r}_2 = R^2 - r_x^2 + r_y^2 $$ Then substituting in $r_x^2 = r^2 - r_y^2$ \begin{align} \vec{r}_1 \cdot \vec{r}_2 &= R^2 - (r^2 - r_y^2) + r_y^2\\ &= R^2 - r^2 + r_y^2 + r_y^2\\ &= R^2 - r^2 + 2r_y^2 \end{align} Since $\|\vec{r}\| = r = R$ \begin{align} \vec{r}_1 \cdot \vec{r}_2 &= R^2 - R^2 + 2r_y^2\\ &= 2r_y^2 \end{align} But this cannot be so. The scalar product must be equal to $0$. How is it that its equal to $2r_y^2$? Where did I go wrong in my math?

That's $r_{2x} = r_x-R$, not the reverse.