Given $2x^2+2xy+y^2+\sqrt2x-\sqrt2y$ and $cot2\theta=\frac{A-C}{B}$, I determined $\theta \approx 31.717^{\circ}$
In order to eliminate the $xy$ term, I applied $x=x'cos\theta-y'sin\theta$ and $y=x'sin\theta + y'cos\theta$. Unfortunately, I continually get to a point where the $xy$ terms are not like, resulting in a non-solution.
Any help regarding eliminating the $xy$ term using these formulae would be appreciated.
First, we compute $\cos \theta$ and $\sin \theta$ from the cotangent relationship. Let $\theta = \frac{1}{2} \cot^{-1} \frac{1}{2}$. Then by considering a right triangle with adjacent leg $1$, opposite leg $2$, and hypotenuse $\sqrt{1^2 + 2^2} = \sqrt{5}$, we see that $$\cos \cot^{-1} \frac{1}{2} = \frac{1}{\sqrt{5}}. \tag{1}$$ Hence by the half-angle identity, $$\cos \theta = \cos \frac{1}{2} \cot^{-1} \frac{1}{2} = \sqrt{ \frac{1 + \cos 2\theta}{2}} = \sqrt{\frac{1 + 1/\sqrt{5}}{2}}, \quad \sin \theta = \sqrt{ \frac{1 - 1/\sqrt{5}}{2} }. \tag{2}$$ Since $\theta$ is in quadrant I, both of the signs for $\cos \theta$ and $\sin \theta$ in $(2)$ will be positive.
Next, we easily compute $$2 \sin \theta \cos \theta = \sin 2\theta = \sin \cot^{-1} \frac{1}{2} = \frac{2}{\sqrt{5}}. \tag{3}$$
So for the transformation $$(x,y) = (x' \cos \theta - y' \sin \theta, x' \sin \theta + y' \cos \theta), \tag{4}$$ we find $$\begin{align} x^2 &= x'^2 \cos^2 \theta - 2x'y' \cos \theta \sin \theta + y'^2 \sin^2 \theta \\ &= \frac{1 + 1/\sqrt{5}}{2} x'^2 - \frac{2}{\sqrt{5}} x'y' + \frac{1 - 1/\sqrt{5}}{2} y'^2, \tag{5} \\ xy &= (x'^2 - y'^2) \cos \theta \sin \theta + x'y' (\cos^2 \theta - \sin^2 \theta) \\ &= (x'^2 - y'^2) \frac{1}{2} \sin 2\theta + x'y' \cos 2\theta \\ &= \frac{1}{\sqrt{5}} (x'^2 + x'y' - y'^2), \tag{6} \\ y^2 &= x'^2 \sin^2 \theta + 2x'y' \cos \theta \sin \theta + y'^2 \cos^2 \theta \\ &= \frac{1 - 1/\sqrt{5}}{2} x'^2 + \frac{2}{\sqrt{5}} x'y' + \frac{1 + 1/\sqrt{5}}{2} y'^2. \tag{7} \end{align}$$
Consequently, $$\begin{align} 2x^2 + 2xy + y^2 &= \left(1 + \frac{1}{\sqrt{5}} + \frac{2}{\sqrt{5}} + \frac{1 - 1/\sqrt{5}}{2}\right) x'^2 + \left(-\frac{4}{\sqrt{5}} + \frac{2}{\sqrt{5}} + \frac{2}{\sqrt{5}}\right) x'y' \\ &\quad + \left(1 - \frac{1}{\sqrt{5}} - \frac{2}{\sqrt{5}} + \frac{1 + 1/\sqrt{5}}{2}\right)y'^2 \\ &= \frac{3+\sqrt{5}}{2} x'^2 + \frac{3-\sqrt{5}}{2}y'^2. \tag{8} \end{align}$$
The linear terms $\sqrt{2} x - \sqrt{2} y$ under the transformation $(4)$ do not contribute any $xy$ terms.