Given the rotation matrix:
$$Q=\begin{bmatrix}\frac{2}{\sqrt{5}}& \frac{1}{\sqrt{5}}\\-\frac{1}{\sqrt{5}}& \frac{2}{\sqrt{5}}\end{bmatrix},$$ I want to find the equation of the conic $C$ given by $$x^2+4xy+4y^2-10\sqrt{5}x=0,$$ which I know is a parabola, before applying the rotation to it.
I was thinking about solving the system of equation $$\begin{cases}x=\frac{2}{\sqrt{5}}\tilde{x}+\frac{1}{\sqrt{5}}\tilde{y}\\ y=-\frac{1}{\sqrt{5}}\tilde{x}+\frac{2}{\sqrt{5}}\tilde{y}\end{cases},$$ and then substituting $x$ and $y$ in the initial equation of the conic. The result should be $$5\tilde{y}^2-20\tilde{x}-10\tilde{y}=0,$$ but it seems that my results are different.
Your approach is entirely correct; plugging in your substitutions into the first equation for $C$ yields the result you claim it should be: \begin{eqnarray*} x^2+4xy+4y^2-10\sqrt{5}x&=& \left(\frac{2}{\sqrt{5}}\tilde{x}+\frac{1}{\sqrt{5}}\tilde{y}\right)^2+4\left(\frac{2}{\sqrt{5}}\tilde{x}+\frac{1}{\sqrt{5}}\tilde{y}\right)\left(-\frac{1}{\sqrt{5}}\tilde{x}+\frac{2}{\sqrt{5}}\tilde{y}\right)\\ &\ &+4\left(-\frac{1}{\sqrt{5}}\tilde{x}+\frac{2}{\sqrt{5}}\tilde{y}\right)^2-10\sqrt{5}\left(\frac{2}{\sqrt{5}}\tilde{x}+\frac{1}{\sqrt{5}}\tilde{y}\right)\\ &=&\frac{1}{5}(4\tilde{x}^2+4\tilde{x}\tilde{y}+\tilde{y}^2)+\frac{4}{5}(-2\tilde{x}^2+3\tilde{x}\tilde{y}+2\tilde{y}^2)\\ &\ &+\frac{4}{5}(\tilde{x}^2-4\tilde{x}\tilde{y}+4\tilde{y}^2)-10(2\tilde{x}+\tilde{y})\\ &=&5\tilde{y}^2-20\tilde{x}-10\tilde{y}. \end{eqnarray*} You seem to have made a mistake in your calculation.