I used the rotation matrix to do this and I ended up with the equation: $$x^2+y^2+2xy+\sqrt{2}x-\sqrt{2}y=0$$
I tried to plot this but none of the graphing softwares that I use would allow it.
Is the above the correct equation for a parabola with vertex (0,0) and axis of symmetry $y=x$ ?
$$\left( \begin{array}{cc} \cos\theta&-\sin\theta\\ \sin\theta&\cos\theta\\ \end{array} \right)\left( \begin{array}{cc} x\\ y\\ \end{array} \right)=\left( \begin{array}{cc} X\\ Y\\ \end{array} \right)$$
For a clockwise rotation of $\frac{\pi}{4}$, $\sin{-\frac{\pi}{4}}=\frac{-1}{\sqrt{2}}$ and $\cos{-\frac{\pi}{4}}=\frac{1}{\sqrt{2}}$
$$\left( \begin{array}{cc} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}}\\ -\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}}\\ \end{array} \right)\left( \begin{array}{cc} x\\ y\\ \end{array} \right)=\left( \begin{array}{cc} X\\ Y\\ \end{array} \right)$$
$$X=\frac{x}{\sqrt{2}}+\frac{y}{\sqrt{2}}$$ $$Y=\frac{-x}{\sqrt{2}}+\frac{y}{\sqrt{2}}$$ $$y=x^2$$ $$\left(\frac{-x}{\sqrt{2}}+\frac{y}{\sqrt{2}}\right)=\left(\frac{x}{\sqrt{2}}+\frac{y}{\sqrt{2}}\right)^2$$ $$\frac{-x}{\sqrt{2}}+\frac{y}{\sqrt{2}}=\frac{x^2}{2}+\frac{2xy}{2}+\frac{y^2}{2}$$ $$-\sqrt{2}x+\sqrt{2}y=x^2+2xy+y^2$$ $$x^2+2xy+y^2+\sqrt{2}x-\sqrt{2}y=0$$
Have I made a mistake somewhere?



Let us start with general conic section
$$Ax^2+Bxy+Cy^2+Dx+Ey+F = 0$$
or equivalently, we can write it as
$$\begin{pmatrix} x & y & 1 \end{pmatrix}\begin{pmatrix} A & B/2 & D/2\\ B/2 & C & E/2\\ D/2 & E/2 & F \end{pmatrix}\begin{pmatrix} x\\ y\\ 1 \end{pmatrix}=0$$
(we will denote the above 3x3 matrix with $M$)
So, let's say you are given a conic section $v^\tau M v = 0$ and let's say we want to rotate it by angle $\varphi$. We can represent appropriate rotation matrix with
$$Q_\varphi=\begin{pmatrix} \cos \varphi& -\sin\varphi & 0\\ \sin\varphi & \cos\varphi & 0\\ 0 & 0 & 1\end{pmatrix}$$
Now, $Q_\varphi$ represents anticlockwise rotation, so we might be tempted to write something like $$(Q_\varphi v)^\tau M (Q_\varphi v) = 0$$ to get conic section rotated by angle $\varphi$ anticlockwise. But, this will actually produce clockwise rotation. Think about it - if $v$ should be a point on the rotated conic, then $Q_\varphi v$ is a point on conic before rotation, thus, the last equation actually means that the new conic rotated anticlockwise will produce the old conic.
So, let us now do your exercise. You have conic $y = x^2$, so matrix $M$ is given by $$ M =\begin{pmatrix} 1 & 0 & 0\\ 0 & 0 & -1/2\\ 0 & -1/2 & 0\end{pmatrix}$$ and you want to rotate your conic clockwise by $\pi/4$, so choose $$Q_{\pi/4}=\begin{pmatrix} \cos \frac\pi4& -\sin\frac\pi4 & 0\\ \sin\frac\pi4 & \cos\frac\pi4 & 0\\ 0 & 0 & 1\end{pmatrix}.$$
Finally, we get equation $$\begin{pmatrix} x & y & 1 \end{pmatrix}\begin{pmatrix} \cos \frac\pi4& -\sin\frac\pi4 & 0\\ \sin\frac\pi4 & \cos\frac\pi4 & 0\\ 0 & 0 & 1\end{pmatrix}^\tau\begin{pmatrix} 1 & 0 & 0\\ 0 & 0 & -1/2\\ 0 & -1/2 & 0\end{pmatrix}\begin{pmatrix} \cos \frac\pi4& -\sin\frac\pi4& 0\\ \sin\frac\pi4 & \cos\frac\pi4 & 0\\ 0 & 0 & 1\end{pmatrix}\begin{pmatrix} x\\ y\\ 1 \end{pmatrix}=0$$
or simplified $$x^2-2xy+y^2-x\sqrt 2-y\sqrt 2 = 0.$$