The quadratic equation $5x^2-6xy+5y^2+20 \sqrt{2}x -28 \sqrt{2}y + 72 = 0$ becomes $(5−3 \sin(2α))x'^2 + (5+3 \sin(2α)) y'^2 - 6\cos(2α)x'y' + 4\sqrt{2}(5\cos(α)-7\sin(α))x' - 4\sqrt{2}(5\sin(α)+7\cos(α))y'+72=0$
after applying the rotation $x=x'cos\alpha - y'\sin\alpha, y=x'\sin\alpha + y'\cos\alpha$.
Determine the type of the conic with equation above. Hence, determine, with respect to the $x',y'$ -coordinate system, the coordinates of the center, focus (foci), vertex (vertices) and where applicable asymptotes. Hence determine the coordinates of the focus (foci) in the $x,y$ coordinate system.
Working: The $x'y'$ coefficient must be $0$, so $\alpha = \frac{\pi}{4}$. This gives after simplification $\frac{(x'-2)^2}{4} + \frac{(y'-3)^2}{1}=1$.
Type: Ellipse
Center: $(2,3)$
Foci: $(2 + \sqrt{3},3) , (2-\sqrt{3},3)$
Vertices: $(4,3) , (0,3)$
No asymptotes
In $x-y$ coordinate system, the foci are $(2+\sqrt{3},3),(2-\sqrt{3},3)$.
Are these correct. I am confused babout the meaning of the finding the things like foci and center in the $x',y'$ coordinate system. Is that basically the $x-y$ axis rotated $\frac{\pi}{4}$ radians anticlockwise?
These are correct.
This is not correct. The foci are given by $$x=(2\pm\sqrt 3)\cos\frac{\pi}{4}-3\sin\frac{\pi}{4}=\frac{-\sqrt 2\pm\sqrt 6}{2}$$ $$y=(2\pm\sqrt 3)\sin\frac{\pi}{4}+3\cos\frac{\pi}{4}=\frac{5\sqrt 2\pm\sqrt 6}{2}$$
What you did is as follows :
You rotated the ellipse $5x^2-6xy+5y^2+20 \sqrt{2}x -28 \sqrt{2}y + 72 = 0$ by $(-\frac{\pi}{4})$ anticlockwise (i.e., $\frac{\pi}{4}$ clockwise) around the origin, and got the ellipse $\frac{(x-2)^2}{4} + \frac{(y-3)^2}{1}=1$. See here.
In general, we can say the followings :
If we rotate a point $(x,y)$ by $\color{red}{\theta}$ anticlockwise around the origin, then we get $(x\cos\theta-y\sin\theta,x\sin\theta+y\cos\theta)$.
If we rotate a curve $f(x,y)=0$ by $\color{red}{(-\theta)}$ anticlockwise (i.e., $\theta$ clockwise) around the origin, then we get a curve whose equation is $f(x\cos\theta-y\sin\theta,x\sin\theta+y\cos\theta)=0$.
Added :
You correctly got $\frac{(x'-2)^2}{4} + \frac{(y'-3)^2}{1}=1$.
In the $x',y'$ -coordinate system, we have, as you wrote
Center: $(2,3)$
Foci: $(2 + \sqrt{3},3) , (2-\sqrt{3},3)$
Vertices: $(4,3) , (0,3)$
Then, you are asked to determine the coordinates of the foci in the $x,y$ coordinate system.
This means that you are asked to determine the coordinates of the foci of the ellipse $$5x^2-6xy+5y^2+20 \sqrt{2}x -28 \sqrt{2}y + 72 = 0\tag1$$
As I wrote, the foci of $(1)$ are given by $$x=(2\pm\sqrt 3)\cos\frac{\pi}{4}-3\sin\frac{\pi}{4}=\frac{-\sqrt 2\pm\sqrt 6}{2}$$ $$y=(2\pm\sqrt 3)\sin\frac{\pi}{4}+3\cos\frac{\pi}{4}=\frac{5\sqrt 2\pm\sqrt 6}{2}$$
This is because if we rotate a point $(x,y)$ by $\color{red}{\theta}$ anticlockwise around the origin, then we get $(x\cos\theta-y\sin\theta,x\sin\theta+y\cos\theta)$.
In the similar way, we can see that the center and the vertices of the ellipse $(1)$ are as follows :
Center : $(-\frac{\sqrt 2}{2},\frac{5\sqrt 2}{2})$
Vertices : $(\frac{\sqrt 2}{2}, \frac{7\sqrt 2}{2}),(-\frac{3\sqrt 2}{2},\frac{3\sqrt 2}{2})$
You might want to see here and here.