I was thinking about how to rotate an ellipse in the xy plane by 30 degrees anticlockwise.
In the below diagram, I was thinking of fixing the original coordiate system xy and expressing the equation in x'y' plane $$\frac{x'^2}{a^2} + \frac{y'^2}{b^2} = 1$$ in terms of $x$ and $y$.
So I substituted $$x'= \frac{\sqrt{3}}{2} x - \frac{1}{2} y$$ and $$y' = \frac{1}{2} x + \frac{\sqrt{3}}{2} y$$
for a = 4 and b = 3 but the result is:
The opposite of what I expected. At what point am I confused?
Your formulae works for rotating the points about the origin . In your case you have rotated the axes counter-clockwise by 30° , so the respective points in the plane have been rotated CLOCKWISE , so you have to rather take the negative value of tha angle rather than the positive value .