I am reading linear algebra textbook. One question is it said for vectors in the unit circle $||x||=1$. I can understand this. However, it said the vectors $y=Ax$ in the ellipse will have $||A^{-1}y||=1$. Why it is an ellipse?
I only know the ellipse can be expressed in this way: $\frac{x^2}{a^2}+\frac{y^2}{b^2} = 1 $
A partial answer: your formula for an ellipse holds if the (semi-minor and semi-major) axes of the ellipse are horizontal and vertical. More generally, the ellipses can point in any two perpendicular directions.
With that in mind, suppose that $\vec u,\vec v$ are orthogonal unit vectors. The component of $(x,y)$ along $\vec u = (u_1,u_2)$ is the dot-product $(u_1,u_2)\cdot (x,y) = \vec u \cdot \vec x$. We can produce the equation of an ellipse with axes in the direction of $u$ and $v$ by using the component of the vector $\vec x$ along $\vec u$ instead of the $x$-value and the component along $\vec v$ instead of the $y$-value. Doing this gives us the equation $$ \frac{(\vec u \cdot \vec x)^2}{a^2} + \frac{(\vec v \cdot \vec x)^2}{b^2} = 1 $$ Now, let $M$ denote the matrix $$ M = \pmatrix{u_1/a & u_2/a\\ v_1/b & v_2/b}. $$ Verify that $M \vec x$ = $((\vec u\cdot \vec x)/a, (\vec v \cdot \vec x)/b)$. It follows that the equation of our ellipse is given by $$ \|M \vec x\|^2 = 1 \implies \|M\vec x\| = 1. $$ In other words, we see that every ellipse can be written in the form $\|M \vec x\| = 1$ for some matrix $M$.
The original question was why is it that $\|A^{-1}x\| = 1$ defines an ellipse for every (invertible) matrix $A$. So far, my question only answers this in the case that the rows of $A^{-1}$ happen to be orthogonal to each other.
One approach to answering this question is to combine our analysis above with the existence of the singular value decomposition.