I tried many times, but I always find isometries.
Let $\mathscr{C}$ be the conic: $x^2+2y^2=1$
Find a linear application $F$, that is not an isometry, s.t. $F(\mathscr{C})=\mathscr{C}$. Are the applications which map $\mathscr{C}$ in itself a finite set?
On
Notice that if we had the circle $x^2 + y^2 = 1$ we would be able to display all the linear self-maps. The ones with positive determinant are $$ \left( \begin{array}{cc} \cos \theta & \sin \theta \\ - \sin \theta & \cos \theta \end{array} \right) $$ while the ones with negative determinant are $$ \left( \begin{array}{cc} \cos \theta & \sin \theta \\ \sin \theta & - \cos \theta \end{array} \right) $$ In order to use these, we need first to map the ellipse to the circle, then use one of the above, then back from the circle to your ellipse. Ellipse to circle, simplest version, is $$ \left( \begin{array}{cc} 1 & 0 \\ 0 & \sqrt 2 \end{array} \right) $$ All of the maps are among $$ \left( \begin{array}{cc} 1 & 0 \\ 0 & \frac{1}{ \sqrt 2} \end{array} \right) \left( \begin{array}{cc} \cos \theta & \sin \theta \\ - \sin \theta & \cos \theta \end{array} \right) \left( \begin{array}{cc} 1 & 0 \\ 0 & \sqrt 2 \end{array} \right) = \left( \begin{array}{cc} \cos \theta & \sqrt 2 \; \sin \theta \\ - \frac{\sin \theta}{\sqrt 2} & \cos \theta \end{array} \right) $$ $$ \left( \begin{array}{cc} 1 & 0 \\ 0 & \frac{1}{ \sqrt 2} \end{array} \right) \left( \begin{array}{cc} \cos \theta & \sin \theta \\ \sin \theta & - \cos \theta \end{array} \right) \left( \begin{array}{cc} 1 & 0 \\ 0 & \sqrt 2 \end{array} \right) = \left( \begin{array}{cc} \cos \theta & \sqrt 2 \; \sin \theta \\ \frac{\sin \theta}{\sqrt 2} & - \cos \theta \end{array} \right) $$
With these you can easily check the fact that each of the matrices $P$ above is an "automorphism" of the quadratic form, represented by half its Hessian matrix. This is exactly what @amd talks about as $$ \color{magenta}{ M^T C M = C } $$
$$ \color{teal}{ \left( \begin{array}{cc} \cos \theta & - \frac{\sin \theta}{\sqrt 2} \\ \sqrt 2 \; \sin \theta & \cos \theta \end{array} \right) \left( \begin{array}{cc} 1 & 0 \\ 0 & 2 \end{array} \right) \left( \begin{array}{cc} \cos \theta & \sqrt 2 \; \sin \theta \\ - \frac{\sin \theta}{\sqrt 2} & \cos \theta \end{array} \right)= \left( \begin{array}{cc} 1 & 0 \\ 0 & 2 \end{array} \right)} $$ $$ \color{blue}{ \left( \begin{array}{cc} \cos \theta & \frac{\sin \theta}{\sqrt 2} \\ \sqrt 2 \; \sin \theta & - \cos \theta \end{array} \right) \left( \begin{array}{cc} 1 & 0 \\ 0 & 2 \end{array} \right) \left( \begin{array}{cc} \cos \theta & \sqrt 2 \; \sin \theta \\ \frac{\sin \theta}{\sqrt 2} & -\cos \theta \end{array} \right)= \left( \begin{array}{cc} 1 & 0 \\ 0 & 2 \end{array} \right) } $$
For one such transformation, consider reflection in the line $x=y$. This maps the principal axes of the ellipse onto each other. Now scale to adjust the semiaxis lengths appropriately. In matrix form, this transformation is $$\begin{bmatrix}\sqrt2&0\\0&\frac1{\sqrt2}\end{bmatrix} \begin{bmatrix}0&1\\1&0\end{bmatrix} = \begin{bmatrix}0&\sqrt2\\\frac1{\sqrt2}&0\end{bmatrix}.$$ This is quite obviously not an isometry, although it is area-preserving. Playing with the signs of the non-zero elements of this matrix gets you a few other transformations that also map the ellipse onto itself.
There are many more that this handful, though. Let $C=\operatorname{diag}(1,2)$. Basically, you’re looking for $2\times2$ invertible matrices $M$ such that $M^{-T}CM^{-1}=C$, equivalently, $M^TCM=C$. If you multiply this out element by element, you get three independent equations in the four elements of $M$. This system is underdetermined, so if there are any solutions at all—which we know there are from above—there will be an infinite number of them. The solution set is a one-parameter family of linear transformations that map $\mathscr C$ onto itself.