First, I want to identify the set of all linear transformations $A$ leaving the Euclidean product invariant. Let $\mathbf{u}$ be a vector: $$ \mathbf{u}:=(x,y) $$
$$ f(\mathbf{u})=||\mathbf{u}||=x^2+y^2=\pmatrix{x\\y}\pmatrix{x& y}=\mathbf{u}^T\mathbf{u} $$
Proof:
$$ f(A\mathbf{u})=(A\mathbf{u})^T(A\mathbf{u}) = \mathbf{u}^TA^TA\mathbf{u} $$
Consequently, $A^TA=I$. This is the orthogonal set $O(2)$ as expected.
Second, I want to repeat the same but for the Lorentz product. Using the same vector, we define $||\mathbf{u}||$ as:
$$ f(\mathbf{u})=||\mathbf{u}||=x^2-y^2=\pmatrix{1 & 0\\0& -1}\pmatrix{x\\y}\pmatrix{x & y}=\eta\mathbf{u}^T\mathbf{u} $$
Proof:
$$ f(A\mathbf{u})=\eta(A\mathbf{u})^T A\mathbf{u}=\eta \mathbf{u}^TA^TA\mathbf{u} $$
I get the same group as the previous case: $O(2)$. Why am I not getting the Lorentz transformations?
Your matrix product does not makes sense. $\pmatrix{1 & 0\\0& -1}\pmatrix{x\\y}$ is a column vector, that is a $2\times 1$ matrix, and $\pmatrix{x & y}$ is a row vector, that is a $1\times 2$ matrix. Your multiplication should give a $2\times 2$ matrix.
Even for computing $A\mathbf{u}$, you get a problem, because $A$ is a $2\times 2$ matrix and $\mathbf{u}$ is a $1\times 2$ matrix with your notation, so you cannot do the product.
In the sequel, I will take a column vector : $\mathbf{u}=\pmatrix{x \cr y}$.
The right way to write your Lorentz norm is $f(\mathbf{u})=\mathbf{u}^T \ \eta \ \mathbf{u}$, where $\eta=\pmatrix{1 & 0\\0& -1}$.
Then a transformation $A$ will preserve the norm if and only if $A^T \ \eta\ A= I_2$.
PS. For the euclidean case, $f(\mathbf{u})=\mathbf{u}^T \mathbf{u}$, with the column vector $\mathbf{u}$.