Notation:
- Two binary quadratic forms $f(x, y) = ax^2 + bxy + cy^2, g(x, y) = kx^2 + lxy + my^2$ are called equivalent, if there exists $\begin{bmatrix}p & q \\ r & s\end{bmatrix} \in \text{SL}_2(\mathbb{Z})$, such that $f(px + qy, rx + sy) = g(x, y)$.
- Two fractional ideals $\mathfrak{a, b} \subseteq K, \mathfrak{a, b} \neq 0$ are called equivalent if there exists $\alpha \in K$ with $N(\alpha)$ satisfying $\mathfrak{b} = \alpha\mathfrak{a}$.
Motivation and Proposition: For a binary quadratic form $f(x, y) = ax^2 + bxy + cy^2$ with $a > 0$ we can define a fractional ideal $\mathfrak{a}_f := \mathbb{Z} + \frac{b + \sqrt{\Delta_K}}{2a}\mathbb{Z}$. This indeed is an fractional ideal. Now I want to show that this assignment maps different equivalent binary quadratic forms to equivalent fractional ideals.
Attempt: Let $f(x, y) = ax^2 + bxy + cy^2, g(x, y) = kx^2 + bxy + cy^2$ be two equivalent binary quadratic forms. I need to show that there exists $\alpha \in K, N(\alpha) > 0$ satisfying $\mathfrak{a}_f = \alpha\mathfrak{a}_g$ or $\mathfrak{a}_g = \alpha\mathfrak{a}_f$ using the existence of $\begin{bmatrix}p & q \\ r & s\end{bmatrix} \in \text{SL}_2(\mathbb{Z})$, such that $f(px + qy, rx + sy) = g(x, y)$. At this point I can't see how to proceed and ask for your help.
Disclaimer: I already saw this post, but don't really see how to apply a similar proof here.
As a first step, we look at how the discriminants of $f$ and $g$ relate. Indeed, we may write $f$ in matrix form. First introduce abbreviations
$$ A = \begin{pmatrix} a & \frac{b}{2} \\ \frac{b}{2} & c \end{pmatrix} $$
and
$$ B = \begin{pmatrix} p & q\\ r & s \end{pmatrix}. $$
Thus,
$$ f(x,y) = \begin{pmatrix} x & y \end{pmatrix} A \begin{pmatrix} x \\ y \end{pmatrix} $$
and by assumption
$$ g(x,y) = \begin{pmatrix} x & y \end{pmatrix} B^T A B \begin{pmatrix} x \\ y \end{pmatrix}. $$
But
$$ \det(B^T A B) = \det(A) $$
because $\det(B) = 1$ and multiplicativity of the determinant, so that the fractional ideals associated to $f$ and $g$ via the formula you gave are not only equivalent, but equal!