Gauss defined the composition of binary quadratic forms $f$ and $g$ to be another binary quadratic form $F$ such that there exist integral quadratic forms
$$ \begin{align} r(x_0,x_1,y_0,y_1) &= p_0 x_0y_0 + p_1 x_0y_1 + p_2 x_1y_0 + p_3 x_1y_1\\ s(x_0,x_1,y_0,y_1) &= q_0 x_0y_0 + q_1 x_0y_1 + q_2 x_1y_0 + q_3 x_1y_1 \end{align}$$
for which
$$ F(r(x_0,x_1,y_0,y_1),s(x_0,x_1,y_0,y_1)) = f(x_0,y_0)g(x_1,y_1) $$
(and satisfying one extra technical condition).
In Section 236 of Gauss's Disquitiones Mathematicae, he shows how to find a form which is composed of two other forms. After laying out the notation, he begins by saying that you can take ad libitum four integers satisfying a pretty minor condition (minor enough that there are usually infinitely many ways to select these integers). He then constructs a composition of two forms, which depends on how you choose the four integers.
In Section 242, he moves to considering the composition of two binary quadratic forms of the same discriminant. At that point, he says that we can take the four integers to be $-1$, $0$, $0$, and $0$ and does so seemingly for the remainder of the work. With this choice, we get the composition operation on forms (and not just on ideal classes) that is common in other works. Namely, the first coefficient of the composition is a divisor of the product of the first coefficients of the forms to be composed, and the middle coefficient of the composition is obtained by solving a certain system of congruences. (A special case of this is Dirichlet's definition of composition.)
By choosing the four integers differently, you can produce other forms that are compositions of two given forms. For instance, the following identities are readily checked:
$$ (5x_0^2 + 7x_0y_0 + y_0^2)(5x_1^2 + 7x_1y_1 + y_1^2) = \begin{cases} 5r_0^2 + 13r_0s_0 + 7s_0^2, \\ 7r_1^2 + 15r_1s_1 + 7s_1^2, \end{cases}$$
with
$$ \begin{align} r_0&=x_0y_0-2x_0y_1-2x_1y_0-3x_1y_1, & s_0 &=x_0y_0+3x_0y_1+3x_1y_0+4x_1y_1\\ r_1&=3x_0y_0 - x_0y_1 - x_1y_0 - 2x_1y_1, & s_1&=-x_0y_0+2x_0y_1+2x_1y_0+3x_1y_1\end{align}$$
Thus, we see both $5x^2 + 13xy + 7y^2$ and $7x^2 + 13xy + 7y^2$ exhibited as compositions of the form $5x^2 + 7xy + y^2$ with itself.
When Gauss turns to defining the class group for a given discriminant beginning in Section 242, he seems to make a rather arbitrary choice of which form to use as the composition of two given forms of the same discriminant. My question is: is there anything canonical about Gauss's definition (other than -1, 0, 0, and 0 being nice, simple numbers)? His operation turns the classes of primitive forms under the action of the matrices of the form $\begin{bmatrix} 1 & c\\ 0 & 1 \end{bmatrix}$ into a group. But could the compositions have been systematically chosen in a different way to produce a group?
Or if it is not canonical, are there several functions from the ideals in a quadratic field to the corresponding binary quadratic forms through which multiplication of ideals translates into different operations on forms?
The composition of binary quadratic forms that has always been used is a version developed by Dirichlet. This can be found on page 49 in the first edition of Cox, with a correction in the second edition.
It is quite recent that Bhargava showed that there were actually 14 distinct ways to, say, implement the conditions described by Gauss. Bhargava got a Fields Medal for the job.