It is easy to see that any binary quadratic form $a^2 + 2bxy + cy^2$ is the same as $XAX^T$ where $X = [x, y]$ and $A = \begin{bmatrix}a & b\\b & c\end{bmatrix}.$
The composition of two binary quadratic forms $f(x,y)$ and $g(z,w)$ is a form $F(s(x,y,z,w),t(x,y,z,w)) = f(x,y)g(z,w)$ where $s = pxz+qxw+ryz+syw$ and similarly for $t$ Is there any way to see this as a matrix operation?
Edit: Corrected a mistake.
Every matrix has corresponding linear transformation... So, $f,g$ have linear transformation respectively and let matrices be $[f]$ and $[g]$..
Now the composition $f\circ g$ is just the product of the matrices $[f].[g]$.
Hope will help...