In this question I want to consider a quadratic form $\mathcal{Q}$ on an even integral finite rank lattice (not necessarily self-dual, unimodular, or positive definite).
Given an explicit such quadratic form, are there elementary methods to write down exactly what the full symmetry group is? If so, say one has found a subgroup of the full symmetry group, is there a way to be sure that in fact the complete symmetry group has been found?
One example of interest to me is the following. Consider the lattice $\Gamma =\mathbb{Z}^{3}$ endowed with quadratic form,
$$\mathcal{Q}(a,b,c) = 2(ab + ac + bc)$$
I found a group representation of $\text{PGL}_{2}(\mathbb{Z})$ on the lattice automorphisms of $\Gamma$ defined by the following three automorphisms:
$$\sigma_{A}(a,b,c) = (b,a,c), \,\,\,$$ $$\sigma_{B}(a,b,c) = (a+2c, b+2c, -c), \,\,\,$$ $$\sigma_{C}(a,b,c) = (a+2b, 2b+c, -b).$$
(The above are the images of the three usual generators of $\text{GL}_{2}(\mathbb{Z})$.) One can show easily that the three automorphisms above preserve the quadratic form $\mathcal{Q}$. Therefore, whatever the full symmetry group of $\mathcal{Q}$ is, I believe it at least contains $\text{PGL}_{2}(\mathbb{Z})$ as a subgroup. Now in this case I'm nearly certain the full symmetry group is much larger, so as indicated in my question above, how can I determine what the rest of it is, if indeed this is possible for this form?
Your form $yz + zx + xy$ is $SL_3 \mathbb Z$ equivalent to $y^2 - zx.$ (Actually its negative, doesn't matter). Therefore the entire automorphism group is the image of $PSL_2 \mathbb Z.$
See, for example, page 23 in Magnus, Noneuclidean Tessellations and Their Groups.
Given $\alpha \delta - \beta \gamma = 1,$ this gives all elements in the oriented automorphism group of the ternary quadratic form $g(x,y,x) = y^2 - zx.$ That is
$$ \left( \begin{array}{ccc} \alpha^2 & \alpha \gamma & \gamma^2\\ 2 \alpha \beta & \alpha \delta + \beta \gamma & 2 \gamma \delta\\ \beta^2 & \beta \delta & \delta^2 \end{array} \right) \left( \begin{array}{ccc} 0 & 0 & -1\\ 0 & 2 & 0 \\ -1 & 0 & 0 \end{array} \right) \left( \begin{array}{ccc} \alpha^2 & 2 \alpha \beta & \beta ^2\\ \alpha \gamma & \alpha \delta + \beta \gamma & \beta \delta\\ \gamma^2 & 2 \gamma \delta & \delta^2 \end{array} \right) = \left( \begin{array}{ccc} 0 & 0 & -1\\ 0 & 2 & 0 \\ -1 & 0 & 0 \end{array} \right) $$
It is not amazingly difficult to prove these are all. The first and third columns of the right hand matrix must be null vectors of $y^2 - z x,$ with coprime entries. It follows that the columns are some $(r^2,rs,s^2)$ and then some $\pm(u^2, uv,v^2).$ Then fiddle with the second column until it works.
This business is also discussed in Cassels, Rational Quadratic Forms, especially pages 301-303, chapter 13 section 5, "Isotropic Ternary Forms." He gives the observation of Fricke and Klein (1897), pages 507-508 as I recall, that any isotropic ternary integrally represents an integer multiple of $y^2 - zx.$ I first found that in a paper by Plesken. I wrote up a proof using no projective geometry methods, let me find that. http://math.stackexchange.com/questions/1972120/ternary-quadratic-forms/1976199#1976199