Given two vectors $a=(1,t, t^{3})$ $b=(0,1,3t^{2})$
I have seen that $|a\times b|^{2} = (1+t^{2}+t^{6})(1+9t^{4})-(t+3t^{5})^{2}$
It seems that its has been done in some shorthand way maybe other than calculating the cross product then the absolute value then squaring.
Yes, it's the identity $(a\times b)\cdot (a\times b)=(a\cdot a)(b\cdot b)-(a\cdot b)^2$, which is essentially the cartesian coordinates version of $\sin^2\theta=1-\cos^2\theta$.