All the classical groups - orthogonal, unitary, symplectic - arise as automorphism groups of bilinear and sesquilinear forms. Over $F=\Bbb R,\Bbb C$ a bilinear form is a map $B:V\times V\to F$ that is $F$-linear in each argument, where $V$ is a vector space over $F$. Over $F=\Bbb C,\Bbb H$ a sesquilinear form is a $B:V\times V\to F$ that, in addition to biadditivity, satisfies $B(v,w\lambda)=B(v,w)\lambda$ and $B(v\lambda,w)=\overline{\lambda}B(v,w)$ for all vectors $v,w$ and scalars $\lambda$, where $V$ is a right vector space over $F$ and $\overline{\lambda}$ denotes the complex or quaternionic conjugate. Then ${\rm Aut}(B)=\{T\in{\rm GL}(V):B(Tu,Tv)=B(u,v)\textrm{ identically}\}$ is the automorphism group. If $B_{\rm sym}$ and $B_{\rm skew}$ are the symmetric and skew-symmetric parts if $B$ is bilinear, or Hermitian and skew-Hermitian parts if $B$ is sesquilinear, then ${\rm Aut}(B)={\rm Aut}(B_{\rm sym})\cap {\rm Aut}(B_{\rm skew})$ so it suffices to narrow our focus to when $B$ is "pure" (don't think the adjective is standard but we'll use it here).
When the classical groups are defined, they are always automorphism groups of nondegenerate bilinear or sesquilinear forms. So what does ${\rm Aut}(B)$ look like if $B$ is a degenerate form?
As an explicit computation, if $B$ is bilinear on $\Bbb R^3$ with matrix
$$\begin{pmatrix} 1 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 0 \end{pmatrix} $$
then (after a number of deductions) I get
$${\rm Aut}(B)=\left\{\begin{pmatrix}\cosh(t) & \sinh(t) & 0 \\ \sinh(t) & \cosh(t) & 0 \\ x & y & z\end{pmatrix} : x,y,z,t\in\Bbb R,z\ne0\right\}.$$
If we let $Y=\{v\in V:B(v,-)=0\textrm{ identically}\}$ and pick a linear complement $X$, then $B$ (which we assume is pure) restricts to a nondegenerate form on $X$ and the zero form on $Y$. Then any automorphism $T\in{\rm Aut}(B)$ satisfies $TY=Y$ and so (writing $V=X\oplus Y$) comes in block form
$$T=\begin{pmatrix} P & 0 \\ Q & R \end{pmatrix} $$
with $P:X\to X$ and $R:Y\to Y$ invertible, but no condition that I can see on $Q$. Can we deduce $P\in{\rm Aut}(B|_X)$? Is there any better description of the automorphism group?