Let $\boldsymbol G_1=\boldsymbol F_1-\boldsymbol n\left(\boldsymbol F_1\cdot\boldsymbol n\right)$ and $\boldsymbol G_2=\boldsymbol F_2-\boldsymbol n\left(\boldsymbol F_2\cdot\boldsymbol n\right)$ are $3\times 3$ matrices, where $\boldsymbol n\in\mathbb{R}^3$ is the unit normal, $\boldsymbol F_1=\left[\boldsymbol F_{01},a~\boldsymbol n\right]$, $\boldsymbol F_{01}$ is a $3\times2$ matrix and $a\in\mathbb{R}$ is a scalar, $\boldsymbol F_2=\left[a\nabla\boldsymbol n,0\right]$. I need to evaluate $\boldsymbol G_1^T\boldsymbol G_1$, $\boldsymbol G_2^T\boldsymbol G_2$ and $\boldsymbol G_1^T\boldsymbol G_2+\boldsymbol G_2^T\boldsymbol G_1$.
Evaluating, for instance, $\boldsymbol G_1^T\boldsymbol G_1$ I wrote
$$\boldsymbol G_1^T\boldsymbol G_1=\left[\boldsymbol F_1-\boldsymbol n\left(\boldsymbol F_1\cdot\boldsymbol n\right)\right]^T\left[\boldsymbol F_1-\boldsymbol n\left(\boldsymbol F_1\cdot\boldsymbol n\right)\right]=\left[\boldsymbol F_1^T-\left(\boldsymbol n\left(\boldsymbol F_1\cdot\boldsymbol n\right)\right)^T\right]\left[\boldsymbol F_1-\boldsymbol n\left(\boldsymbol F_1\cdot\boldsymbol n\right)\right]=\boldsymbol F_1^T\boldsymbol F_1-\boldsymbol F_1^T\boldsymbol n\left(\boldsymbol F_1\cdot\boldsymbol n\right)-\left[\boldsymbol n\left(\boldsymbol F_1\cdot\boldsymbol n\right)\right]^T\boldsymbol F_1+\left[\boldsymbol n\left(\boldsymbol F_1\cdot\boldsymbol n\right)\right]^T\boldsymbol n\left(\boldsymbol F_1\cdot\boldsymbol n\right),$$ but stucked.
Terms like $\boldsymbol F_1^T\boldsymbol F_1$ are easy. Any idea how to evaluate terms like $\left[\boldsymbol n\left(\boldsymbol F_1\cdot\boldsymbol n\right)\right]^T\boldsymbol n\left(\boldsymbol F_1\cdot\boldsymbol n\right)$?
I'll assume that you made a typo, and $F_2= a\frac{\partial n}{\partial x}$. Since its dimensions are already $(3\times 3)$, there's no need to pad it with a zero-vector.
First, define the matrix $$P=I-nn^T$$ and note that it is an orthoprojector $$P^2=P=P^T$$ and $n$ is in its nullspace (i.e. $P\,n=0$).
The nullspace properties means that $PF_1=[F_{01},\,0]$.
The normalization condition $(n^Tn=1)$ implies that $(n^T\frac{\partial n}{\partial x}=0)$, and therefore $PF_2 = F_2$.
And finally, notice that the G-matrices are $\,\,G_k=PF_k$.
Putting all of the pieces together $$\eqalign{ G_j^TG_k &= (PF_j)^T(PF_k) \cr\cr G_1^TG_1 &= [F_{01}, 0]^T[F_{01}, 0] \cr G_2^TG_2 &= F_2^TF_2 \cr G_1^TG_2 &= [F_{01}, 0]^TF_2 \cr G_2^TG_1 &= F_2^T[F_{01}, 0] \cr }$$