The following is the Question:
Consider a $3 \times 3$ real matrix $A=\left(\begin{array}{ccc}1 & -2 & 0 \\ -2 & 5 & -1 \\ 0 & -1 & 2\end{array}\right)$.
(a) Show that $A$ is nonsingular and hence find $A^{-1}$.
(b) Let $B$ be a $2 \times 3$ matrix satisfied both of the following conditions:
$\operatorname{det}\left(B^T B\right)=\operatorname{det}\left(B B^T\right)$;
$A^{-1} B^T B A=\left(\begin{array}{ccc}16 & -16 & -36 \\ 4 & -4 & -9 \\ -8 & 8 & 18\end{array}\right)$
Find a nonzero matrix $C$ such that $$ B C B^T=\mathcal{O} $$ where $\mathcal{O}$ denotes the $2 \times 2$ zero matrix.
I've done part a and show that $A^{-1}=\left(\begin{array}{ccc}9 & 4 & 2 \\ 4 & 2 & 1 \\ 2 & 1 & 1\end{array}\right)$. And I have question about part b. Actually I have no clue how to find the matrix $C$. Moreover, I would like is there any related concepts or theorem related to the question in part b. I have learned "diagonalization" in matrix before and found that the concept may be similar but not sure how they are related . Thank you.
Presumably the matrices have real elements. Then $\operatorname{rank}(B^TB)=\operatorname{rank}(B)=\operatorname{rank}(B^T)=\operatorname{rank}(BB^T)$. The condition $\det(B^TB)=\det(BB^T)$ thus implies that the rank of $B$ is at most $1$. Hence we may write $B=uv^T$, where $u\in\mathbb R^2$ and $v\in\mathbb R^3$.
Now note that $$ M:=\pmatrix{16 & -16 & -36 \\ 4 & -4 & -9 \\ -8 & 8 & 18} $$ is a rank-one matrix. Therefore it is possible to find two nonzero vectors $x$ and $y$ such that $M=xy^T$. After they are found, one needs $$ xy^T=M=A^{-1}B^TBA=A^{-1}vu^Tuv^TA=\|u\|^2(A^{-1}v)(Av)^T. $$ Therefore $v$ must be chosen such that $A^{-1}v$ is a nonzero scalar multiple of $x$. One may simply pick $v=Ax$.
Hence $BCB^T=uv^TCvu^T=(v^TCv)uu^T$ is zero if and only if $v^TCv=0$. It should be easy to construct an appropriate matrix $C$. E.g. one may pick any nonzero vector $w\perp v$ and set $C=ww^T$, or pick any nonzero vector $(c_1,c_2,c_3)\perp(v_1^2,v_2^2,v_3^2)$ and set $C=\operatorname{diag}(c_1,c_2,c_3)$.