Let $b,a$ be $1\times 3$ vectors and $X$ be $3 \times 3$.
If my calculations are correct this leads to $b^Tb-2b^TaX+X^Ta^TaX$ . How can approach this?
EDIT: Given X is an orthogonal matrix $3 \times 3$, take $|b-Xa|^2=0$. for b,a $3 \times 1$ . This leads to: $b^Tb-2b^TXa+a^Ta$. Now everything is a scalar right? X=?
If you're using the inner product on row vectors defined by $\langle x,y\rangle=xy^T$, the equation $|b-aX|^2=0$ is equivalent to $b=aX$.
If $X$ is orthogonal, then it is invertible and the above is also equivalent to $a=bX^T$.
Nothing more can be said except that the norms of $a$ and $b$ are the same: indeed, $$ |b|^2=|aX|^2=(aX)(aX)^T=aXX^Ta^T=aa^T=|a|^2 $$
Conversely, given two row vectors $a$ and $b$ such that $|a|=|b|$ there exist infinitely many orthogonal matrices $X$ such that $b=aX$.