So I have a set of vectors that I verified are orthonormal. $$ \begin{equation*} A = \begin{pmatrix} \frac{1}{3\sqrt{2}} & \frac{2}{3} & \frac{1}{\sqrt{2}} \\ \frac{1}{3\sqrt{2}} & \frac{2}{3} & -\frac{1}{\sqrt{2}} \\ -\frac{4}{3\sqrt{2}} & \frac{1}{3} & 0 \end{pmatrix} \end{equation*}$$ The column vectors of $A$ are orthonormal since I multiplied $A^TA$ to get $I_3$. I want to get the vector $x= \begin{pmatrix} 1 \\ 2 \\ 5 \end{pmatrix}$ written as a linear combination of the column vectors of $A$ above.
I know in order to get a linear combination of a vector in terms of another set you would append the additional column vector to matrix $A$, get the matrix into reduced row echelon form, and multiply each column(vector) by the corresponding components in the last column then sum them. But this matrix is an orthonormal set. Is there a more concise way to do this, knowing we have a orthonormal set of vectors, and use that to get $||x||^2$?
Yes, there is a concise way of doing it, your intuition is correct. Let $u_1 = (\frac{1}{3 \sqrt{2}}, \frac{1}{3 \sqrt{2}}, \frac{-4}{3 \sqrt{2}})$, $u_2 = (\frac{2}{3}, \frac{2}{3}, \frac{1}{3})$, and $u_3 = (\frac{1}{\sqrt{2}}, \frac{-1}{\sqrt{2}}, 0).$
We know from general theory of orthonormal bases that any vector $v \in \mathbb R^3$ can be uniquely expressed as \begin{equation} \begin{split} v = \langle v, u_1 \rangle u_1 + \langle v, u_2 \rangle u_2 + \langle v, u_3 \rangle u_3. \end{split} \end{equation}
So a quick way is just to compute each of these inner products, which become the coefficients of the linear combination.