Let $A$ be 3×3 orthogonal real matrix.
For $\mathbf{u}$ $= \begin{pmatrix} 1 \\ 1 \\ 1 \end{pmatrix}, \mathbf{v}$ $= \begin{pmatrix} 0 \\ 0 \\ \sqrt3 \end{pmatrix}, \mathbf{w}$ $= \begin{pmatrix} 1 \\ -1 \\ 0 \end{pmatrix}$,
$Au=v, Aw=-w$ and $detA=-1$
Find $A$.
I noted that $u, v, w$ are linearly independent. And I write $A\begin{pmatrix} 1 & 0 & 1 \\ 1 & 0 & -1 \\ 1 & \sqrt3 & 0 \end{pmatrix}=\begin{pmatrix} 0 & a & -1 \\ 0 & b & 1 \\ \sqrt3 & c & 0 \end{pmatrix}.$
Then $A=\begin{pmatrix} 0 & a & -1 \\ 0 & b & 1 \\ \sqrt3 & c & 0 \end{pmatrix}\begin{pmatrix} \frac{1}{2} & \frac{1}{2} & 0 \\ -\frac{1}{2\sqrt3} & -\frac{1}{2\sqrt3} & \frac{1}{\sqrt3} \\ \frac{1}{2} & -\frac{1}{2} & 0 \end{pmatrix}$
Now, how can I find $a, b, c$?? Am I doing this right?
If there is a better solution, please help me.
Your approach is fine; you can proceed by using the fact that $\det(A)=-1$ and $A$ is orthogonal. From $$-1=\det(A)=\det\begin{pmatrix} 0 & a & -1 \\ 0 & b & 1 \\ \sqrt3 & c & 0 \end{pmatrix}\det\begin{pmatrix} \frac{1}{2} & \frac{1}{2} & 0 \\ -\frac{1}{2\sqrt3} & -\frac{1}{2\sqrt3} & \frac{1}{\sqrt3} \\ \frac{1}{2} & -\frac{1}{2} & 0 \end{pmatrix},$$ you find that $a+b=-2$, so you can substitute $b=-(a+2)$. Now multiply the two matrices and choose $a$ and $c$ so that the matrix is orthogonal.
Alternatively, the fact that $Au=v$, i.e. $$A\begin{pmatrix}1\\1\\1\end{pmatrix}=\begin{pmatrix}0\\0\\\sqrt{3}\end{pmatrix},$$ shows that the first two rows of $A$ are orthogonal to $u$. Because $A$ is an orthogonal matrix, it follows that the third row is orthogonal to the first two, so the third row is a scalar multiple of $u$. Its inner product with $u$ equals $\sqrt{3}$, so the third row must equal $\frac{u}{\sqrt{3}}$.
Can you proceed in this way with $Aw=-w$ to determine $A$?