English is not my first language so I apologize for any mistakes.
Question:
Find the matrix for the linear transformation F in the room that is defined by $\vec{u}$ first mirroring in x+z=0 and then have the mirror image $S(\vec{u})$ depicted on $S(\vec{u})+(1,1,1)$ x $S(\vec{u})$.
I solved the mirroring fairly easily and got: $\begin{pmatrix} 0 &0 & 1 \\ 0 & -1 & 0 \\ 1 & 0 & 0 \end{pmatrix}$
Which means that $S(\vec{u})= \begin{pmatrix}z \\ -y \\ x \end{pmatrix}$
Now comes the part that I'm not completly sure with. What I did was taking the cross product between $S(\vec{u})+(1,1,1)$ x $S(\vec{u})$ which is $\vec{v}=\begin{pmatrix}x+y \\ z-x \\ -y-z \end{pmatrix}$ then take the scalar product between $\vec{v}$ and $S(\vec{u})$ for the angle (which is $90^\circ$) and rotate around the z-axis.
This gives me: $\begin{pmatrix} 0 &-1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{pmatrix}$
and my final answer would then be: $\begin{pmatrix} 0 &-1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{pmatrix}$ $\begin{pmatrix} 0 &0 & 1 \\ 0 & -1 & 0 \\ 1 & 0 & 0 \end{pmatrix}$= $\begin{pmatrix} 0 &1 & 0 \\ 0 & 0 & 1 \\ 1 & 0 & 0 \end{pmatrix}$ $\begin{pmatrix}x \\ y \\ z \end{pmatrix}$
Unfortunately I have no way of controlling my answer and I feel like I'm completely wrong.
1) why does S(u) do anything to the y, component in the reflection?
Where do the principle component vectors go?
Assume $u = (1,0,0)$, then $S(u) = (0,0,1)$
$(1,1,1)\times (0,0,1) = (1,-1,0)\\ S(u) + (1,1,1)\times S(u) = (1,-1,1)$
Your final matrix must take $u$ to $F(u)$
$\begin {bmatrix} 1\\-1\\1 \end{bmatrix}$ must be the first column of $F$
Now do this for the other components.
Alternatives...
$S(u)+ (1,1,1)\times S(u) = (I + (1,1,1)\times I)S(u)$
$\left(\begin{bmatrix} 1\\&1\\&&1\end{bmatrix} + \begin{bmatrix} 0&-1&1\\1&0&-1\\-1&1&0\end{bmatrix}\right)\begin{bmatrix} &&1\\&1\\1\end{bmatrix}$
Alternative 2.
$u = (x,y,z)\\ S(u) = (z,y,x)\\ (1,1,1)\times (z,y,x) = (x-y,z-x,y-z)\\ S(u)+(x-y,x-z,y-z) = (x-y+z, -x+y+z,x+y-z)$