By a rotation in ${\bf R}^3$ I mean an orthogonal linear transformation $f:{\bf R}^3\to {\bf R}^3$ represented by a matrix $A$ (i.e. $fx=Ax$) with $\det A=1$. By a reflection (through $S$) I mean an orthogonal linear transformation such that for some subspace $S$, $f\mid S={\rm id}$ and $f\mid S^{\perp}=-{\rm id}$. I am currently stuck in two tasks:
$(1)$ Let $\mathscr L_1=\langle(1,1,0)\rangle+(2,0,1)$ and $\mathscr L_2=\langle (2,1,3)\rangle+(1,0,4)$. I have to find a rotation such that $f(\mathscr L_1)=\mathscr L_2$. Now, the distance of both lines to the origin is $\sqrt 3$. The first line accomplishes this with $(1,-1,1)$, while the second line accomplishes this with $(-1,-1,1)$. I tried various times to define a rotation, but I failed. In particular, I know I should map $(1,-1,1)$ to $(-1,-1,1)$.
$(2)$ Let $\Pi_1=\{(x_1,x_2,x_3):x_1-x_2+2x_3=k\}$ and $\Pi_2=\langle (1,0,1),(0,1,2)\rangle+(1,-1,1)$. I have to find $k$ such that there exists a reflection that maps $\Pi_1$ to $\Pi_2$ and find $f(\Pi_2)$. Now, the distance to $\Pi_1$ to the origin is $|k|/\sqrt 6$ and that of $\Pi_2$ is $3/\sqrt 6$ which gives me $k=3,-3$. I don't really know how to continue now.
(1) You found the orthogonal projections of the origin onto these lines. So we have $$ \mathcal{L}_1=p_0+\mathbb{R}u \quad\mathcal{L}_2=q_0+\mathbb{R}v\quad \mbox{with}\;p_0=\pmatrix{1\\-1\\1}\;u=\pmatrix{1\\1\\0}\;q_0=\pmatrix{-1\\-1\\1}\;v=\pmatrix{2\\1\\3} $$ We are looking for a rotation, that is a direct (determinant one) isometry $f$ such that $f(\mathcal{L}_1)=\mathcal{L}_2$ or, equivalently, $f(p_0)=q_0$ and $f(u)=\lambda v$. For such a rotation to exist, it is necessary and sufficient that $\|p_0\|=\|q_0\|$, which is the case here.
Let us normalize these vectors $$p_1=\pmatrix{1/\sqrt{3}\\-1/\sqrt{3}\\1/\sqrt{3}}\;\qquad u_1=\pmatrix{1/\sqrt{2}\\1/\sqrt{2}\\0}\;\qquad q_1=\pmatrix{-1/\sqrt{3}\\-1/\sqrt{3}\\1/\sqrt{3}}\;\qquad v_1=\pmatrix{2/\sqrt{14}\\1/\sqrt{14}\\3/\sqrt{14}} $$ By construction, $\{p_1,u_1\}$ and $\{q_1,v_1\}$ are pairs of orthogonal unit vectors. So $$ \{p_1,u_1,p_1\times u_1\}\quad\mbox{and}\quad \{q_1,v_1,q_1\times v_1\}\quad \mbox{where}\;p_1\times u_1=\pmatrix{-1/\sqrt{6}\\1/\sqrt{6}\\2/\sqrt{6}} \; q_1\times v_1=\pmatrix{-4/\sqrt{42}\\5/\sqrt{42}\\1/\sqrt{42}} $$ are two direct orthonormal bases. Therefore $$ f(p_1):=q_1\qquad f(u_1):=v_1\qquad f(p_1\times u_1):=q_1\times v_1 $$ defines the (not unique) direct isometry we are looking for. After a bit of tedious computations, we find the following matrix form for $f$ with respect to the canonical basis $$ f=\pmatrix{-\frac{1}{3}+\frac{5}{3\sqrt{7}} & \frac{1}{3}+\frac{1}{3\sqrt{7}} & -\frac{1}{3}-\frac{4}{3\sqrt{7}}\\ -\frac{1}{3}+\frac{1}{3\sqrt{7}}& \frac{1}{3}+\frac{4}{3\sqrt{7}} & -\frac{1}{3}+\frac{5}{3\sqrt{7}} \\ \frac{1}{3}+\frac{4}{3\sqrt{7}}& -\frac{1}{3}+\frac{5}{3\sqrt{7}} & \frac{1}{3}+\frac{1}{3\sqrt{7}}} $$
(2) In dimension $3$, reflections are isometries with eigenvalues $\{1,1,-1\}$.
Denote $\Pi_1=p+u^\perp$ and $\Pi_2=q+v^\perp$ where $$ p=\pmatrix{k\\0\\0} \quad u=\pmatrix{1\\-1\\2}\quad\qquad q=\pmatrix{1\\-1\\1}\quad v=\pmatrix{-1\\-2\\1} $$ where $v$ is the cross product of the two vectors spanning the linear plane in the definition of $\Pi_2$. The orthogonal projections of the origin onto $\Pi_1$ and $\Pi_2$ are $$ p_0=\pmatrix{k/6\\-k/6\\k/3}\qquad q_0=\pmatrix{-1/3\\-2/3\\1/3} $$
Note that now $$ \Pi_1=p_0+ p_0^\perp \qquad\mbox{and}\qquad \Pi_2=q_0+q_0^\perp $$ For an isometry $f$ to take $\Pi_1$ onto $\Pi_2$, it is necessary and sufficient that $$ f(p_0)=q_0\qquad \mbox{and}\qquad f(p_0^\perp)=q_0^\perp $$ Now note that a reflection $f$ is an involution and has $f^*=f=f^{-1}$. It follows that for a reflection $f$ $$ f(\Pi_1)=\Pi_2\quad\iff\quad f(\Pi_2)=\Pi_1 \quad \iff \quad f(p_0)=q_0 $$
Moreover, such an $f$ leaves the line $p_0^\perp\cap q_0^\perp$ invariant. This means that if we consider the vector $z:=p_0\times q_0$, we have $f(z)=\pm z$. But the case $f(z)=-z$ is impossible here since this would force $f$ to be the identity on $z^\perp$, whence $p_0=q_0$. So we must have $f(z)=z$.
For $f$ to exist, we need $\|p_0\|=\|q_0\|$, which amounts to $k=\pm 2$ in our case. It is also sufficient.
In any case, let us normalize our vectors $$ p_1:=\frac{p_0}{\|p_0\|}\quad p_1:=\frac{p_0}{\|p_0\|}\quad z_1:=\frac{p_0\times q_0}{\|p_0\times q_0\|} $$ and consider the following direct orthonormal bases $$ \{p_1,z_1,p_1\times z_1\}\qquad \mbox{and}\qquad \{q_1,z_1,q_1\times z_1\} $$ We are looking for a reflection which fixes $z_1$, and sends $p_1$ onto $q_1$. Drawing a picture in the plane $z_1^\perp$ where $p_1,q_1,p_1\times z_1, q_1\times z_1$ all lie, we see that we have no choice: we must have $f(p_1+q_1)=p_1+q_1$ and $f(p_1-q_1)=q_1-p_1$ whence $f(p_1\times z_1)=-q_1\times z_1$. And indeed $$ f(p_1):=q_1\qquad f(z_1):=z_1\qquad f(p_1\times z_1):=-q_1\times z_1 $$ does define the reflection we are looking for. And it is unique.
For $k=-2$, we get $$ p_1=\pmatrix{-1/\sqrt{6}\\1/\sqrt{6}\\-2/\sqrt{6}}\;q_1=\pmatrix{-1/\sqrt{6}\\-2/\sqrt{6}\\1/\sqrt{6}} \; z_1=\pmatrix{-1/\sqrt{3}\\1/\sqrt{3}\\1/\sqrt{3}} \; p_1\times z_1=\pmatrix{1/\sqrt{2}\\1/\sqrt{2}\\0} \;q_1\times z_1=\pmatrix{-1/\sqrt{2}\\0\\-1/\sqrt{2}} $$ So our reflection is $f(x,y,z)=(x,z,y)$.
For $k=2$, we get $$ p_1=\pmatrix{1/\sqrt{6}\\-1/\sqrt{6}\\2/\sqrt{6}}\;q_1=\pmatrix{-1/\sqrt{6}\\-2/\sqrt{6}\\1/\sqrt{6}} \; z_1=\pmatrix{1/\sqrt{3}\\-1/\sqrt{3}\\-1/\sqrt{3}} \; p_1\times z_1=\pmatrix{1/\sqrt{2}\\1/\sqrt{2}\\0} \;q_1\times z_1=\pmatrix{1/\sqrt{2}\\0\\1/\sqrt{2}} $$ In the canonical basis, we find that the matrix of $f$ is
$$ f=\pmatrix{-1/3&-2/3&-2/3\\-2/3&2/3&-1/3\\-2/3&-1/3&2/3} $$