Let C be a cube in $R^3$, $C=\{(x,y,z): 0\leq x,y,z,\leq 1\}$.
Find a reflection of a diagonal of a face with respect to a plane orthogonal to main diagonal.
I am trying to study Vector Calculus by myself and do not have a proper explanation of this question. I have checked a Wikipedia page, but there is no formulas on reflections... Please, give ma an idea how to solve this question.
It’s probably easiest to tackle this in two stages. First, find a formula for a reflection in a plane through the origin, then extend this to any parallel plane.
As explained in the article linked in the comments, if $\mathbf n$ is the normal to a plane through the origin, then the reflection of a vector $\mathbf v$ relative to this plane can be found by reversing the component of $\mathbf v$ parallel to $\mathbf n$, i,e, $$ R_0\mathbf v = \mathbf v - 2\pi_{\mathbf n}\mathbf v = \mathbf v - 2{\mathbf n\cdot\mathbf v\over\mathbf n\cdot\mathbf n}\mathbf n = \left(1-{\mathbf n\mathbf n^T\over\mathbf n^T\mathbf n}\right)\mathbf v.\tag{*}$$ An arbitrary plane with the same normal is just this plane through the origin shifted by some multiple $\alpha\mathbf n$ of the normal, so the general reflection can be accomplished by translating to the origin, reflecting, and then translating back: $$ R_\alpha\mathbf v = R_0(\mathbf v-\alpha\mathbf n)+\alpha\mathbf n = R_0\mathbf v+2\alpha\mathbf n$$ since $R_0\mathbf n=-\mathbf n$.
I’ll leave it to you to work out how to apply this to the endpoints of the cube’s diagonals. You could represent $R_\alpha$ by a single $4\times4$ matrix by passing to homogeneous coordinates, but that’s not really necessary here since equation (*) tells us that if we take $\mathbf n=(1,1,1)$ then we can compute the general reflection by reflecting relative to the plane through the origin and then adding $2\alpha$ to every coordinate.