\begin{array}{l} ( 5-1) \ From\ the\ question,\ the\ normal\ vector\ of\ plane\ x+y+z=0,\ \vec{u} =( 1,1,1)^{T}\\ According\ to\ Householder\ Transformation,\ g=I-2\vec{u}\vec{u} =\begin{pmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & 0 & 1 \end{pmatrix} -\begin{pmatrix} 2\\ 2\\ 2 \end{pmatrix}\begin{pmatrix} 1 & 1 & 1 \end{pmatrix} =\begin{pmatrix} -1 & -2 & -2\\ -2 & -1 & -2\\ -2 & -2 & -1 \end{pmatrix} \end{array}
For (5-2), I don't understand what the image is. And how to relate the composition transformation with the spherical surface? Thanks for your help.
You must have been warned by obtaining the unity matrix for the matrix of a symmetry...
Your error comes from the fact that Householder formula $I-2NN^T$ is valid for a unit(arized) vector $N$. So take $N=(1,1,1)/\sqrt{3}$ instead of $N=(1,1,1)$.
Now, what can you expect for the final result ?
Inspection of the diagonal blocks of the first matrix make it an "elongator" by factor $k$ in the $x$ direction, and a "rotator" around this same $x$ direction. Therefore, the image of the unit sphere is an ellipsoid. with semi-axes $k,1,1$. Composition by a symmetry will not change the general shape (still an ellipsoid with the same semi-axes).