I will omit the source of the problem to avoid the possibility of an outright answer to a book exercise.
The problem is to prove that the matrix of the shape operator for basis vectors $r_u, r_v$ is $\rm I^{-1} II.$ This is supposed to be a straightforward calculation based on the fact that the shape operator is a linear map from the tangent space to itself.
However, I am stuck in the most unfruitful and error-prone expansions trying to get some simplification based on
- The shape operator is the negative of the directional derivative with respect to a tangent a vector of the normal vector to the surface, and therefore, $S(r_u)=-n_u$ and $S(r_v)=-n_v.$
- I can express the second differential form entries as either $N\cdot r_{uu}$ or $-N_u\cdot r_u$
$$\rm I^{-1} II= \frac{1}{EG-F^2}\begin{bmatrix} G & -F \\ -F & E\end{bmatrix}\begin{bmatrix} e & f \\ f & g\end{bmatrix}$$
Replacing each element in these two matrix by dot products of tangent basis vectors among themselves and between them an the partial derivatives of the normal vector to the surface results in a nasty expansion that doesn't seem to simplify, and that it is not worth typing here.
What is the way of proving this is true maybe based on the linearity of the shape operator?
I don't see a shorter way than the "nasty expansion" I was referring to with some matching at the end that makes it less tedious.
The exercise suggests starting with the premise that the shape operator $S_P$ (i.e. at a point $P$), defined as
$$S_P(\vec v) = - D_\vec v \vec N(P)$$
which reads: the negative of the directional derivative of the normal vector to the surface at point $P,$ i.e. $\vec N(P)$ in the direction of a velocity (tangent vector) along a curve as it passes through the point $P.$ Since a plane is defined by its normal, and $T_P\cal M$ (the tangent plane) changes with differential displacements on the surface in the direction of the vector $\vec v,$ the shape operator is a very intuitive way of capturing the bending of the surface as it measures the differential tilting of $\vec N$ in the direction of $\vec v.$
This leads to my point $1$ in the OP, which is simply that for the basis vectors $\{{\bf r}_u, {\bf r}_v\}$ (or as in the exercise $\{{\bf x}_u, {\bf x}_u\}$) the shape operators can simply expressed as $-{\bf N}_u$ and $-{\bf N}_v.$
Since this shape operator is a linear transformation from $T_P{\cal M} \to T_P \cal M,$ the exercise suggests starting off as in the comment by Professor Shifrin's above - i.e. considering that
$$\begin{align} S_P({\bf x}_u)= -{\bf N}_u = a {\bf x}_u + b{\bf x}_v\\ S_P({\bf x}_v) = -{\bf N}_v = c {\bf x}_u + d{\bf x}_v \end{align}\tag 1$$
and expressing it in matrix form:
$$\begin{bmatrix}T({\bf x}_u) & T({\bf x}_v)\end{bmatrix}\equiv\begin{bmatrix}a & c\\ b & d \end{bmatrix} \tag 2 $$
and then considering the definition of the entries of the second fundamental form $l$, $m$ and $n.$ That is the cue for point $2$ in the OP: The entries of the second fundamental form can be expressed as two different dot products:
$${\bf II}=\begin{bmatrix} {\bf N} \cdot{\bf x}_{uu} & {\bf N} \cdot {\bf x}_{uv} \\ {\bf N} \cdot {\bf x}_{uv} & {\bf N} \cdot {\bf r}_{vv}\end{bmatrix}=\begin{bmatrix}- {\bf N}_u \cdot{\bf x}_{u} & - {\bf N}_v \cdot {\bf x}_{u} \\ - {\bf N}_u \cdot {\bf x}_{v} & -{\bf N}_v \cdot {\bf x}_{v}\end{bmatrix} \tag 3$$
which follows from the fact that the tangent plane is orthogonal to ${\bf N}$ and hence that dotting any of the basis vectors with is is zero, and then applying the product rule.
Now to match $(1),$ $(2)$ and $(3),$ instead of the dead-end in the OP, I can just dot both sides of $(1)$ by the basis vectors:
\begin{align} -{\bf N}_u {\bf x}_u &= a{\bf x}_u {\bf x}_u+ b{\bf x}_v {\bf x}_u\\ -{\bf N}_u {\bf x}_v &= a{\bf x}_u {\bf x}_v+ b{\bf x}_v {\bf x}_v\\ -{\bf N}_v {\bf x}_u&= c{\bf x}_u {\bf x}_u+ d{\bf x}_v {\bf x}_u\\ -{\bf N}_v {\bf x}_v&= c{\bf x}_u {\bf x}_v + d{\bf x}_v {\bf x}_v \end{align}
which would then be collected as:
$${\rm II}=\begin{bmatrix}- {\bf N}_u \cdot{\bf x}_{u} & - {\bf N}_v \cdot {\bf x}_{u} \\ - {\bf N}_u \cdot {\bf x}_{v} & -{\bf N}_v \cdot {\bf x}_{v}\end{bmatrix} = \begin{bmatrix}{\bf x}_{u} \cdot{\bf x}_{u} & {\bf x}_{v} \cdot {\bf x}_{u} \\ {\bf x}_{u} \cdot {\bf x}_{v} & {\bf x}_{v} \cdot {\bf x}_{v}\end{bmatrix} \;\begin{bmatrix}a & c\\ b &d \end{bmatrix}$$