$L(R^n,R^m)$ is the set of all linear transformations from $R^n$ to $R^m.$ For $A\in L(R^n,R^m),$ we define the norm $\|A\|$ of $A$ to be the $\sup$ of all numbers $|A\mathbf{x}|,$ where $\mathbf{x}$ ranges over all vectors in $R^n$ with $\left|\mathbf{x}\right|\le 1.$
It can be shown that
If $A\in L(R^n,R^m)$ and $B\in L(R^m,R^k),$ then $$\|BA\|\le\|B\|\|A\|.\tag{1}$$
So now suppose $\mathbf{A}\in L(R,R^n)$ then \begin{align} \|\mathbf{A}\| &=\sup \{\left|\mathbf{A}(x)\right|: |x|\le 1\}\\ &=\sup \{\left|\mathbf{A}(1\cdot x)\right|: |x|\le 1\}\\ &=\sup \{\left|x\cdot \mathbf{A}(1)\right| : |x|\le 1\}\\ &=\sup\{\left|\mathbf{A}(1)\right|\}\\ &= |\mathbf{A}(1)|. \tag{2} \end{align}
Following is theorem $9.19$ from Baby Rudin :
EDIT: I think the way I had asked the question earlier was quite confusing and hence the edit. Let me rephrase what's been bothering me :
$\bullet$ If $\mathbf{g'}(t) \in L(R,R^m) $ then why does Rudin write $\left|\mathbf{g'}(t)\right|$ instead of $\|\mathbf{g'}(t)\|$? Did he mean to say $\left|\mathbf{g'}(t)(\mathbf{x})\right|$?
$\bullet$ Similarly $\gamma '(t)= \mathbf{(b-a)}$ is in $L(R,R^n),$ again the interchange of $||$ and $\|\|$ is confusing!
As of now, this is how I made sense of it :
Assuiming $\left|\mathbf{g'}(t)\right| = \left|\mathbf{g'}(t)(x)\right|$ for all $x \in [0,1]$ and $\gamma '(t)$ is the linear transformation which maps $x\in [0,1]$ to the vector $x\mathbf{(b-a)}$ in $R^n$
Since $x\in [0,1]$ we have $\left|\mathbf{g'}(t)(x)\right|\le \|\mathbf{g'}(t)\|.$ Also $\|\mathbf{b-a}\|=\underbrace{\left|\mathbf{(b-a)}(1)\right|}_{\text{using } (2)}=\left|\mathbf{b-a}\right|$ Thus $$\left|\mathbf{g'}(t)(x)\right|\le \|\mathbf{g'}(t)\|=\|\mathbf{f'}(\gamma (t))\mathbf{(b-a)}\|\le\|\mathbf{f'}(\gamma (t))\|\cdot\|\mathbf{(b-a)}\|=\|\mathbf{f'}(\gamma (t))\|\cdot\left|\mathbf{b-a}\right|.$$ and finally
$$\left|\mathbf{g'}(t)(x)\right|\le \|\mathbf{f'}(\gamma (t))\|\cdot\left|\mathbf{b-a}\right|.$$
Have I done it right?
The operator norm he gives satisfies the following:
$$\vert T(x)\vert \leq \Vert T \Vert \, \vert x \vert \quad \text{for all }x$$
$\mathbf{b-a}$ is just a vector and is not a linear transformation here.
Let $T(\mathbf{x}):=\mathbf{f}'(\gamma(t)) \mathbf{\cdot x} $. The dot there refers to the dot product.
Now we have $|\mathbf{g}'(t)| = |\mathbf{f}'(\gamma(t))\, \cdot \mathbf{b-a}| = |T(\mathbf{b-a})| \leq \Vert T \Vert \, |\mathbf{b-a}| = \Vert \mathbf{f}'(\gamma(t)\Vert \, |\mathbf{b-a}|$