I'm reading through a book on numerical analysis right now and in its treatment of linear iterative method the author takes the norm of a matrix with respect to another matrix but never seems to say what this actually means.
For a little more context we are dealing with iterative methods of the form $x^{(k + 1)} = Bx^{(k)} + f$ to solve $Ax = b$, then splitting $A = P - N$ where $P$ is suitably nice and $B = P^{-1}N$. Then the author makes guarantees on the spectral radius of $B$ given certain criteria on $A$ and $P$ and states during this:
$$\rho(B) = \Vert B \Vert_A = \Vert B \Vert_P < 1$$
Perhaps it is the induced norm of $\Vert \cdot \Vert_A$, ie.
$$\Vert B \Vert_A = \sup_{\Vert x \Vert_A = 1} \frac{(Bx)^TA(Bx)}{x^TAx}?$$
What does $\Vert \cdot \Vert_A$ with respect to another matrix mean?
This is called an "elliptical" norm, or the Euclidean norm relative to the scalar product defined by $A$. Which obviously demands that $A$ is s.p.d.
The unit sphere of the $A$ norm is an ellipsoid in standard Euclidean coordinates, thus the name.
Note that your formula defines $\|B\|_A^2$.