So, I've been working my way through the following as I'm messing about with programming some helper functions for orbital mechanics: https://downloads.rene-schwarz.com/download/M002-Cartesian_State_Vectors_to_Keplerian_Orbit_Elements.pdf
Given I don't have the greatest understanding of mathematical notation I've run into a spot of bother understanding what $||r||$ in the following formula means:
$\frac{r}{||r||}$
Keep in mind that r is a vector from element 1.b
This is too long to put in a comment, so I am giving it as an answer, though Lubin already answered your actual question and this is in response to your comments.
Your vectors are column vectors, so technically, they should be written as $$\begin{bmatrix} 0 \\ 0 \\ 1\end{bmatrix} \times h = \begin{bmatrix} -h_y \\ h_x \\ 0\end{bmatrix}$$
But they wanted to write it on just one line, so they used the transpose operator $$\begin{bmatrix} 0 \\ 0 \\ 1\end{bmatrix}^T = \begin{bmatrix} 0 & 0 & 1\end{bmatrix} \qquad\text{and}\qquad\begin{bmatrix} 0 & 0 & 1\end{bmatrix}^T = \begin{bmatrix} 0 \\ 0 \\ 1\end{bmatrix}$$
to represent the column vector as row vectors instead.
The operation "$\times$" here is the vector cross-product. It is defined by: $$\begin{bmatrix} a \\ b \\ c\end{bmatrix} \times \begin{bmatrix} x \\ y \\ z\end{bmatrix} = \begin{bmatrix} bz - cy \\ cx - az \\ ay - bx\end{bmatrix}$$
You can check yourself that if $h = \begin{bmatrix} h_x & h_y & h_z\end{bmatrix}^T$, then the cross-product really is as given.
An important fact about the cross-product is that it is orthogonal to (i.e. makes a right angle with) both of the multiplicands. That is, $n$ will be perpendicular to both $\begin{bmatrix} 0 & 0 & 1\end{bmatrix}^T$ and to $h$, which is doubtless why they are making use of it here. They want something perpendicular to both directions.
$\langle e, r\rangle$ is the inner product, or dot product of the two vectors $e$ and $r$. The second name is because it is also common to represent it as $e \cdot r$ instead of $\langle e, r\rangle$. It is defined by $$\langle e, r\rangle = e^Tr = e_xr_x + e_yr_y + e_zr_z$$ Note that unlike the cross-product, the inner product is a number, not a vector.
The inner product has two uses that are particularly important:
In particular, for your vectors above, $\langle n, h\rangle =0$ and $\langle n, \begin{bmatrix} 0 & 0 & 1\end{bmatrix}^T\rangle = 0$, as you can easily verify by plugging into the formulas.