As Timbuc mentions this probably means the absolute value of the inner product of the vectors.
However, the inner product can mean different things in different contexts - the point of an inner product is to associate any two vectors in some space with a scalar in a nice manner (specifically this manner). Without further information, and if you are working with vectors in a Euclidean space, the inner product probably refers to the dot product between the two vectors. As a reminder, if the two vectors are $A=(a_1,a_2,\dots,a_n)$ and $B=(b_1,b_2,\dots,b_n)$, then the dot product is defined as $a_1b_1+a_2b_2+\dots+a_nb_n$, so in this case we would have
As Timbuc mentions this probably means the absolute value of the inner product of the vectors.
However, the inner product can mean different things in different contexts - the point of an inner product is to associate any two vectors in some space with a scalar in a nice manner (specifically this manner). Without further information, and if you are working with vectors in a Euclidean space, the inner product probably refers to the dot product between the two vectors. As a reminder, if the two vectors are $A=(a_1,a_2,\dots,a_n)$ and $B=(b_1,b_2,\dots,b_n)$, then the dot product is defined as $a_1b_1+a_2b_2+\dots+a_nb_n$, so in this case we would have
$$|\langle A,B\rangle|=|a_1b_1+a_2b_2+\dots+a_nb_n|$$