In the paper http://www.goshen.edu/physix/mathphys/gco/TensorGuideAJP.pdf at page 499, the authors defined the dot product of two vectors in a weird way: In the 3D Euclidian space let $\textbf{A}$ and $\textbf{B}$ two vectors and let ${\{\hat{x_a}:a=1,2,3\}}$ the orthonormal basis vectors. $\textbf{A}$ and $\textbf{B}$ are defined as: $$ \textbf{A}=\sum_{a=1}^3A_a\hat{x_a} \;\;\;\;\;\;\;\;\;\; \textbf{B}=\sum_{b=1}^3B_b\hat{x_b} $$
They defined the dot product of $\textbf{A}$ and $\textbf{B}$ as: $$\textbf{A} \cdot \textbf{B} = \sum_{a=1}^3\sum_{b=1}^3(A_a\hat{x_a})\cdot (B_b\hat{x_b})=\sum_{a=1}^3\sum_{b=1}^3(A_aB_b)(\hat{x_a}\cdot \hat{x_b})=\sum_{a=1}^3\sum_{b=1}^3(A_aB_b)\delta_{ab}=\sum_{a=1}^3A_aB_a$$ Where $\delta_{ab}$ is the Kronecker delta.
As far as I know the dot product is only defined as the multiplication of the components belonging to the same axes and then summing them. But in this definition dot product is shown as the regular, element wise multiplication of $(A_1\hat{x_1}+A_2\hat{x_2}+A_3\hat{x_3})$ and $(B_1\hat{x_1}+B_2\hat{x_2}+B_3\hat{x_3})$. So, what is the actual definition of dot product here? Which one is correct?
All of the paper is using the usual dot product.
It starts with the two vectors being expressed as a linear combination of some basis vectors, and applies the calculation rules of the dot product. $X=aU+bV$ and $Y=u'U'+b'V'$ can be multiplied together: $$X\cdot Y=(uU+vV)\cdot(u'U'+v'V')= uu'U\cdot U'+uv'U\cdot V'+vu'V\cdot U'+vv'U'\cdot V',$$ by distributivity of the dot product over vector addition, and associativity of the scalar product.
When the basis vectors are biorthogonal (hence the Kronecker delta), this simplifies to $uu'+vv'$.
In particular, when $U=U'=1_x$ and $V=V'=1_y$, you find the familiar $A\cdot B=a_xb_x+a_yb_y.$