In this video (check it out, it's worth it), F. Holmér nicely derives the dot and cross product (with some insights into quaternions, the wedge product and much more), just by using ordinary multiplication and two definitions
- $i^2=-1$
- ${\bf{v}}^2 = \mathbf{vv}=\|{\bf{v}}\|^2$
Given the abundance of the results that follow, one is tempted to accept them without much questioning. And yet, I'm not entirely convinced of point 2.
It seems an ordinary dot product, but using the dot product to derive (among other things) itself seems a circular argument. Is there hence some other principle underneath?
Point 2 is an axiom, used to define vector multiplication. I wouldn't say that this is used to define the dot product. Instead, one must already have a definition of the dot product, specifically, one must know what the value is for the dot products between any basis vectors (such a set of dot products for the basis vectors, usually arranged in matrix form, is called the metric of the space.)
Once this is done, it is possible to see that a product of two vectors, subject to all the appropriate axioms, does have a dot product component. This isn't really defining the dot product that way, but instead seeing the dot product as a term in a derived result.
Here's an outline of the argument. Our assumptions are that we have an algebra, that is a vector space equipped with a bilinear product, and that for our product, if $ \mathbf{a}, \mathbf{b}, \mathbf{c} $ are vectors, we require
- $ \mathbf{a} (\mathbf{b} \mathbf{c}) = (\mathbf{a} \mathbf{b}) \mathbf{c} $, and
- $ \mathbf{a}^2 = \mathbf{a} \cdot \mathbf{a} $ (the contraction axiom),
as well as all the usual vector space properties (products are associative with respect to addition, zero element, ...)Suppose that $ \mathbf{e}_1, \mathbf{e}_2 $ are unit vectors, then $$\begin{aligned}\left( { \mathbf{e}_1 + \mathbf{e}_2 } \right)^2&= \left( { \mathbf{e}_1 + \mathbf{e}_2 } \right) \left( { \mathbf{e}_1 + \mathbf{e}_2 } \right) \\ &= \mathbf{e}_1 \mathbf{e}_1 + \mathbf{e}_1 \mathbf{e}_2 + \mathbf{e}_2 \mathbf{e}_1 + \mathbf{e}_2 \mathbf{e}_2 \\ &= 1 + \mathbf{e}_1 \mathbf{e}_2 + \mathbf{e}_2 \mathbf{e}_1 + 1,\end{aligned}$$ but by 2, we also have $ \left( { \mathbf{e}_1 + \mathbf{e}_2 } \right)^2 = \left( { \mathbf{e}_1 + \mathbf{e}_2 } \right) \cdot \left( { \mathbf{e}_1 + \mathbf{e}_2 } \right) = 2 $, so $$2 + \mathbf{e}_1 \mathbf{e}_2 + \mathbf{e}_2 \mathbf{e}_1 = 2,$$ or $$\mathbf{e}_2 \mathbf{e}_1 = -\mathbf{e}_1 \mathbf{e}_2.$$ Clearly, this argument can be repeated with any set of indexes $ i \ne j $, so we can conclude that for orthonormal vectors $ \mathbf{e}_i \ne \mathbf{e}_j $ we have $$\mathbf{e}_j \mathbf{e}_i = -\mathbf{e}_i \mathbf{e}_j.$$
Now let's look at a product of vectors (in $\mathbb{R}^{3}$), say $$\begin{aligned}\mathbf{a} &= \sum_{i = 1}^3 a_i \mathbf{e}_i \\ \mathbf{b} &= \sum_{j = 1}^3 a_j \mathbf{e}_j,\end{aligned}$$ for which we have $$\begin{aligned}\mathbf{a} \mathbf{b}&=\left( { \sum_{i = 1}^3 a_i \mathbf{e}_i } \right) \left( { \sum_{j = 1}^3 a_j \mathbf{e}_j } \right) \\ &=\sum_{i = j} a_i b_j \mathbf{e}_i \mathbf{e}_j+\sum_{i \ne j} a_i b_j \mathbf{e}_i \mathbf{e}_j \\ &=\sum_{i=1}^3 a_i b_i \mathbf{e}_i \mathbf{e}_i+\sum_{i < j} a_i b_j \mathbf{e}_i \mathbf{e}_j+\sum_{i > j} a_i b_j \mathbf{e}_i \mathbf{e}_j \\ &=\sum_{i=1}^3 a_i b_i+\sum_{i < j} a_i b_j \mathbf{e}_i \mathbf{e}_j+\sum_{j < i} a_j b_i \mathbf{e}_j \mathbf{e}_i.\end{aligned}$$ We have a scalar term, clearly the dot product, as one part of this vector product, and a set of terms that all have vector products. We can make use of the identity $ \mathbf{e}_j \mathbf{e}_i = -\mathbf{e}_i \mathbf{e}_j $, to group the vector product terms above, to find $$\mathbf{a} \mathbf{b}= \mathbf{a} \cdot \mathbf{b} + \sum_{i < j} \left( { a_i b_j - a_j b_i } \right) \mathbf{e}_i \mathbf{e}_j.$$ By virtue of having been lazy with the index bounds in the summation above, note that this decomposition actually holds for any Euclidean vector space, not just $\mathbb{R}^{3}$.
We can introduce a notation for this second non-scalar term, and call it the wedge product, designated $$\mathbf{a} \wedge \mathbf{b}= \sum_{i < j}\begin{vmatrix}a_i & a_j \\ b_i & b_j \\ \end{vmatrix}\mathbf{e}_i \mathbf{e}_j,$$ and after doing so, we can write $$\mathbf{a} \mathbf{b} = \mathbf{a} \cdot \mathbf{b} + \mathbf{a} \wedge \mathbf{b}.$$ What we've really done here is decompose a product of two vectors into it's scalar and bivector parts, and not define the dot product in terms of the geometric product.
The relation to the $\mathbb{R}^{3}$ cross product comes by observing that we can factor out a quantity $ I = \mathbf{e}_1 \mathbf{e}_2 \mathbf{e}_3 $ from the wedge product, using $$\mathbf{e}_1 \mathbf{e}_2=\mathbf{e}_1 \mathbf{e}_2 \mathbf{e}_3 \mathbf{e}_3 \\ =I \mathbf{e}_3,$$ $$\mathbf{e}_2 \mathbf{e}_3=\mathbf{e}_1 \mathbf{e}_1 \mathbf{e}_2 \mathbf{e}_3=\mathbf{e}_1 I,$$ and $$\mathbf{e}_1 \mathbf{e}_3=\mathbf{e}_1 \mathbf{e}_2 \mathbf{e}_2 \mathbf{e}_3=-\mathbf{e}_2 \mathbf{e}_1 \mathbf{e}_2 \mathbf{e}_3=-\mathbf{e}_2 I.$$ It's not hard to show that $ I $ commutes with any vector, so we find
$$\begin{aligned}\mathbf{a} \wedge \mathbf{b}&= \sum_{i < j}\begin{vmatrix}a_i & a_j \\ b_i & b_j \\ \end{vmatrix}\mathbf{e}_i \mathbf{e}_j \\ &=\begin{vmatrix}a_1 & a_2 \\ b_1 & b_2 \\ \end{vmatrix}\mathbf{e}_1 \mathbf{e}_2+\begin{vmatrix}a_1 & a_3 \\ b_1 & b_3 \\ \end{vmatrix}\mathbf{e}_1 \mathbf{e}_3+\begin{vmatrix}a_2 & a_3 \\ b_2 & b_3 \\ \end{vmatrix}\mathbf{e}_2 \mathbf{e}_3 \\ &=\begin{vmatrix}a_1 & a_2 \\ b_1 & b_2 \\ \end{vmatrix}I \mathbf{e}_3-\begin{vmatrix}a_1 & a_3 \\ b_1 & b_3 \\ \end{vmatrix}I \mathbf{e}_2+\begin{vmatrix}a_2 & a_3 \\ b_2 & b_3 \\ \end{vmatrix}I \mathbf{e}_1 \\ &=I \left( { \mathbf{a} \times \mathbf{b} } \right).\end{aligned}$$
This lets us decompose the $\mathbb{R}^{3}$ vector product into dot product and cross product terms $$\mathbf{a} \mathbf{b} = \mathbf{a} \cdot \mathbf{b} + I \left( { \mathbf{a} \times \mathbf{b} } \right).$$
This isn't defining the dot or cross products, but just identifying them as constituents of the vector product, provided you define that vector product using the contraction axiom.
That said, some geometric algebra authors will treat the decomposition $ \mathbf{a} \mathbf{b} = \mathbf{a} \cdot \mathbf{b} + \mathbf{a} \wedge \mathbf{b} $ as fundamental, as opposed to derived, and then flip it around and define the dot and wedge products in terms of this identity. That's done by making note of the antisymmetry of the wedge product to write $$\mathbf{b} \mathbf{a} = \mathbf{a} \cdot \mathbf{b} - \mathbf{a} \wedge \mathbf{b},$$ and then rearrange to find $$\mathbf{a} \cdot \mathbf{b} = \frac{1}{{2}} \left( { \mathbf{a} \mathbf{b} + \mathbf{b} \mathbf{a} } \right),$$ and $$\mathbf{a} \wedge \mathbf{b} = \frac{1}{{2}} \left( { \mathbf{a} \mathbf{b} - \mathbf{b} \mathbf{a} } \right).$$ These are both true identities, but should really be treated as consequences of the contraction axiom $ \mathbf{a}^2 = \mathbf{a} \cdot \mathbf{a} $. When they are treated as fundamental defining properties, it leads to a great deal of confusion.