According to one of the basic axioms of geometric algebra, the square of a vector with itself is a scalar. For two vectors $a$ and $b$, this results in $ab+ba = (a+b)^2-a^2-b^2$. Therefore $ab+ba$ must be a scalar. When then "define"
$$ a \cdot b = \frac{1}{2}(ab+ba)$$
The dot product is connected to geometry because it relates to the cosine of the angle between the vectors $a$ and $b$. Could we have just as easily "defined" the dot product to be something even more complicated, like $ab+ba = 3.5\lVert a\rVert \lVert b\rVert \log(\cos(\sin([\textrm{angle between} a \textrm{ and } b])))$ (perhaps notwithstanding the infinity in the logarithm)? This formula also respects the fact that $ab+ba$ is a scalar. Is the cosine relationship actually required?
A geometric algebra is a structure as follows: You have a vector space $V$, then you have an inner product (i.e. a non-degenerate, symmetric bilinear form) $V\ni a,b\mapsto a\cdot b$, and then the geometric algebra is some algebra $A$ that contains scalars in $\mathbb{C}$ and vectors in $V$ (and bi-vectors and tri-vectors etc.), such that for $a,b\in V$ we have
We also have:
Now we can derive the conclusion: Let $a,b\in V$ be two vectors. We can do two things:
So using the inner product, we have $(a+b)\cdot(a+b) = a\cdot a + a\cdot b + b\cdot a + b\cdot b$, where everything is a scalar. Furthermore, $b\cdot a = a\cdot b$ by symmetry. On the other hand, using the algebra product we have $(a+b)(a+b) = aa + ab + ba + bb$, where $ab$ and $ba$ are not scalars, but bi-vectors.
Using that $aa=a\cdot a1_\mathbb{C}$, we obtain that $$a\cdot a1_\mathbb{C}+b\cdot b1_\mathbb{C}+2a\cdot b1_\mathbb{C}=(a+b)\cdot(a+b)1_\mathbb{C} $$$$= (a+b)(a+b)=aa + ab + ba + bb = a\cdot a1_\mathbb{C}+b\cdot b1_\mathbb{C} + ab+ba$$ Therefore $$a\cdot b1_\mathbb{C} = \frac 12(ab+ba).$$
This follows from the fact that $\cdot$ is an inner product.
When you replace $\cdot$ with any other function that is bilinear and symmetic, everything still works out. However, your funny function is not bilinear, so thing will not work out as smoothly.