I'm struggling with bivector wedge bivector. If i do via the formula $$A\wedge B=\frac12(AB-BA)$$ I get the correct answer but i can't do it directly. For Example, $$A=\hat i\hat j+\hat j\hat k+\hat k\hat i$$ $$B=\hat i\hat j+2\hat j\hat k+3\hat k\hat i$$
$$A\wedge B=(\hat i\hat j+\hat j\hat k+\hat k\hat i)\wedge (\hat i\hat j+2\hat j\hat k+3\hat k\hat i)$$ $$A\wedge B=(\hat i\wedge\hat j+\hat j\wedge\hat k+\hat k\wedge\hat i)\wedge (\hat i\wedge\hat j+2\hat j\wedge\hat k+3\hat k\wedge\hat i)$$
$$A\wedge B=\hat i\wedge\hat j\wedge\hat i\wedge\hat j+2\hat i\wedge\hat j\wedge\hat j\wedge\hat k+3\hat i\wedge\hat j\wedge\hat k\wedge\hat i+$$ $$\hat j\wedge\hat k\wedge\hat i\wedge\hat j+2\hat j\wedge\hat k\wedge\hat j\wedge\hat k+3\hat j\wedge\hat k\wedge\hat k\wedge\hat i+$$ $$\hat k\wedge\hat i\wedge\hat i\wedge\hat j+2\hat k\wedge\hat i\wedge\hat j\wedge\hat k+3\hat k\wedge\hat i\wedge\hat k\wedge\hat i$$
Since $a\wedge a$ is $0$, $A\wedge B$ is also $0$
But, $$AB=(\hat i\hat j+\hat j\hat k+\hat k\hat i)(\hat i\hat j+2\hat j\hat k+3\hat k\hat i)$$
$$AB=\hat i\hat j\hat i\hat j+2\hat i\hat j\hat j\hat k+3\hat i\hat j\hat k\hat i+$$ $$\hat j\hat k\hat i\hat j+2\hat j\hat k\hat j\hat k+3\hat j\hat k\hat k\hat i+$$ $$\hat k\hat i\hat i\hat j+2\hat k\hat i\hat j\hat k+3\hat k\hat i\hat k\hat i$$
$$AB=-1-2\hat k\hat i+3\hat j\hat k+$$ $$\hat k\hat i-2-3\hat i\hat j+$$ $$-\hat j\hat k+2\hat i\hat j-3$$
$$AB=-6-\hat i\hat j+2\hat j\hat k-\hat k\hat i$$ Similarly, $$BA=-6+\hat i\hat j-2\hat j\hat k+\hat k\hat i$$ So $A\wedge B$ is $-\hat i\hat j+2\hat j\hat k-\hat k\hat i$
What did i do wrong in the first example?
Probably the second result is right because bivector is a pseudovector in 3d and $IA$ is a vector where $I=\hat i\hat j\hat k$. Since vectro wedge vector is a bivector, $$IA\wedge IB=I^2A\wedge B [Since I^2=-1]$$ $$IA\wedge IB=-A\wedge B=B\wedge A$$ which implies bivector wedge bivector is also a bivector
$ \newcommand\proj[1]{\langle#1\rangle} $
Because your formula $$ A\wedge B = \frac12(AB - BA) $$ is incorrect if $A, B$ are bivectors. This just isn't how the wedge product works. In fact, it can't work like this: even-grade multivectors commute with all other multivectors under the wedge product, precisely because they are made up of an even number of vectors.
The first way you computed the wedge product which resulted in $A\wedge B = 0$ is correct, and in fact we must get $0$ since $A$ and $B$ live in the same 3D space which can't have a 4D component (i.e. a 4-vector).
What is true is that if $v$ is a vector and $X$ is any multivector then $$ v\wedge X = \frac12(vX + \hat Xv) $$ where $\hat X$ is the grade involution of $X$, i.e. odd-grade components are negated. We also get the reversed formula $$ X\wedge v = \frac12(Xv + v\hat X). $$ It follows that if $A_r$ is homogeneous with odd grade $r$ then $$ v\wedge A_r = \frac12(vA_r - A_rv). $$ In particular for vectors $v,w$ we have the formula you were likely trying to mimic $$ v\wedge w = \frac12(vw - wv). $$
What we could do for bivectors, if you were so inclined, is note that $$ AB = \proj{AB}_0 + \proj{AB}_2 + \proj{AB}_4, $$ from which it follows that $$ \frac12(AB + BA) = \proj{AB}_0 + \proj{AB}_4 = A\cdot B + A\wedge B, $$$$ \frac12(AB - BA) = \proj{AB}_2 = A\times B. $$ We see that $\frac12(AB-BA)$ isn't even related to the wedge product at all. But we should expect this: even-grade multivectors commute under the wedge product. What we can say for bivectors $A$ and $B$ clearly reflects this: $$ A\wedge B = \frac12(AB + BA) - A\cdot B. $$