I'm given a non-orthogonal basis system $\vec{u}, \vec{v}, \vec{t}$ and i need to express linearly the following vector $\vec{u} \times \vec{v}$ as a linear combination of the basis vectors. So assuming $ \vec{u} \times \vec{v} = \alpha_1\vec{u} + \alpha_2\vec{v} + \alpha_3\vec{t}$ and using the properties of the cross product I get the system of equations:
$\begin{cases} \alpha_1\vec{u}^2 + \alpha_2\vec{u}\vec{v} + \alpha_3\vec{u}\vec{t} & = & 0 \\ \alpha_1\vec{u}\vec{v} + \alpha_2\vec{v}^2 + \alpha_3\vec{v}\vec{t}& =& 0\\ (\alpha_1\vec{u} + \alpha_2\vec{v} + \alpha_3\vec{t})^2 & = & (\vec{u}\vec{v})^2\sin^2{(\vec{u}, \vec{v})} \end{cases}$
which seems really hard to solve. Is there any other way i can solve this?
This problem is indeed really hard. In order to simplify let me suppose $\vec{u}$, $\vec{v}$, and $\vec{t}$ are at least unit vectors, and that they form a right-handed basis. Let us note $$\alpha=(\vec{v},\vec{t});~~~\beta=(\vec{t},\vec{u});~~~\gamma=(\vec{u},\vec{v})$$ the angles between the vectors of the basis.
Then let $\vec{t'}$ be a unit vector of same direction and sense than $\vec{u}\times\vec{v}$, and let $\vec{v'}$ be a unit vector in the same plane than $\vec{u}$ and $\vec{v}$ such that $\vec{u}$, $\vec{v'}$, $\vec{t'}$ is orthonormal. Note that as $\vec{t'}$ is colinear to $\vec{u}\times\vec{v}$ it is orthogonal to the plane containing $\vec{u}$ and $\vec{v}$ so that $\vec{v'}=\vec{t'}\times\vec{u}$ exists. Finally let $\delta$ be the angle $(\vec{t},\vec{t'})$.
Now that notations are set let me express $\vec{t}$ in the basis $\vec{u}$, $\vec{v'}$, $\vec{t'}$. As it is orthonormal we can use the scalar product: $$\vec{t}\cdot\vec{u}=\cos\beta;~~~\vec{t}\cdot\vec{t'}=\cos\delta$$ Finding $\vec{t}\cdot\vec{v'}$ is a bit more difficult: we need to express $v'$. As it is in the same plane than $\vec{u}$ and $\vec{v}$, as it is orthogonal to $\vec{u}$, and as $\alpha=(\vec{u},\vec{v})$, we get $$v=\cos\gamma\vec{u}+\sin\gamma\vec{v'}$$ But $\sin\gamma\neq 0$ as $\vec{u}$ and $\vec{v}$ are not colinear. Thus, $$\vec{v'}=\frac{1}{\sin\gamma}\vec{v}-\frac{\cos\gamma}{\sin\gamma}\vec{u}$$ giving $$\vec{t}\cdot\vec{v'}=-\frac{\cos\gamma\cos\beta}{\sin\gamma}+\frac{\cos\alpha}{\sin\gamma}$$ and $$\vec{t}=\cos\beta\vec{i}+\left(\frac{\cos\alpha-\cos\gamma\cos\beta}{\sin\gamma}\right)\vec{v'}+\cos\delta\vec{t'}~~~~(1)$$ This equality between vectors means equality of their squared norm:$$1=\cos^2\beta+\left(\frac{\cos\alpha-\cos\gamma\cos\beta}{\sin\gamma}\right)^2+\cos^2\delta$$ I will spare the details but it gives $$\cos\delta=\frac{\sqrt{1-2\cos\alpha\cos\beta\cos\gamma-\cos^2\alpha-\cos^2\beta-\cos^2\gamma}}{\sin\gamma}$$ Note that the choice of orientation of the two basis ensure that $\cos\delta\ge 0$ making sure we chose the right squared root.
By definition of $\vec{t'}$ we have $$\vec{u}\times\vec{v}=\sin\gamma\vec{t'}$$ Using $(1)$ and after some more involved calculus we get $$\vec{u}\times\vec{v}=\frac{(\cos\alpha\cos\gamma-\cos\beta)\vec{u}+(\cos\gamma\cos\beta-\cos\alpha)\vec{v}+\sin^2\gamma\vec{t}}{\sqrt{1-2\cos\alpha\cos\beta\cos\gamma-\cos^2\alpha-\cos^2\beta-\cos^2\gamma}}$$ Using symmetry arguments this easily gives $\vec{v}\times\vec{t}$ and $\vec{t}\times\vec{u}$. Note also that when the original basis is orthonormal, all cosines are equal to $0$ and the sine is equal to $1$: we find the usual $\vec{u}\times\vec{v}=\vec{t}$.
Finally I will comment your question. Looking at the form of the solution it seems very unlikely that there is an easier solution furthermore if $\vec{u}$, $\vec{v}$, and $\vec{t}$ are not unit vectors. Maybe some calculus can be avoided but it remains necessary at some point. This proof rests on finding $\cos\delta$. I have spent most of the day trying to find a more geometrical approach to express $\delta$ using $\alpha$, $\beta$, and $\gamma$ with no success... It still is another way from what you were doing though.