Given that $\vec{a}$ and $\vec{b}$ are two non-zero vector. The two vectors form 4 resultant vectors such that $\vec{a} + 3\vec{b}$ and $2\vec{a} - 3\vec{b}$ are perpendicular, $\vec{a} - 4\vec{b}$ and $\vec{a} + 2\vec{b}$ are perpendicular. How can I find the angle between $\vec{a}$ and $\vec{b}$?
The answer given here is 114.09. Any help is much appreciated.
On
\begin{cases} (\vec{a}+3\vec{b}).(2\vec{a}-3\vec{b})=0,\\ (\vec{a}-4\vec{b}).(\vec{a}+2\vec{b})=0. \end{cases} $\Rightarrow$ \begin{cases} 2|\vec{a}|^2-9|\vec{b}|^2+3\vec{a}.\vec{b}=0,\\ |\vec{a}|^2-8|\vec{b}|^2-2\vec{a}.\vec{b}=0. \end{cases} $\Rightarrow$ \begin{cases} |\vec{a}|^2=-6\vec{a}.\vec{b},\\ |\vec{b}|^2=-\vec{a}.\vec{b}. \end{cases} then $$\cos\theta=\dfrac{\vec{a}.\vec{b}}{|\vec{a}||\vec{b}|}=\dfrac{\vec{a}.\vec{b}}{\sqrt{|\vec{a}|^2|\vec{b}|^2}}=\dfrac{\vec{a}.\vec{b}}{-\sqrt{6}\vec{a}.\vec{b}}=-\dfrac{1}{\sqrt{6}}$$ since $\vec{a}.\vec{b}=-|\vec{b}|^2<0$, so $$\theta=\pi-\arccos\dfrac{1}{\sqrt{6}}=\pi-65^\circ.9=\color{blue}{114^\circ5'41''.4}$$
You are given that $\left\langle a+3b,2a-3b\right\rangle=0$ and $\left\langle a-4b,a+2b \right\rangle=0$. Hence
$$\left\langle a+3b,2a-3b\right\rangle=2\left\langle a,a\right\rangle-9\left\langle b,b\right\rangle+3\left\langle a,b\right\rangle=0$$ and $$\left\langle a-4b,a+2b \right\rangle=\left\langle a,a \right\rangle-8\left\langle b,b \right\rangle-2\left\langle a,b \right\rangle=0.$$(Here I assumed that you are working with a real inner-product space). By subtracting the second equation twice from the first, we obtain $$\left\langle b,b \right\rangle+\left\langle a,b \right\rangle=0.$$
Plugging the previous equation into the second yields $$\|a\|^2=6\|b\|^2.$$ Hence $\frac{\|b\|}{\|a\|}=\sqrt{\frac{1}{6}}.$
Now you know that $\left\langle a,b \right\rangle=\cos(\theta)\|a\|\|b\|.$ Thus, after plugging the previous results in the third equation, we get
$$\cos(\theta)=-\frac{\|b\|}{\|a\|}=-\sqrt{\frac{1}{6}}.$$