In a triangle $ABC$ let's let $\overrightarrow{a}=\text{vector }\overrightarrow{BC}$, let $\overrightarrow{b}=\text{vector }\overrightarrow{CA}$, and let $\overrightarrow{c}=\text{vector }\overrightarrow{BA}$,. Let's let $\gamma=\text{angle }ACB$.
Then;
$$\overrightarrow{c}=\overrightarrow{a}+\overrightarrow{b}$$
$$\overrightarrow{c}\cdot\overrightarrow{c}=\left(\overrightarrow{a}+\overrightarrow{b}\right)\cdot\left(\overrightarrow{a}+\overrightarrow{b}\right)$$
$$\overrightarrow{c}\cdot\overrightarrow{c}=\overrightarrow{a}\cdot\overrightarrow{a}+\overrightarrow{b}\cdot\overrightarrow{b}+2\overrightarrow{a}\cdot\overrightarrow{b}$$
$$|c|^2=|a|^2+|b|^2+2|a||b|\cos(\gamma)$$
But, the cosine law says: $|c|^2=|a|^2+|b|^2-2|a||b|\cos(\gamma)$.
So I guess I made a mistake somewhere in the process. What was my mistake?
Because the angle ACB should be the angle formed by vector CA and CB instead of vector CA and vector BC, that was where you made the mistake.