Two vectors of lengths $a$ and $b$ make an angle $\theta$ with each other when placed tail to tail. Show that the magnitude of their resultant is : $$r = \sqrt{ a^2 + b^2 +2ab\cos(\theta)}.$$
I understand that if we placed the two vectors head-to-tail instead of tail-to-tail, the Law of Cosines dictates that the resultant would be: $$\sqrt{ a^2 + b^2 -2ab\cos(\theta)}$$
However, In the situation actually described, the direction of vector $a$ has been reversed, which changes the sign of $2ab$ without changing the sign of $a^2$. But how do I prove that mathematically?
You got everything that you need.
Theorem: Given vectors $a$ and $b$ enclosing an angle $\theta$. Then the magnitude of the sum, $|a + b|$, is given by $\sqrt{ a^2 + b^2 +2ab\cos(\theta)}$.
Proof: Assuming that the Law of Cosines works for a case like the following, where $a$ and $b$ are the thick lines. The thin lines are just mirrors of the vectors.
http://chaos.stw-bonn.de/users/mu/uploads/2013-09-12/sp9.png
If our vectores are defined like I just stated, this holds: $$|a + b| = \sqrt{ a^2 + b^2 - 2ab\cos(\theta)}$$
Now we position the vector $b$ at the head of $a$. It looks like this:
http://chaos.stw-bonn.de/users/mu/uploads/2013-09-12/sp10.png
Note how the angle is now: $$\theta' = \pi - \theta$$
And with $\cos(\pi - \theta) = - \cos(-\theta) = -\cos(\theta)$ we get that $-$-sign and yield the formula $$|b - a| = \sqrt{ a^2 + b^2 + 2ab\cos(\theta)}$$
q.e.d.