A triangle with integer sides a, b and 13 is given so that a is also a prime. The angle opposite to c=13 is 120° (m(C)=120°).
Find a and b.
The answer is a=7 and b=8. Howewer, how can we solve it using modular arithmetical analysis without trying immediate integer values?
Apply the law of cosines.
You must have that
$$(13)^2 = a^2 + b^2 - 2ab[\cos(120^\circ)] \implies $$
$$(13)^2 = a^2 + b^2 + ab = (a + b)^2 - ab. \tag1 $$
This implies that $~(a + b + 13) \times (a + b - 13) = ab.$
Since $~a~$ is prime, $~a~$ must divide either $~(a + b + 13),~$ or $~(a + b - 13).~$ Further, it is immediate from (1) above, that both $~a~$ and $~b~$ are less than $~13.$
Therefore, $~a > a + b - 13 > 0.$
Therefore, the prime $~a~$ must divide $~a + b + 13.$
Therefore, the prime $~a~$ must divide $~b + 13.~$
Further, $~a + b - 13 > 0 \implies (a+b) > 13.$
You know that $~a \in \{2,3,5,7,11\}.$
$~a = 11~$ can be rejected, since $~a~$ must divide $~b + 13,~$ which would require $~b = 9.~$ Immediate that the ordered pair $~(a,b) = (11,9)~$ is simply too large.
You also have that $~b^2 + ba + (a^2 - 169) = 0,~$
which can be interpreted as a quadratic equation in $~b.~$
Therefore,
$$b = \frac{1}{2} \left[ -a \pm \sqrt{a^2 - 4(a^2 - 169)}\right].$$
This implies that
$$676 - 3a^2$$ is a perfect square, where $~a \in \{2,3,5,7\}.$
At this point, I am forced to resort to trial and error. Only $~a = 7,~$ results in $676 - 3a^2$ being a perfect square.
So:
$~a = 7.~$
$a + b > 13 \implies 6 < b < 13.$
$a~$ divides $~b + 13 \implies b \in \{1,8\}.$
So, the only try is $~(a,b) = (7,8).$