I have a triangle $T=ABC$. I want to calculate $\max (a-b)$, where the the angle $ABC = \beta$, and $|AB|=c$ is fixed (pre-known). My guess is $c\times\cos (\beta)$, but I want to prove it.
Let $A$,$B$,$C$ denote the vertices of $T$, and $|AB|=c$,$|AC|=b$, and $|BC|=a$.
We can use the cosine formula
$$b^2 = a^2 + c^2 - 2ac\cos \beta \Rightarrow b = \sqrt{a^2 + c^2 - 2ac\cos \beta}\ .$$
Write $f(a) = a-b = a- \sqrt{a^2 + c^2 - 2ac\cos \beta}$, then
$$f'(a) = 1- \frac{a-c\cos\beta}{\sqrt{a^2 + c^2 - 2ac\cos \beta}}\ .$$
As $a^2 + c^2 - 2ac\cos \beta = (a-c\cos\beta)^2 - c^2 \cos^2\beta + c^2 = (a-c\cos\beta)^2 + c^2 \sin^2\beta$, we see that
$$\bigg|\frac{a-c\cos\beta}{\sqrt{a^2 + c^2 - 2ac\cos \beta}} \bigg|\leq 1$$
and $f'(a)>0$ for all $a$. This means $f$ is increasing. So the maximum value of $f$ is never attained (that is the reason you cannot find $a$), but we can still find out the upper bound of $f$ by calculating
$$\lim_{a\to +\infty} f(a) = \lim_{a\to +\infty} \big(a- \sqrt{a^2 + c^2 - 2ac\cos \beta}\big) = c\cos\beta \ \ \text{(How?)}\ .$$
Thus your guess is almost correct: the value $c\cos\beta$ can never be attained, but is the smallest upper bound of $a-b$.